Traffic sign recognition

Contact person: Anton Konushin

Team: Anton Konushin, Vlad Shakhuro


Despite many years of active research traffic sign detection and recognition is still an open problem. Recognition accuracy and false detections rate are still not sufficient for replacing human operators with automatic recognition system. We are working on several topics in this area:

  • high-speed detection
  • sign recognition with multi-layer neural networks
  • synthetic datasets generation for training

Russian traffic sign images dataset (RTSD)

This dataset is intended for training and testing the algorithms of traffic sign recognition. Frames are obtained from widescreen digital video recorder which captures 5 frames per second. Frame resolution is from 1280×720 to 1920×1080. Frames are captured in different seasons (spring, autumn, winter), time of day (morning, afternoon, evening) and in different weather conditions (rain, snow, bright sun). This dataset surpasses other public traffic sign datasets in number of frames, signs classes, physical signs and images of signs. Sign labeling on frames was spent in two steps. On first step tracks of physical objects were selected on sequential frames. On the second step indistinguishable signs were discarded and every physical sign was assigned a class.

It consists of 205 classes, of which 99 are found only in the test set and are completely absent in the training set, and 106 classes are present in the training set.

Statistics for detection task RTSD dataset:

Images Signs
Train 47639 80277
Test 11389 25232

Statistics for classification task RTSD dataset:

All Rare Frequent
Train 79896 0 79896
Test 25613 1622 23991

Synthesis of data for the classifier

The problem of recognizing road signs is unsolved for a large number of classes of signs, since there are many rare classes of signs. For systems that recognize road signs, the ability to quickly adapt in the event of a new type of road signs is important. With the emergence of a new class of signs, it is difficult to collect a sufficient number of real training examples. In the presence of synthetically generated images of this sign, the ability to classify a new type of sign can be quickly introduced into the system. High-quality synthetic samples allow us to solve the problem with missing classes and data collection with examples of these signs.

Existing methods for traffic sign processing:

  • Synt - this is a simple synthetic, which was obtained by embedding signs on the background and applying a transformation of sign with random parameters to the icon: rotate, shift, contrast change, Gaussian blur, motion blur.
  • CGI - samples, which were obtained by rendering three-dimensional models of traffic signs on pillars in real road images.
  • CGI-GAN - in this sample, traffic signs are transformed from the CGI collection to better ones using CycleGAN.
  • Pasted - this approach is based on embedding synthetic traffic sign on a real traffic sign place. The real traffic sign itself is inpainted. Then embedded sign is being improved with GAN.
  • Cycled - this is improvement of Pasted method. Here, the second data stream is added to the training process. The icon of the sign of the same class, which was in a real patch, is embedded. As a result, the entire neural network should ideally get a picture identical to the original one.
  • Styled - this approach uses StyleGAN architecture. Neural network generates synthetic traffic sign based on the background image and on a sign icon of required class. The architecture of this approach is in the next figure:

Architecture of generator for processing in Styled approach

Architecture of generator for processing in Styled approach

Synthesis of data for the detector

The quality of the detectors can also be improved with synthetic signs. In this task, the consistency of the appearance of the sign with the background is especially important. The correct positioning of the new synthetic road sign is also very important.

Existing methods for embedding of traffic signs in images:

  • Inpaint – this is a simple synthetic data for the detector, in which an icon of a traffic sign is drawn in the image without any processing.
  • Pasted, Cycled, Styled - these methods are initially designed to generate a sign consistent with the input background. Further, the processed new sign is cut out and inserted in the desired place.
  • KDE - new places of traffic signs were determined or using kernel density estimation.
  • NN-placement - new places of traffic signs were determined or a special neural network.
Real image

Real image

Image with additional signs

Image with additional signs

Current best results

Classification of signs in RTSD dataset:

All, Accuracy Rare, Recall Frequent, Recall
RTSD 88.87 0 94.88
RTSD + CGI-GAN 93.52 70.16 95.09
RTSD + Styled 94.11 76.33 95.31
RTSD + Styled + SWA 97.17 77.19 98.52

Detection of signs in RTSD dataset (AUC) with classifier:

All Rare Frequent
RTSD 86.01 58.56 86.61
RTSD + CGI 83.84 48.51 85.15
RTSD + Styled 85.39 64.20 86.13
RTSD + KDE-placement 85.99 64.83 86.54
RTSD + NN-placement 86.16 64.96 86.70
RTSD + NN-placement + ATSS detector 88.87 70.45 89.01


Selected Publications

  1. Data Distillation for Traffic Sign Detection. A. Popov, V. Shakhuro, A. Konushin. Proceedings of the 30th International Conference on Computer Graphics and Machine Vision, 2020
  2. Классификация редких дорожных знаков. Б. Фаизов, В. Шахуро, В. Санжаров, А. Конушин. Компьютерная оптика, 2020
  3. Rare traffic sign recognition using synthetic training data. V. Shakhuro, B. Faizov, A. Konushin. ICVIP 2019: Proceedings of the 3rd International Conference on Video and Image Processing, 2019
  4. Синтез изображений дорожных знаков с помощью условных порождающих противоборствующих нейросетей. П. Хрушков, В. Шахуро, А. Конушин. Proceedings of the 28th International Conference on Computer Graphics and Vision GraphiCon'2018, 2018
  5. Синтез обучающих выборок для классификации дорожных знаков с помощью нейросетей. В. Шахуро, А. Конушин. Компьютерная оптика, 2018
  6. Российская база изображений автодорожных знаков. В. Шахуро, А. Конушин. Компьютерная оптика, 2016
  7. Evaluation of Traffic Sign Recognition Methods Trained on Synthetically Generated Data. B. Moiseev, A. Konev, A. Chigorin, A. Konushin. Advanced Concepts for Intelligent Vision Systems, 2013
  8. A system for large-scale automatic traffic sign recognition and mapping. A. Chigorin, A. Konushin. ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences, 2013