Traffic sign recognition for highly automated driving (SAE Level 4)

Highly automated vehicles rated for SAE Level 4 have to be able to reliably recognize every road sign, be it a regulatory, warning, or guide sign. We drew on deep learning methods to develop a recognition and tracking module.

The challenge: Recognize road signs based on image data sourced from different camera systems

Highly automated vehicles rated for SAE Level 4 have to accurately recognize, interpret, and respond to every regulatory, warning, and guide sign. The vehicle takes over at this level of automation, so the driver no longer has to maintain control at all times. This is why the systems’ fail-safety is absolutely crucial. Our project brief here was to first design a module to recognize main and supplementary signs using image data sourced from several camera systems, and integrate it into the given architecture. Then we installed and tested the module in a test vehicle.

Traffic road sign: highly automated vehicles

Particularly tricky for recognition systems – text-based road signs

The system has to be able to recognize, extract, and understand the writing on text signs. This is no trivial task at higher speeds or in bad weather. Some countries’ main signs consist only of text with no symbols. Any misinterpretation of these words could precipitate a life-threatening situation.

The solution: Develop a recognition and tracking module based on deep learning methods

Starting with a concept that culminated in a prototype, our team developed a convolutional neural networks (CNN)-based recognition and tracking module for different camera types. We also integrated an optical character recognition (OCR) module to reliably interpret text-based signs. Supplementary signs contain text that furnishes traffic-related information. This too has to be recognized and accurately interpreted. After we developed the module on an NVIDIA hardware platform, we optimized and integrated the prototype in an actual vehicle.

Featured technologies:

Separate networks for object detection and classification
Optimization of networks with TensorRT
Development in TensorFlow
NVIDIA hardware
System communication via ROS
Setting up of automatic software tests
Continuous integration of software components

The added value: A reliable recognition system and reusable data sets

This project produced a fully functional detection system for main and supplementary road signs that is suitable for SAE Level 4 and 5. The solution supports multiple camera types and features an integrated tracking system that delivers improved performance and greater safety. The module may be ported to other platforms using the same sensor set. The road sign data set is reusable.