The neural network model is the most crucial part of an autonomous vehicle. Fitting a proper model is therefore an important task. Part of this task is processing of the images taken by the vehicle in manual driving and used for training. Burro includes an easily customizable image pre-processing pipeline for training neural networks from driving datasets. This post provides a brief description thereof.
Why a transformation pipeline?
Prior to training a predictive model it is common to perform transformations on available data. Such transformations may be required due to the nature of the training data (e.g. normalizing training values, converting between formats etc.). Alternatively, transformations are useful in “augmenting” training data, by generating variations of the existing examples. Another case is functions that sample the training data to correct heavily skewed distributions.
The pipeline comprises components that are in fact just Python generator functions. They accept a tuple of image and steering values as input, and output a tuple of transformed values. Depending on the nature of the component it may act on the image, the steering value, or both. Below an example that accepts a steering angle in radians and returns it’s sinus value:
The function includes a simple for loop that iterates over tuples of input (image) and output (steering angles). It makes use of the
yield keyword to output one tuple at a time. The output consists of the same image as the input, and the sinus of the angle.
The generators are “linked” together at runtime to form a chain. A total of 19 components are available in four modules: File related, image related, numpy related and miscellaneous. One special component scans a specified directory for image files with special file names and outputs them. This component is always at the head of the chain.
Using the components outlined above a training pipeline is constructed. The pipeline is responsible for transforming image and steering values at the input and feeding them into the training algorithm. The figure below highlights the sequence of components in the pipeline included in Burro:
Below the function of each of the components in detail:
filename_generatorenumerates files present in the defined directory
nth_selecteither includes or excludes every nth item for cross-validation purposes
equalize_probsselectively skips examples based on the frequency of appearance, so as to equalize occurrences of different value ranges
image_generatorloads a PIL image corresponding to the filename passed
image_cropapplies mirror, resize and crop transformations to the image
array_generatorconverts the image to a numpy array
center_normalizezero-centers and normalizes image data
brightness_shifterrandomly adjusts image brightness
gaussian_noiseadds a randomized level of gaussian noise to the image
category_generatorconverts the scalar output value to a one-hot encoding, if specified
There are two pipelines available out of the box: The first trains categorical networks and the second regression networks. You can choose between them using a command line argument, as we will show in a bit.
Training a Model
Once run it will create a new folder for the training infrastructure and set up all necessary files. You should have the necessary system-wide libraries installed. Burro does not install any system-wide packages to avoid messing with your system. Mainly you should have Tensorflow installed for your specific hardware (CPU or GPU).
Once you have the training variant of Burro installed, you can train a regression model like so:
or alternatively a categorical model like so:
The training will start and Keras messages will periodically inform you of the progress.
This post presented a brief outline of the autonomous driving model training functionality in Burro. Using the method presented you can fit your own models on existing datasets, or even on your own datasets that you record while driving your autonomous vehicle around.
If you do build your own vehicle or driving model, make sure to share your experience in the comments below!