Time Delay Neural Network
A Time Delay Neural Network (TDNN to German about Neural Delayed Network ) is a multilayer artificial neural network , which through the use of time slots is for multiple inputs in a position to process time dependencies of inputs. The name comes from the use of delay elements ( delays ) with which several time steps are applied in parallel.
An artificial neural network is a model from neuroinformatics that is motivated by biological neural networks . Artificial neural networks can learn tasks and are often used where explicit modeling of a problem is difficult or impossible. Examples are face and speech recognition .
structure
In order to allow the network to process invariant in time , the inputs of several points in time are created at the same time in a TDNN. The use of multiple layers of neurons supports detection regardless of the timing of occurrence in the input stream.
The inputs are arranged in a matrix at a TDNN. A column can contain several values and corresponds to a time step. With each new entry, the columns of the previous entries are shifted by one step, whereby the oldest entry is omitted.
The input matrix is then passed to a hidden layer made up of a matrix of neurons. Each column of neurons there receives only one selection (window) of the inputs. This window is shifted one step at a time for the next columns of the neuron matrix. The window width can be freely selected, but must be suitably determined for the desired invariance . There can be several hidden layers arranged one behind the other. The output shift finally receives the complete time window of the previous hidden shift.
They can be learned using the back propagation method . Since the neurons shifted in time to the respective first column, the hidden layers, each receive the same inputs from previous time steps and thus perform the same processing, the weights should also match the neurons in the first column. For this purpose, the learning process is modified in such a way that neurons of different time steps experience the same change, which is determined via the mean value of the weight changes . When calculating the neuron activations, a recalculation of the temporally shifted neurons can be omitted and instead the activations of previous time steps can be used.
properties
This step-by-step refinement ensures that the TDNN processes complex, non-linear inputs and that recognition is invariant over time. In addition, the nesting ensures that the number of connections remains low.
The disadvantage of a TDNN is the fixed window width used. The constant number of inputs created is a severe limitation for signals of different lengths.
application
TDNNs were mainly used in speech recognition . In the meantime, however, they have lost their importance there to the Hidden Markov Models , which in turn have lost their importance to LSTM .
literature
- Alexander Waibel et al., Phoneme Recognition Using Time-Delay Neural Networks , IEEE Transactions on Acoustics, Speech and Signal Processing, Volume 37, No. 3, pp. 328-339, March 1989 .
- TDNN Fundamentals , chapter from the SNNS online manual