Recurrent neural network

As recurrent or feedback neural networks are called neural networks , which in contrast to the to feedforward networks by compounds of neurons of a layer neurons to the same or a previous layer distinguished. In the brain , this is the preferred way of interconnecting neural networks, especially in the neocortex . In artificial neural networks , recurrent interconnections of model neurons are used to discover time-coded information in the data. Examples of such recurrent neural networks are the Elman network , the Jordan network , the Hopfield network and the completely interconnected neural network.

Different types of feedback: the blue edge ( ) is direct, the green ( ) indirect, and the red ( ) side feedback. ${\displaystyle w_{d}}$ ${\displaystyle w_{i}}$ ${\displaystyle w_{l}}$

Recurrent networks can be divided as follows:

• In a direct feedback ( English direct feedback ) the own output of a neuron is used as a further input.
• The indirect feedback ( English indirect feedback ) connects the output of a neuron with a neuron of the previous layers.
• The side feedback ( English laterally feedback ) connects the output of a neuron to another neuron same layer.
• With a full connection, every neuron output has a connection to every other neuron.

Recurrent neural networks are used in practice for problems that require the processing of sequences. Examples are handwriting recognition , speech recognition, and machine translation . The predominant type of recurrent neural networks here are LSTMs or similar variants that are based on direct feedback.

Training of recurrent neural networks

Recurrent artificial neural networks are difficult to train using machine learning methods . A popular approach is therefore not to train the network but to train the network to be read out. The recurrent neural network is viewed as a so-called reservoir . In the case of LSTMs, the networks are converted into a feedforward network according to the sequence length using backpropagation through time (see backpropagation ) during the training process. This makes the complexity of the learning behavior similar to that of conventional feedforward networks.

literature

• Andreas Zell: Simulation of neural networks. R. Oldenbourg Verlag, Munich 1997, ISBN 3-486-24350-0 .

Individual evidence

1. ^ Rudolf Kruse et al .: Neural Networks | Computational Intelligence. In: Computational Intelligence: A methodical introduction to Artificial Neural Networks, Evolutionary Algorithms, Fuzzy Systems and Bayesian Networks. Second edition. Springer-Vieweg, Wiesbaden, 2015, accessed on April 5, 2017 . 
2. ↑ Rudolf Kruse et al .: Computational Intelligence: A methodical introduction to Artificial Neural Networks, Evolutionary Algorithms, Fuzzy Systems and Bayesian Networks. Second edition. Springer-Vieweg, Wiesbaden 2015, ISBN 978-3-658-10903-5 , pp. 515 .
3. ^ Reservoir Computing. (No longer available online.) Reservoir Lab Ghent, May 30, 2008, archived from the original on April 5, 2010 ; Retrieved April 2, 2010 . Info: The archive link was inserted automatically and has not yet been checked. Please check the original and archive link according to the instructions and then remove this notice.  
4. ↑ Chris Nicholson, Adam Gibson: A Beginner's Guide to Recurrent Networks and LSTMs - Deeplearning4j: Open-source, distributed deep learning for the JVM. In: deeplearning4j.org. Retrieved July 16, 2016 .