Rectifier (neural networks)

Rectifier activation function

In the context of artificial neural networks , a rectifier is an activation function of an artificial neuron , which is defined as the positive part of its argument:

{\ displaystyle f (x) = \ max (0, x)}

with as the input value of the artificial neuron. ${\ displaystyle x}$

This activation function was used in 2000 by Hahnloser et al. Introduced for the first time in dynamic neural networks with strong biological motivations and mathematical justifications. Rectifying activation functions were used in the neuronal abstraction pyramid proposed by Behnke in order to separate specific excitation and non-specific inhibition. This hierarchical recurrent convolutional architecture was supervised and trained to iteratively solve different computer vision tasks. It was first demonstrated in 2011 that training deep networks with rectifying activation functions is more successful than with activation functions that were widespread before 2011, such as B. the sigmoid function . Rectifiers are currently (as of 2019) the most popular activation functions for deep neural networks. A unit that uses the rectifier is also known as a “rectified linear unit” (ReLU). Such ReLUs are used in deep learning , for example in machine vision and speech recognition .

Individual evidence

↑ Richard HR Hahnloser, Rahul Sarpeshkar u. a .: Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit. In: Nature. 405, 2000, p. 947, doi : 10.1038 / 35016072 .
↑ Sven Behnke: Hierarchical Neural Networks for Image Interpretation (= Lecture Notes in Computer Science), Volume 2766. Springer, 2003, doi : 10.1007 / b11963 .
↑ Xavier Glorot, Antoine Bordes, Yoshua Bengio: Deep Sparse Rectifier Neural Networks . In: Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics (= Proceedings of Machine Learning Research ). tape 15 . PMLR, Fort Lauderdale, FL, USA April 2011, pp. 315-323 ( mlr.press [PDF]).
^ Dan Becker: Rectified Linear Units (ReLU) in Deep Learning. In: Kaggle . Retrieved November 19, 2018 .
↑ Xavier Glorot, Antoine Bordes and Yoshua Bengio: Deep sparse rectifier neural networks. 2011, accessed November 19, 2018 .
^ László Tóth: Phone Recognition with Deep Sparse Rectifier Neural Networks. Retrieved November 19, 2018 .

[1] Richard HR Hahnloser, Rahul Sarpeshkar u. a .: Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit. In: Nature. 405, 2000, p. 947, doi : 10.1038 / 35016072 .

[NeuralAbstractionPyramid-2] Sven Behnke: Hierarchical Neural Networks for Image Interpretation (= Lecture Notes in Computer Science), Volume 2766. Springer, 2003, doi : 10.1007 / b11963 .

[3] Xavier Glorot, Antoine Bordes, Yoshua Bengio: Deep Sparse Rectifier Neural Networks . In: Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics (= Proceedings of Machine Learning Research ). tape 15 . PMLR, Fort Lauderdale, FL, USA April 2011, pp. 315-323 ( mlr.press [PDF]).

[4] Dan Becker: Rectified Linear Units (ReLU) in Deep Learning. In: Kaggle . Retrieved November 19, 2018 .

[5] Xavier Glorot, Antoine Bordes and Yoshua Bengio: Deep sparse rectifier neural networks. 2011, accessed November 19, 2018 .

[6] László Tóth: Phone Recognition with Deep Sparse Rectifier Neural Networks. Retrieved November 19, 2018 .