Resilient propagation

Resilient backpropagation (Rprop) or elastic propagation is an iterative method for determining the minimum of the error function in a neural network . The algorithm is sometimes assigned to the second-order learning method group, since the last change in weight is included in the determination of the current change in weight. The weight is only changed according to the sign of the gradient. The step size is determined with a weight- specific parameter . The weight change is carried out in two steps. First step: For each weight, the change parameter for the kth iteration is determined as follows: ${\ displaystyle \ gamma}$ ${\ displaystyle \ gamma}$

{\ displaystyle \ gamma _ {ij} ^ {(k + 1)} = {\ begin {cases} \ min \ left (\ gamma _ {ij} ^ {(k)} \, \ cdot \ eta ^ {+ }, \ gamma _ {\ mathrm {max}} \ right) & {\ text {if}} \ \ nabla _ {ij} \, E ^ {(k)} \, \ cdot \, \ nabla _ {ij } \, E ^ {(k-1)}> 0 \\\ max \ left (\ gamma _ {ij} ^ {(k)} \, \ cdot \ eta ^ {-}, \ gamma _ {\ mathrm {min}} \ right) & {\ text {if}} \ \ nabla _ {ij} \, E ^ {(k)} \, \ cdot \, \ nabla _ {ij} \, E ^ {(k -1)} <0 \\\ gamma _ {ij} ^ {(k)} & {\ text {otherwise}} \ end {cases}}}

With ${\ displaystyle 0 <\ eta ^ {-} <1 <\ eta ^ {+}}$

Step size and maximum step size for a forward step or a backward step are defined with the parameters . Good values for the parameters are: ${\ displaystyle \ eta ^ {+}, \ eta ^ {-}, \ gamma _ {\ mathrm {min}}, \ gamma _ {\ mathrm {max}}}$ ${\ displaystyle \ eta ^ {+} = 1 {,} 2 \ ,, \, \ eta ^ {-} = 0 {,} 5 \ ,, \, \ gamma _ {\ mathrm {max}} = 50 \ ,, \, \ gamma _ {\ mathrm {min}} = 10 ^ {- 6}}$

In the second step, the change in the weights of the k-th iteration is determined:

{\ displaystyle w_ {ij} ^ {(k + 1)} = w_ {ij} ^ {(k)} + \ Delta \, w_ {ij} ^ {(k)}}

With

{\ displaystyle \ Delta \, w_ {ij} ^ {(k)} = - \ gamma _ {ij} ^ {(k)} \ operatorname {sgn} \ left (\ nabla _ {ij} \, E ^ { (k)} \ right)}

Here, the weight of the neuron j i for receipt and E is the sum of the errors. ${\ displaystyle w_ {ij}}$

The Rprop algorithm generally converges faster than the error feedback ( backpropagation ), but the extreme can be skipped due to the point of discontinuity at the minimum of the local approximation.

The Rprop algorithm was first presented in 1992 by Martin Riedmiller and Heinrich Braun in their work "Rprop - A Fast Adaptive Learning Algorithm".

Christian Igel and Michael Hüsken (2003) proposed a slight modification of the algorithm that increases its stability and speed. By Aristoklis D. Anastasiadis et al. (2005), global convergence of the procedure can be proven.

literature

M. Riedmiller and H. Braun: Rprop - A Fast Adaptive Learning Algorithm . Proceedings of the International Symposium on Computer and Information Science VII, 1992
M. Riedmiller and H. Braun: A direct adaptive method for faster backpropagation learning: The Rprop algorithm . Proceedings of the IEEE International Conference on Neural Networks, 586-591, IEEE Press, 1993
M. Riedmiller: Advanced supervised learning in multi-layer perceptrons - From backpropagation to adaptive learning algorithms . Computer Standards and Interfaces 16 (5), 265-278, 1994
C. Igel and M. Hüsken: Empirical Evaluation of the Improved Rprop Learning . Neurocomputing 50, 105-123, 2003
AD Anastasiadis, GD Magoulas and MN Vrahatis: New globally convergent training scheme based on the resilient propagation algorithm . Neurocomputing 64, 253-270, 2005

Resilient propagation

literature

Web links