2.4.5 Local Minimum Problem

The backpropagation algorithm, as just described, employs gradient descent by following the slope of RMS error value $E_p$ downward along with the change in all the weight values. The weight values are constantly adjusted until the value of $E_p$ is no longer decreasing. Since the RMS error value is very complex function with many parameter values of weights, it is possible that the backpropagation network may converge into a local minima instead of the desired global minimum. This phenomenon of ``learning paralysis" can be avoided with several solutions suggested [Gur97]. One is the matter of order in presenting training samples to the learning network. Adding noise to the weights while being updated could be also the solution. Another answer is to utilize momentum, which gradually increases the weight adjustment rate $\beta$. All of these solutions are the way to escape from the trap of a local minimum.