2.3.3 Artificial Neuron with Continuous Characteristics

Based on the McCulloch-Pitts model described previously, the general form an artificial neuron can be described in two stages shown in figure 2.5. In the first stage, the linear combination of inputs is calculated. Each value of input array is associated with its weight value, which is normally between 0 and 1. Also, the summation function often takes an extra input value $\theta$ with weight value of 1 to represent threshold or bias of a neuron. The summation function will be then performed as,

$$x = \mathop{\sum_{i = 1}^{N}}{A_{i} W_{i}} + \theta.$$ (2.4)

Figure 2.5: More General Neural Model

The sum-of-product value is then passed into the second stage to perform the activation function which generates the output from the neuron. The activation function ``squashes" the amplitude the output in the range of $[0,1]$, or alternately $[-1,1]$ [Hay99]. The behavior of the activation function will describe the characteristics of an artificial neuron model.

The signals generated by actual biological neurons are the action-potential spikes, and the biological neurons are sending the signal in patterns of spikes rather than simple absence or presence of single spike pulse. For example, the signal could be a continuous stream of pulses with various frequencies. With this kind of observation, we should consider a signal to be continuous with bounded range. The linear threshold function should be ``softened" [BL96].

One convenient form of such ``semi-linear" function is the logistic sigmoid function, or in short, sigmoid function as shown in figure 2.6. As the input $x$ tends to large positive value, the output value $y$ approaches to 1. Similarly, the output gets close to 0 as $x$ goes negative. However, the output value is neither close to 0 nor 1 near the threshold point. This function is expressed mathematically as follows:

$$y = \frac{1}{1 + exp(-x)}.$$ (2.5)

Figure 2.6: Sigmoid Function

Additionally, the sigmoid function describes the ``closeness" to the threshold point by the slope. As $x$ approaches to $-\infty$ or $\infty$, the slope is zero; the slope increases as $x$ approaches to 0. This characteristic often plays an important role in learning of neural networks.