2

I am reading the Simon Haykin's cornerstone book, "Neural Networks, A Comprehensive Foundation, Second Edition" and I cannot understand a paragraph below:

The analysis of the dynamic behaviour of neural networks involving the application of feedback is unfortunately complicated by virute (or virtue I cannot get word appropriately) of the fact that the processing units used for the construction of the network are usually nonlinear. Further consideration of this issue is deferred to the latter part of the book.

Before the paragraph, the author analysis the affects of weight of synapsis to the neural network's stability. Roughly speaking, he says, if |w| >= 1 the neural network become unstable.

Could you please explain the paragraph? Thanks in advance.

tahasozgen
  • 287
  • 1
  • 2
  • 7
  • Is "(or virtue I cannot get word appropriately)" part of the quote, or a note by you that you are having trouble reading the word? Seems a bit odd. Using square brackets usually indicates that you are commenting, and it is not part of the quote . . . the word should be "virtue". In addition, could you give chapter or other context for this paragraph . . . – Neil Slater Jun 20 '18 at 20:58
  • Yes, you are right, it was my comment. The passage is in page 20, Chapter 1, "Introduction" – tahasozgen Jun 21 '18 at 08:04
  • I think I'll add some features about amplifiers to make it more readable for you and all –  Jun 23 '18 at 08:37

1 Answers1

0

Although you have not given the entire context but if I were to speculate I would suggest the author is simply trying to tell that Neural Networks are quite difficult to analyse mathematically due to their non-linear nature (due to the use of non-linear activation). There are many questions on this Stack about the mathematical basis of NN.

Do scientists know what is happening inside artificial neural networks?

Also forward propagation is difficult enough, and now we that we have included feedback it becomes exponentially tough. You can easily understand the difficulty posed in such analysis if you take the case of Feedback Amplifiers in electronics. In normal sound amplifiers to avoid distortion we use a negative feedback. This roughly has an inverted waveform compared to i/p signal. It adds up with the input signal and smoothens out the distortions (as opposite wavefroms of not entirely same magnitude adds up).

|W| > 1 should be the case of Positive feedback, which can cause massive variations in output due to small changes due to its self reinforcing property. The wave-forms are of the same shape and they add up, this results in an even more larger waveform in the next cycle and so on. So probably the author is speaking something along these lines.