3

I am familiar with the currently popular neural network in deep learning, which has weights and is trained by gradient descent.

However, I found many papers that were popular in the 1980s and 1990s. These papers have titles like "Neural networks to solve optimization problems". For example, Hopfield first use this name, and they used "Neural network" to solve linear programming problems [1]. Later, Kennedy et.al used "Neural network" to solve nonlinear programming problems [2].

I summarize the difference between the current popular neural network and the "Neural networks":

  1. They do not have parameter weights and bias to train or to learn from data.
  2. They used a circuit diagram to present the model.
  3. The model can be simplified as an ODE system and has a Lyapunov function as objective.

Please take a look at these two papers in the 1980s:

  1. [Neurons with graded response have computational properties like those of two state neurons][2] (J.J. Hopfield)

  2. Neural Networks for Non-linear Programming (M.P Kennedy & L.O Chua)

Reference:

[1]: J. J. Hopfield, D. W. Tank, “neural” computation of decisions in optimization problems, Biological265 cybernetics 52 (3) (1985) 141–152.

[2]: M. P. Kennedy, L. O. Chua, Neural networks for nonlinear programming, IEEE Transactions on Circuits and Systems 35 (5) (1988) 554–562.

Mithical
  • 2,885
  • 5
  • 27
  • 39
dawen
  • 131
  • 3
  • You seem to be asking us why John Hopfield used a circuit diagram when he wrote his paper describing his neural networks. Given that we're not able to read Hopfield's mind, I don't think it's possible to answer this question (aside from giving the obvious answer, which is "because Hopfield thought that was a good way to describe his neural networks"). – Tanner Swett Apr 29 '21 at 21:29
  • I agree with the previous comment. I don't really understand what your question really is here. Can you clarify that? Moreover, in the body, you only talk about Hopfield networks, while in the title you're talking about "some neural networks". Which ones, apart from the Hopfield network? In other words, if you're just referring to the Hopfield network, don't use "some", but be as much specific as possible. – nbro Apr 30 '21 at 10:56
  • 1. I don't agree with the user 'nbro' comment. I believe I have stated the question clearly. I am asking about a general type of neural network models(many) presented by circuit diagram, and I referred to the Hopfield network(one) as a classic example. The Hopfield network(one) obviously belongs to the general type of neural network models(many) I was talking about. I don't see any confusion about this. – dawen Apr 30 '21 at 12:31
  • 2. I don't agree with the user ' Tanner Swett' comment. It is possible to answer the question. You don't need to go inside someone brain to understand his thought. There must be some reason or some benefit to using the circuit diagram, the reason may state somewhere I don't know. And this is why I am asking the question here, hope someone knows it can give an answer. – dawen Apr 30 '21 at 12:38
  • I've rolled back an edit that changed the question and invalidated the existing answers. In the future, please don't substantially change the question after answers have already been posted. Thanks! – Mithical May 14 '23 at 07:09

2 Answers2

3

In the early days of neural networks the theorists and practitioners were educated in mathematics, psychology, neurophysiology, electrical engineering, and neurobiology. Computer science was still in its infancy. The first neural networks were modeled as electrical circuits.

There is evidence of this in the 1943 paper by Warren McCulloch and Walter Pitts [1], and a 1956 paper by Rochester et al. [2].

The latter paper uses terms such as 'circuits' and 'switching'. One idea in the paper is explained in terms of a "Eccles-Jordan Flip Flop circuit" although there are no drawings. Nathanial Rochester had designed the IBM 701k [3] and "led the first effort to simulate a neural network" [4]

Brain structure was discussed in terms of 'neural circuits' as early as 1937 [5].

I am not sure when the first electrical circuit diagram appeared in publication, but it makes sense that early neural network designers, would have thought of their implementation as such.

References:

Brian O'Donnell
  • 1,853
  • 6
  • 20
2

Well, the goal of any paper is to allow the reader to understand what the author is trying to describe.

A lot of people have a lot of experience looking at circuit diagrams and figuring out those circuits will do. For these people, a circuit diagram may be the clearest and easiest way for them to understand how a particular thing works. So, it makes sense that an author would include a circuit diagram, in order to make it easy for those people to understand the concepts.

There are two particular reasons why a circuit diagram is especially likely to show up in a paper about neural networks:

The first reason is that analog circuits are closely related to ordinary differential equations, and digital circuits are closely related to sequential logic. So, if you have a neural network or something that uses ordinary differential equations or sequential logic, then a circuit diagram might be a simple way to express how it works.

The second reason is that a lot of researchers who are familiar with computers are also familiar with electronic circuits. This was especially true in the early days of computers, when people had to be familiar with electronics, math, or both in order to understand how to program a computer.

Tanner Swett
  • 151
  • 6
  • Thanks for your answer. Can you give more detail or explanation or reference that circuit diagrams are closely related ODE system? – dawen Apr 30 '21 at 17:02
  • @dawen That's the material that would be covered in an "introduction to electronic circuits" course. – Tanner Swett Apr 30 '21 at 17:27