4

I've read that ANNs are based on how the human brain works. Now, I am reading about dropout. Is some kind of dropout used in the human brain? Can we say that the ability to forget is some kind of dropout?

nbro
  • 39,006
  • 12
  • 98
  • 176
Seal-Trainer
  • 41
  • 1
  • 3

1 Answers1

5

The human brain works by having neurons constantly fire at different rates. So, if the firing rate increases, the neuron is transmitting overly exciting or calming information to further neurons connected to it. How other neurons connected to the former neuron respond on the messages sent by it, depends on the strength of the connection between the connected neurons. However, having true dropout in the brain would be like having a neuron stop firing entirely for some time. As far as I know, this is not gonna happen unless a neuron dies. So, no, there doesn't seem to be any real biological equivalent to dropout in the brain. If you forget something, that is just caused by a weakening of some connections in the brain. But such a weakening (or strengthening alternatively) is a gradual process and is a function of the degree of firing synchrony between any two connected nodes.

It is true that the introductory texts to Neural Nets tell you about those nets' neuro-scientific inspiration, but this similarity holds only in the most abstract way. Yes, you have neurons in both biological and artificial neural networks. But communication in the brain works in an asynchronous, continuous bio-chemical way, while communication in an artificial neural net works by taking the numeric state of one node, scaling it by the weight of the connection between two nodes, and then adding the weighted contribution of the previous node to the activation of the receiving node. So, already the fundamentals are very different between the two approaches (with the biological variant being much more complex and elaborate).

If you want to find something that possesses more biologically inspired properties, look at how Convolutional Neural Networks are theoretically inspired by the working of the (human) visual system. Besides that, dropout is a purely mechanical strategy to regularize a model well, in order to enhance the model's generalization ability, i.e. making the model work well also on unseen data. It's just part of an algorithmic optimization procedure.

Daniel B.
  • 805
  • 1
  • 4
  • 13
  • so when a physical neuron fires at minimal rate, like zero output, it's dropout? – Dee Jun 24 '20 at 06:52
  • 1
    No, you can think of a physical neuron like a switch or feature detector. If enough activation has accumulated in the neuron (as received from previous neurons), some feature has been detected and it changes into the 'active'- or 'on'-state (increasing its firing rate). But if it has received too little activation to overcome its firing threshold, that merely indicates the absence of the feature the neuron is supposed to detect. Dropout is like forcefully preventing a neuron from firing in the presence of its feature. That's just not happening in the brain. – Daniel B. Jun 24 '20 at 07:18