Why are deep neural networks and deep learning insufficient to achieve general intelligence?

Question

Everything related to Deep Learning (DL) and deep(er) networks seems "successful", at least progressing very fast, and cultivating the belief that AGI is at reach. This is popular imagination. DL is a tremendous tool to tackle so many problems, including the creation of AGIs. It is not enough, though. A tool is a necessary ingredient, but often insufficient.

Leading figures in the domain are looking elsewhere to make progress. This report/claim gathers links to statements by Yoshua Bengio, Yann LeCun and Geoff Hinton. The report also explains:

The main weaknesses of DL (as I see them) are: reliance on the simplest possible model neurons (“cartoonish” as LeCun calls them); use of ideas from 19th century Statistical Mechanics and Statistics, which are the basis of energy functions and log-likelihood methods; and the combination of these in techniques like backprop and stochastic gradient descent, leading to a very limited regime of application (offline, mostly batched, supervised learning), requiring highly-talented practitioners (aka “Stochastic Graduate Descent”), large amounts of expensive labelled training data and computational power. While great for huge companies who can lure or buy the talent and deploy unlimited resources to gather data and crunch it, DL is simply neither accessible nor useful to the majority of us.

Although interesting and relevant, such kind of explanation does not really address the gist of the problem: What is lacking?

The question seems broad, but it may be by lack of a simple answer. Is there a way to pin-point what DL is lacking for an AGI ?

Answer 1

Everyone dealing with neural networks misses an important point when comparing systems with human like intelligence. A human takes many months to do anything intelligible, let alone being able to solve problems where adult humans can barely manage. That and the size of human brain is enormous compared to our neural networks. Direction might be right, but the scale is way off. Number of neurons in human brain can be matched memory-wise but the amount of parallelism to simulate it real-time cannot yet be achieved (at least for a random researcher). While a little old this might give you an idea of how much we lack the processing power.

Answer 2

Deep Learning is mostly successful in supervised learning, whereas the brain builds categories mostly in an unsupervised way. We don't yet know how to do that. (Take a look at google brain: 16,000 cores and all this thing can do is recognise cats and human faces with pretty abysmal accuracy.)

Deep Learning uses highly unstructured activations, i.e. the high level representations of "dog" and "cat" in a neural network classifier don't have to be similar at all. The brain on the other hand uses inhibitory neurons to create sparse distributed representations which are decomposable into their semantic aspects. That's probably important for abstraction and reasoning by analogy.

The brain has many different parts which work together. Deep Learning researchers are only just beginning to integrate memory or attention mechanisms into their architecture.

The brain integrates information from many different senses. Most Deep Learning applications use just one type of input, like text or pictures.

The brain is capable of modelling sequences as categories. (Basically every verb names a sequential (i.e. temporal) category.) It can then arrange these categories into long-term hierarchical plans. So far I haven't seen anything in that direction in Deep Learning.

Also neural networks can't yet operate on the same scale as the human brain. If you look at the answers to this question, the human brain will be ahead in neuron count for another couple of decades. A neural network might not need the same number of neurons as the brain to reach a similar performance (because of higher accuracy), but right now for example video processing is still pretty limited in terms of input and throughput.

Answer 3

IMHO the first hurdle is scale: even Google's largest DNN doesn't come close to the scale of the brain, and by a factor of several orders of magnitude...

Answer 4

I think it's missing still the aspects what makes a human brain; having a lot of different networks working with each other.

Just like meditation improves cognitive abilities by having the brain work more synergistically, we could apply that to machines too.

For example google is learning a computer to dream, just like we do, to reinforce what we already learned. https://medium.com/@tannistho/why-is-google-teaching-its-ai-to-dream-e9ae9ecd0e3a#.gljal6pww

And here is pathnet, a network of neural network. https://medium.com/@thoszymkowiak/deepmind-just-published-a-mind-blowing-paper-pathnet-f72b1ed38d46#.ed0f6pdq7

Creating all these mechanics and putting them all together, with enough power and we will get pretty close!

Answer 5

Artificial intelligence proponents today are focused on the problem of computability - the ability to solve complex problems fast. It is my belief that any amount of success in this direction will not lead to human (general) intelligence although it certainly will outperform humans in certain domains. Instead, efforts should be toward a study of what neurological events cause sensation (the experience of qualia). Of course, this is the hard problem of philosophy but I believe it is the unique key to general intelligence and its capabilities. Reverse engineering and also testable theories should be advanced toward this end.