5

Consider the following statement(s) from Deep Learning book (p. 333, chapter 9: Convolutional Networks) by Ian Goodfellow et al.

Convolution is thus dramatically more efficient than dense matrix multiplication in terms of the memory requirements and statistical efficiency.

Book is saying that statistical efficiency is due to the decrease in the number of parameters due to convolution (using kernel) compared to fully connected feed forward neural networks.

What is meant by statistical efficiency in this context? And how does decrease in the number of parameters increase statistical efficiency?

hanugm
  • 3,571
  • 3
  • 18
  • 50

1 Answers1

5

Statistical efficiency in this context essentially means that a CNN would require fewer training examples than a fully connected network to learn. Intuitively this seems reasonable: more parameters to learn should mean more samples needed. Of course it is always desirable to minimise the number of training samples needed, so that's a definite advantage of CNNs.

There is a paper on the efficiency of CNNs which attempts to make that statement more precise. They examine the case of a convolutional network using a linear activation function.

htl
  • 1,000
  • 1
  • 4
  • 13