For questions related to the often not well-defined or vaguely defined notion of capacity of a neural network (or any other machine learning model). The capacity of a neural network can be precisely defined as its VC dimension.
Questions tagged [capacity]
7 questions
10
votes
3 answers
Are there any rules of thumb for having some idea of what capacity a neural network needs to have for a given problem?
To give an example. Let's just consider the MNIST dataset of handwritten digits. Here are some things which might have an impact on the optimum model capacity:
There are 10 output classes
The inputs are 28x28 grayscale pixels (I think this…
Alexander Soare
- 1,319
- 2
- 11
- 26
8
votes
3 answers
What is the difference between hypothesis space and representational capacity?
I am reading Goodfellow et al Deeplearning Book. I found it difficult to understand the difference between the definition of the hypothesis space and representation capacity of a model.
In Chapter 5, it is written about hypothesis space:
One way…
Qwarzix
- 83
- 1
- 4
7
votes
2 answers
How to estimate the capacity of a neural network?
Is it possible to estimate the capacity of a neural network model? If so, what are the techniques involved?
jaeger6
- 308
- 1
- 7
5
votes
1 answer
Is running more epochs really a direct cause of overfitting?
I've seen some comments in online articles/tutorials or Stack Overflow questions which suggest that increasing the number of epochs can result in overfitting. But my intuition tells me that there should be no direct relationship at all between the…
Alexander Soare
- 1,319
- 2
- 11
- 26
3
votes
1 answer
Does adding a model complexity penalty to the loss function allow you to skip cross-validation?
It's my understanding that selecting for small models, i.e. having a multi-objective function where you're optimizing for both model accuracy and simplicity, automatically takes care of the danger of overfitting the data.
Do I have this right?
It…
Redrock
- 33
- 2
2
votes
2 answers
In classification, how does the number of classes affect the model size and amount of data needed to train?
When solving a classification problem with neural nets, be it text or images, how does the number of classes affect the model size and amount of data needed to train?
Are there any soft or hard limitations where the number of outputs starts to stall…
conscious_process
- 176
- 7
1
vote
1 answer
What is the minimum number of neurons and hidden layers needed to learn a Boolean function that maps $N$ bits to $1$ bit?
Suppose I have a Boolean function that maps $N$ bits to $1$ bit. If I understand correctly, this function will have $2^{2^N}$ possible configurations of its truth table.
What is the minimum number of neurons and hidden layers I would need in order…
wil3
- 119
- 1