Artificial Intelligence Stack Exchange
Artificial Intelligence Stack Exchange

Questions tagged [cross-validation]

For questions related to the cross-validation techniques (e.g. k-fold cross-validation or leave-one-out cross-validation) used in machine learning to assess the quality (e.g. average accuracy) of the models.

47 questions
16
votes
1 answer

Will parameter sweeping on one split of data followed by cross validation discover the right hyperparameters?

Let's call our dataset splits train/test/evaluate. We're in a situation where we require months of data. So we prefer to use the evaluation dataset as infrequently as possible to avoid polluting our results. Instead, we do 10 fold cross validation…
Philipp Cannons
  • 161
  • 6
7
votes
1 answer

What is the best measure for detecting overfitting?

I wanted to ask about the methodology of testing the ML models against overfitting. Please note that I don't mean any overfitting reducing methods like regularisation, just a measure to judge whether a model has overfitting problems. I am currently…
GKozinski
  • 1,240
  • 8
  • 19
6
votes
4 answers

Is k-fold cross-validation more effective than splitting the dataset into training and test datasets to prevent overfitting?

I want to prevent my model from overfitting. I think that k-fold cross-validation (because it is doing this each time with different datasets) may be more effective than splitting the dataset into training and test datasets to prevent overfitting,…
jennifer ruurs
  • 579
  • 2
  • 8
5
votes
1 answer

How to decide a train-test split?

In almost every ML model, a train-test (or train-test-val split) is critical to assess the model's performance. However, I have always wondered what the rationale is to decide a particular train-test split. I've seen that some people like an 80-20…
user48670
4
votes
1 answer

What are "development test sets" used for?

This is a theoretical question. I am a newbie to artificial intelligence and machine learning, and the more I read the more I like this. So far, I have been reading about the evaluation of language models (I am focused on ASR), but I still don't get…
little_mice
  • 143
  • 2
3
votes
1 answer

Does adding a model complexity penalty to the loss function allow you to skip cross-validation?

It's my understanding that selecting for small models, i.e. having a multi-objective function where you're optimizing for both model accuracy and simplicity, automatically takes care of the danger of overfitting the data. Do I have this right? It…
Redrock
  • 33
  • 2
3
votes
2 answers

Calculating accuracy for cross validation

I'm struggling with calculating accuracy when I do cross-validation for a deep learning model. I have two candidates for doing this. 1. Train a model with 10 different folds and get the best accuracy of them(so I get 10 best accuracies) and average…
Juna
  • 31
  • 1
3
votes
1 answer

What is the relationship between the training accuracy and validation accuracy?

During model training, I noticed various behaviour in between training and validation accuracy. I understand that 'The training set is used to train the model, while the validation set is only used to evaluate the model's performance...', but I'd…
nilsinelabore
  • 241
  • 2
  • 12
3
votes
1 answer

Should I use leave-one-out cross-validation for testing?

I am currently working with a small dataset of 20x300. Since I have so few data points, I was wondering if I could use an approach similar to leave-one-out cross-validation but for testing. Here's what I was thinking: train/test split the data,…
Diogo Bastos
  • 39
  • 1
2
votes
1 answer

Should I continue training if the neural network attains 100% training accuracy?

I have a neural network where there are two hidden layers. Each hidden layer has 128 neurons. The input layer has 20 inputs, and the output layer has 3 outputs. I have 1 million records of data. 80% is used to train the network, 20% is used for…
user366312
  • 351
  • 1
  • 12
2
votes
1 answer

How exactly does nested cross-validation work?

I have trouble understanding how nested cross-validation works - I understand the need for two loops (one for selecting the model, and another for training the selected model), but why are they nested? From what I understood, we need to select the…
stoic-santiago
  • 1,121
  • 5
  • 18
2
votes
1 answer

After having selected the best model with cross-validation, for how long should I train it?

When using k-fold cross-validation in a deep learning problem, after you have computed your hyper-parameters, how do you decide how long to train your final model? My understanding is that, after the hyperparameters are selected, you train your…
FourierFlux
  • 783
  • 1
  • 4
  • 14
2
votes
2 answers

Should I choose the model with highest validation accuracy or the model with highest mean of training and validation accuracy?

I'm training a deep network in Keras on some images for a binary classification (I have around 12K images). Once in a while, I collect some false positives and add them to my training sets and re-train for higher accuracy. I split my training into…
Tina J
  • 973
  • 6
  • 13
2
votes
3 answers

How do you interpret this learning curve?

Loss is MSE; orange is validation loss, blue training loss. The task is NN regression (18 inputs, 2 outputs), one layer 300 hidden units. Tuning the lr, mom, l2 regularization parameters this is the best validation loss I can obtain. Can be…
Andrea Favilli
  • 21
  • 2
2
votes
1 answer

Can we say: the more we increase the numbers of cross validation the less likely it is that we overfit?

Based on the answer of my previous question: How can I avoid overfitting when doing parameter tuning? Can we say: the more we increase the numbers K of cross validation the less likely it is that we overfit?
jennifer ruurs
  • 579
  • 2
  • 8
1
2 3 4