For questions related to the validation of a machine learning model, which is different from testing a model, which is done after training. Validation is usually performed for early stopping (i.e. assess when the model is over-fitting) or hyper-parameter optimization during training. However, often people use (either correctly/intensionally or not) the terms "validation" and "testing" interchangeably, so context needs to be taken into account.
Questions tagged [validation]
8 questions
10
votes
3 answers
Should I choose a model with the smallest loss or highest accuracy?
I have two Machine Learning models (I use LSTM) that have a different result on the validation set (~100 samples data):
Model A: Accuracy: ~91%, Loss: ~0.01
Model B: Accuracy: ~83%, Loss: ~0.003
The size and the speed of both models are almost the…
malioboro
- 2,729
- 3
- 20
- 46
2
votes
0 answers
Which evaluation metrics should be used in training, validation and testing of a model?
Which specific performance evaluation metrics are used in training, validation, and testing, and why? I am thinking error metrics (RMSE, MAE, MSE) are used in validation, and testing should use a wide variety of metrics? I don't think performance is…
user9645302
- 53
- 3
1
vote
0 answers
Why is my validation accuracy fluctuating between two inverse values?
I am currently going through the FastAI course and to practise, I wanted to code a neural network that classifies the FashionMNIST dataset from scratch.
Lately, I've been running into an issue where I get a consistent validation accuracy score of…
DerOeko
- 13
- 3
0
votes
0 answers
If not chosen all the data in the train partition, is it still k-fold cross validation?
I have a dataset of 900 images, distributed across 6 classes, with 150 images per class. To develop a classifier and assess its performance, I will utilize k-fold cross-validation. In this case, I will employ 3-fold cross-validation.
For each fold,…
noone
- 123
- 4
0
votes
0 answers
Very high ACC (ca. 95%) with 1DConvNet for Time Series
Does this sound legit, for people working with CNN and Time Series?
I have a Framework that applies Dynamic Tim Warping (DTW) on time series, using the DTW distance matrix, I cluster my data and create labels for each time series and then apply a…
Skobo Do
- 51
- 6
0
votes
0 answers
What are the standard ways to measure the quality of a set of numerical predictions that include uncertainties?
I have a radial basis function that supplies uncertainties (standard deviations) with its predictions, which are numerical values.
This function is computed for a particular point by computing its relative distance to a large set of other reference…
PJ7
- 101
- 1
0
votes
0 answers
Why would my neural network have either an accuracy of 90% or 10% on the validation data, given a random initialization?
I'm making a custom neural network framework (in C++, if that is of any help). When I train the model on MNIST, depending on how happy the network is feeling, it'll give me either 90%+ accuracy, or get stuck at 10-9% (on validation set).
I shuffle…
Ilknur Mustafa
- 115
- 4
-1
votes
1 answer
how to decide the optimum model?
I have split the database available into 70% training, 15% validation, and 15% test, using holdout validation. I have trained the model and got the following results: training accuracy 100%, validation accuracy 97.83%, test accuracy 96.74%
In…
user50778
- 1
- 1