-1

I have split the database available into 70% training, 15% validation, and 15% test, using holdout validation. I have trained the model and got the following results: training accuracy 100%, validation accuracy 97.83%, test accuracy 96.74%

In another trial for training the model, I got the following results: Training accuracy 100%, validation accuracy 97.61%, test accuracy 98.91%

The same data split is used in each run. Which model should I choose, the first case in which the the test accuracy is lower than the validation? or the second case in which the test is higher than the validation?

Noha
  • 111
  • 2
user50778
  • 1
  • 1
  • Do you have the same data split for each run - i.e. exactly the same examples in train, validation and test sets? Also, can you give size of each set? Are your numbers *exact* or have you simplified them? There would be a difference in analysis if your values were 98.3% vs 98.7% instead of 97.6% vs 99.2% for instance. What matters is the ratio of error rates - whilst 98% vs 99% appears as a ratio of 2, if you have rounded nearest then the ratio could be anything from 1.1 (not really meaningful) to 5.0 (impressive). – Neil Slater Nov 04 '21 at 21:29

1 Answers1

1

Testing each time on a test set is against the point of a train-val-test split. The reason test is important, is that you are only supposed to test on it when you think your model is good and ready with all final model and hyperparameter decisions made.

A good description can be found in this article: https://machinelearningmastery.com/difference-test-validation-datasets/

but to sum it up, test should be unbiased. The more you test against it, the more you bias the result.

mshlis
  • 2,349
  • 7
  • 23
  • More techincally for the last paragraph, the testing does not create bias, but the *selecting* does. If OP were to select a model using test, they could no longer trust the measurement, especially for really close values as in their question. I have other concerns with the question too though, where I have asked for clarification from the OP. – Neil Slater Nov 05 '21 at 14:30