0

I have a binary classification problem.

My neural network is getting between 10% and 45% accuracy on the validation set and 80% on the training set. Now, if I have a 10% accuracy and I just take the opposite of the predicted class, I will get 90% accuracy.

I am going to add a KNN module that shuts down that process if the inputted data is, or is very similar to the data present in the data set.

Would this be a valid approach for my project (which is going to go on my resume)?

nbro
  • 39,006
  • 12
  • 98
  • 176
jr123456jr987654321
  • 235
  • 1
  • 7
  • 1
    At a guess without more information, this sounds like a relatively severe case of overfitting, or perhaps the test data is distributed rather differently to the training data. Although in principle what you're saying would work, it sounds like a bit of a hack rather than a genuine solution to the problem. Perhaps focus on trying to figure out why you're getting such a dramatic discrepancy? We might be able to help here too. – htl Mar 24 '21 at 11:40
  • 1
    @htl ok sure I have been working on it for weeks and nothing works. If u wanna find the source code check out this link https://github.com/Julius-R-UI/Heart-Failure-Predictions-Model – jr123456jr987654321 Mar 24 '21 at 12:56
  • Investors do this. They call it 'contrarian trading'. George Costanza, on the Seinfeld show, did this quite successfully for many decisions in his life: https://www.youtube.com/watch?v=cKUvKE3bQlY – Brian O'Donnell Mar 24 '21 at 23:12
  • The part "KNN module" is not clear to me. What do you mean by that sentence? Do you mean that you will use the k-nearest neighbour to compute what exactly? – nbro Mar 26 '21 at 17:28

2 Answers2

2

The short answer is no, you shouldn't do that.

There is a "distribution shift" thing when you have different x-y relation on the validation set then on the train set. The distribution shift would deteriorate your model performance and you should try to avoid that. The reason it's bad - ok, you find the way to fix the model for validation data, but what about novel test data? Will it be like a train? Will it be like validation? You don't know and your model is worthless in fact.

What you can do

  1. You should redefine the train/validation scheme. I would recommend making cross-validation with at least three splits
  2. As mentioned in a comment, your model seems to overfit; try to apply some regularization techniques, like dropout, batch norm, data augmentation, model simplification and so on
  3. Add some data if it's possible, that would better cover all the possible cases.
Faizy
  • 1,074
  • 1
  • 6
  • 30
Kirill Fedyanin
  • 549
  • 2
  • 13
0

I'm new to all this so take what I say with a grain of salt and not as fact, I don't have any formal education or training. I believe when you're referring to inversion predictions, you're not overthinking you're underthinking. For anything to have value it must also have an inverse or else there's no way to cognitively perceive it (contrast) otherwise you're looking at a white paper against white paper. Now since you're referring to data set prediction, you need to define it linearly (x, y) or f(x) to scale and plot. Therefore x and y BOTH must retain inverse proportionate values (I made up that term) in order to, in the context of assigning value, exist. So you need to have 4 quadrants of data for predictive, so now you're looking at quantum data processing in order to facilitate predictions in a non-linear context. Use a matrix, I believe Diroches Matrix should be applicable here. Also, remember that predictions are always changing and updating based on empirical and real-time data, so don't get your programming stuck in an ONLY RIGHT NOW mindset, matrices are designed to be constantly moving and evolving. Therefore your z-axis should always retain a state of variability, or it should always be Z don't attach a value to it. Good luck. I'm jealous I would love to ACTUALLY be working on something cool :/

Faizy
  • 1,074
  • 1
  • 6
  • 30
Bobby Letendre
  • 9
  • 1