0

I have a radial basis function that supplies uncertainties (standard deviations) with its predictions, which are numerical values.

This function is computed for a particular point by computing its relative distance to a large set of other reference points in high dimensional space, and compositing a prediction from them.

Over the training set I can compute R to get a correlation between prediction and actual. Weights are assigned to each dimension and optimized to maximize R.

Over a validation set, it seems I'd want to calculate something other than R to measure the model's predictive power, since its predictions are not single values, but ranges.

PJ7
  • 101
  • 1
  • Hello. What do you mean by "agreement between a validation data set, and predictions for that set that include uncertainty"? Please, explain what you mean by "agreement" here and "set that include uncertainty". Also, what is your ultimate goal? What are you trying to achieve? – nbro Jun 24 '21 at 16:05
  • Maybe you mean Accuracy, Precision, Recall & F1 Score? – Aray Karjauv Jun 24 '21 at 18:25
  • Huh. You're saying I could make a binary classification (false positives / false negatives) from whether or not the prediction error falls outside a certain multiple of the predicted standard deviation? Then I pick the confidence (say, two sigma) and evaluate the usefulness of the model that way? That sounds cool. Think I'll try that. – PJ7 Jun 24 '21 at 18:42
  • @PJ7 When you talk about uncertainties, it sounds like Bayesian neural networks to me, which are different from regular neural networks. – Aray Karjauv Jun 24 '21 at 18:58
  • We're using radial basis functions. – PJ7 Jun 24 '21 at 18:59
  • @PJ7 It might make sense if you add more details about it in your question. ps: don't forget to add @ to your comments to notify users – Aray Karjauv Jun 24 '21 at 19:12
  • Can you please explain how you're using "radial basis function" as a machine learning algorithm/model/function that provides uncertainties? Could you provide a reference to read more about the topic? I've never seen "radial basis functions" being used for this purpose, so I am trying to understand how they would be used for that. – nbro Jun 24 '21 at 22:10
  • 1
    @nbro I have to walk the line between trying to seek help for stuff out of my expertise, and respecting an NDA. Supposedly it's a novel AI design though nobody really publishes the valuable stuff. I'll try to add more to the description while honoring my commitments. – PJ7 Jun 24 '21 at 22:14
  • Ok, but given a prediction for a point with the RBF, how exactly do you compute an "uncertainty" for that prediction? – nbro Jun 24 '21 at 22:40
  • @nrbo It's a standard deviation. – PJ7 Jun 24 '21 at 23:45

0 Answers0