2

Which specific performance evaluation metrics are used in training, validation, and testing, and why? I am thinking error metrics (RMSE, MAE, MSE) are used in validation, and testing should use a wide variety of metrics? I don't think performance is evaluated in training, but not 100% sure.

Specifically, I am actually after deciding when to use (i.e. in training, validation, or testing) correlation coefficient, RMSE, MAE and others for numeric data (e.g. Willmott's Index of Agreement, Nash-Sutcliffe coefficient, etc.)

Sorry about this being broad - I have actually been asked to define it generally (i.e. not for a specific dataset). But datasets I have been using have all numeric continuous values with supervised learning situations.

Generally, I am using performance evaluation for environmental data where I am using ANNs. I have continuous features and am predicting a continuous variable.

nbro
  • 39,006
  • 12
  • 98
  • 176
user9645302
  • 53
  • 3
  • Welcome to ai.se...it indeed is too broad a question...choice of evaluation metrics solely depend on the problem and requirements at hand..also it depends whether you are using it to train the network...F-score is a measurement metric not used to train network..so I suggest you to narrow down the scope by editing the question a little bit and giving it some context –  Apr 14 '18 at 14:01
  • Thanks for that. Sorry about that - I will try to explain it more. I am actually after deciding when to use specifically correlation coefficient, RMSE, MAE and others for numeric data (e.g. Willmott's Index of Agreement, Nash-Sutcliffe coefficient, etc.) – user9645302 Apr 14 '18 at 14:04
  • Since I actually need to explore all ways performance evaluation is used in training, validation and testing, I would be happy to know how it used to train the network. Thank you – user9645302 Apr 14 '18 at 14:11
  • Actually there is so many number of optimizations you can't possibly track them all...like i said any1 can invent a new metric according to his choice...so it is quite difficult to answer your question unless you provide some more info..also you are asking some very mathematical questions whose details might not fit in a single answer –  Apr 15 '18 at 09:32
  • @DuttaA Thank you for your reply and sorry that this is not clear. Actually in my course I am doing it is all about using ANN to model data. The question I have been asked to answer is about discussing which stages out of training, validation and testing should the performance metrics correlation coefficient, RMSE, MAE, Willmott's Index, Nash-Sutcliffe coefficient, Legate and McCabe's Index and Percentage Peak Deviation be used in? We haven't been given any more information or data. I think what I would like to know is probably when do we use error metrics and when to use non-error metrics? – user9645302 Apr 15 '18 at 11:56

0 Answers0