6

I am solving many sequence-to-sequence prediction problems using RNN/LSTM.

What type of evaluation metrics can be used for sequence prediction problems?

One metric is the mean squared error (MSE) that we can give as a parameter during the training model. Currently, the accuracy of my sequence-to-sequence problems is very low.

What are other ways through which we can compare the performance of our models?

nbro
  • 39,006
  • 12
  • 98
  • 176
Asif Khan
  • 181
  • 1
  • 6
  • 3
    It depends on what type of data you're having and what kind of results you're interested in. For example you could use root mean square error (RMSE) which will greatly penalize large errors. If you want your results to be easily interpreted you could use mean absolute percentage error. – razvanc92 Nov 14 '19 at 07:37

2 Answers2

0

I would recommend taking a look at Bilingual Evaluation Understudy(BELU) score which is commonly used in evaluating machine translation results by sequence to sequence model. Here is the reference https://en.wikipedia.org/wiki/BLEU

Srivatsan Ramesh
  • 69
  • 1
  • 4
0

This really depends on your data. MSE and its variant, the RMSE, are good for regression problems. In other words, when you want to produce a real number as an output, for example, in a time series forecasting situation. The MAPE is good and can be interpreted as a percentage, but it does not work well if you have zeros in your dataset. As the other answer mentions, BLEU is good too, but only if your model is doing machine translation, or working with some other type of categorical data; however, you would not use BLEU if you are trying to predict house prices, for example.

To find the best evaluation metric you have to consider your data and the problem you are trying to solve. Its good to go over a list of potential loss functions and see what fits your problem best.

Marcus
  • 206
  • 1
  • 7