1

In natural language processing, I came across the concept of "relevant document" several times. And several analytical formulas, such as precision, recall are based on the relevant documents.

Precision = $\dfrac{\text{Number of documents that are relevant and retrieved to the query Q}}{\text{Number of retrieved documents to the query Q}}$

Recall = $\dfrac{\text{Number of documents that are relevant and retrieved to the query Q}}{\text{Number of relevant documents to the query Q}}$

What is meant by "relevance" in such cases? Is it a universally objective term or subjective term, decided by the designer, based on that particular context?

nbro
  • 39,006
  • 12
  • 98
  • 176
hanugm
  • 3,571
  • 3
  • 18
  • 50

2 Answers2

1

It means precisely the same as true/false positive and true/false negative in the classic formulation of precision, recall, F-score for classification tasks.

  • relevant and retrieved: true positive
  • relevant and not retrieved: false positive
  • not relevant and retrieved: false negative
  • not relevant and not retrieved: true negative

And yes, the relevance depends on the task you're performing. To make an example, you might be interested in classifying, through sentiment analysis, offensive vs nonoffensive tweets. In this case, you'll have:

  • relevant and retrieved: offensive tweets labeled as such
  • relevant and not retrieved: offensive tweets labeled as nonoffensive
  • not relevant and retrieved: nonoffensive tweets labeled as offensive
  • not relevant and not retrieved: nonoffensive tweets labeled as such
nbro
  • 39,006
  • 12
  • 98
  • 176
Edoardo Guerriero
  • 5,153
  • 1
  • 11
  • 25
1

Precision and Recall are concepts that have been introduced in the field of information retrieval. Imagine you have a large set of documents, and you want to find the ones that are relevant to a particular issue.

You can be sure to find all relevant documents if you simply return the whole lot -- you won't miss a single relevant document. So your recall is 1.0, the maximum: you got all the relevant ones. That you also got a large number of irrelevant ones is not important for recall, but means your query wasn't very efficient (and actually quite pointless!).

If you do get all documents, your precision will be low: the number of retrieved documents that are relevant is the same, but the denominator is now much bigger, and depending on how many relevant documents there are, your precision value is small.

The opposite extreme is to not return any documents: Now your precision is high (effectively infinite, as it is zero divided by zero), but your recall is zero (as you don't get any relevant documents).

So in an ideal world, you want to optimise both precision and recall (which is why usually a third metric is used which combines the two, the F-Score.

Now, you can only calculate precision and recall if you know what the correct values are already, so you need to know how many documents are relevant. Obviously you will need to have inspected all documents to decide whether they are relevant to your issue or not. This is usually a subjective value judgment, as relevance tends to be non-binary. If I'm looking for articles on the efficiency of petrol engines, will those about Diesel engines be relevant? Probably, if you're looking at a general dataset. But if the documents are all about the efficiency of different types of engines, then I would most likely not be interested in those about Diesel engines.

From Information Retrieval, the concepts of precision and recall have been generalised to any binary classification tasks, where you can judge the quality of the classification by using these metrics. So you might come across situations where the 'relevance' criterion is more objective than whether a document is relevant to a certain topic. But that depends on the context of what you are classifying.

Oliver Mason
  • 5,322
  • 12
  • 32