What happens when the output length in the brevity penalty is zero?

Question

The brevity penalty is defined as

$$bp = e^{(1- r/c)},$$

where $r$ is the reference length and $c$ is the output length.

But what happens if the output length gets zero? Is there any standard way of coping with that issue?

Answer 1

Division by zero is not mathematically defined.

A usual or standard way of dealing with this issue is to raise an exception. For example, in Python, the exception ZeroDivisionError is raised at runtime if you happen to divide by zero.

If you execute the following program

zero = 0
numerator = 10
numerator / zero

You will get

Traceback (most recent call last):
  File "main.py", line 3, in <module>
    numerator / zero
ZeroDivisionError: division by zero

However, if you want to avoid this runtime exception, you can check for division by zero and deal with this issue in a way that is appropriate for your program (without needing to terminate it).

In the paper BLEU: a Method for Automatic Evaluation of Machine Translation that introduced the BLEU (and brevity penalty) metric, the authors defined the brevity penalty as

\begin{align} BP = \begin{cases} 1, & \text{if } c > r\\ e^{(1- r/c)} & \text{if } c \leq r\\ \end{cases} \label{1} \tag{1} \end{align}

This definition does not explicitly take into account the division by zero.

The Python package nltk does not raise an exception, but it (apparently, arbitrarily) returns zero when c == 0. Note that the BLEU metric ranges from 0 to 1. For example, if you execute the following program

from nltk.translate.bleu_score import brevity_penalty, closest_ref_length

reference1 = list("hello") # A human reference translation. 
references = [reference1] # You could have more than one human reference translation.

# references = [] Without a reference, you will get a ValueError.

candidate = list() # The machine translation.
c = len(candidate)

r =  closest_ref_length(references, c)
print("brevity_penalty =", brevity_penalty(r, c))

You will get

brevity_penalty = 0

In the example above, the only human reference (translation) is reference1 = list("hello") and the only candidate (the machine translation) is an empty list. However, if references = [] (you have no references), then you will get the error ValueError: min() arg is an empty sequence, where references are used to look for the closest reference (the closest human translation) to the candidate (the machine translation), given that there could be more than one human reference translation, and one needs to be chosen to compute the brevity penalty, with respect to your given candidate.

In fact, in the documentation of the brevity_penalty function, the following comment is written

# If hypothesis is empty, brevity penalty = 0 should result in BLEU = 0.0.

where hypothesis is a synonym for candidate (the machine translation) and the length of the candidate is $c$ in the formula \ref{1} (and c in the example above).

To answer your second question more directly, I don't think there's a standard way of dealing with the issue, but I've not fully read the BLEU paper yet.