Which function $(\hat{y} - y)^2$ or $(y - \hat{y})^2$ should I use to compute the gradient?

Question

The MSE can be defined as $(\hat{y} - y)^2$, which should be equal to $(y - \hat{y})^2$, but I think their derivative is different, so I am confused of what derivative will I use for computing my gradient. Can someone explain for me what term to use?

Answer 1

The derivative of $\mathcal{L_1}(y, x) = (\hat{y} - y)^2 = (f(x) - y)^2$ with respect to $\hat{y}$, where $f$ is the model and $\hat{y} = f(x)$ is the output of the model, is

\begin{align} \frac{d}{d \hat{y}} \mathcal{L_1} &= \frac{d}{d \hat{y}} (\hat{y} - y)^2 \\ &= 2(\hat{y} - y) \frac{d}{d \hat{y}} (\hat{y} - y) \\ &= 2(\hat{y} - y) (1) \\ &= 2(\hat{y} - y) \end{align}

The derivative of $\mathcal{L_2}(y, x) = (y - \hat{y})^2 = (y - f(x))^2$ w.r.t $\hat{y}$ is

\begin{align} \frac{d}{d \hat{y}} \mathcal{L_2} &= \frac{d}{d \hat{y}} (y - \hat{y})^2 \\ &= 2(y -\hat{y}) \frac{d}{d \hat{y}} (y -\hat{y}) \\ &= 2(y - \hat{y})(-1)\\ &= -2(y - \hat{y})\\ &= 2(\hat{y} - y) \end{align}

So, the derivatives of $\mathcal{L_1}$ and $\mathcal{L_2}$ are the same.

Answer 2

The MSE can be defined as $(\hat{y} - y)^2$, which should be equivalent to $(y - \hat{y})^2$

They are not just "equivalent". It is actually the exact same function, with two different ways to write it.

$$(\hat{y} - y)^2 = (\hat{y} - y)(\hat{y} - y) = \hat{y}^2 -2\hat{y}y + y^2$$

$$(y - \hat{y})^2 = (y -\hat{y})(y - \hat{y}) = y^2 -2y\hat{y} + \hat{y}^2$$

These are exactly the same function. Not just "equivalent" or "equivalent everywhere", but actually the same function. It is therefore no surprise that any derivative is also the same - including the partial derivative with respect to $\hat{y}$ which is what you typically use to drive gradient descent.

The two ways of writing the function is because it is a square and thus has two factorisations. When you write it as a square you can choose which form to use for the inner term.

Which function [form] should I use to compute the gradient?

You can use either form, it does not matter. They represent the same function and have the same gradient.

Answer 3

The derivative is the same as far as I understand it.

If $y$ is constant and $\hat{y}$ is the variable the result will be:
$((\hat{y} - y)^2)' = 2(\hat{y} - y)$
and for the other formula:
$((y - \hat{y})^2)' = -2(y - \hat{y})$
which is the same.