How is direction of weight change determined by Gradient Descent algorithm

Question

The result of gradient descent algorithm is a vector. So how does this algorithm decide the direction for weight change? We Give hyperparameters for step size. But how is the vector direction for weight change, for the purpose of reducing the Loss function in a Linear Regression Model, determined by this algorithm?

Answer 1

First thing is that what does gradient descent do? Gradient Descent is a tool of calculus which we use to determine the parameters (here weights) used in a Machine Learning algorithm or a Neural Network, by running the Gradient Descent algorithm iteratively.

What does the vector obtained from one iteration of gradient descent tell us? It tells us the direction of weight change (when weights are treated as a vector) for the maximum reduction in the value outputted by a loss function. The intuition behind why Gradient Descent gives such a direction can be found here:

• Why the gradient is the direction of steepest ascent - Khan Academy
• Why is gradient the direction of steepest ascent?
• Why does gradient descent work?

In general cases the cost/loss function is an n-dimensional paraboloid (we design function in such a way such that its convex).

A 2-D paraboloid with x and y as independent variables.

Now why do we reduce the weights in the direction of negative gradient only? Why not some other direction?

Since we want to reduce the cost in Linear Regression for better predictions we choose only the negative Gradient Descent direction, as it is the direction of steepest descent even though we could have chosen some other possible vector direction to reduce cost, but the negative gradient descent direction ensures that:

• Cost is always decreased (if the step size is correct and in general only in cases of convex functions e.g paraboloid).
• Cost is decreased by the maximal amount if we move in that direction, so we don't have to worry whether cost will decrease or not if we move in that direction.

Also we use learning rate alpha to scale by how much amount we want to decrease the weights.

EDIT: As pointed by @pasaba the error function may not be a paraboloid but in general a good cost function looks like a paraboloid with skewed axis.

Answer 2

We can analyze a basic common example: approximation of AND logic gate by a NN.

The inputs to the NN will be "x1" and "x2" and its output is "y". The data to be learned by the NN is:

The basic NN has one intermediate cell with activation function sigmoid and one output cell identity function. That means:

(note by symmetry we assume w1=w2)

Then, the error is:

These are some plots of this function:

And this is the w derivate of e :

Sage Math code (link) for this graph:

s(x)=1/(1+e^(-1*x))
e(w,b)=(1/2)*(s(b)^2+2*s(w+b)^2+(1-s(2*w+b))^2)
plot3d(e(w,b),(w,-5,5),(b,-5,5),adaptive=True, color=rainbow(60, 'rgbtuple'))

edw=e.derivative(w)
plot3d(edw(w,b),(w,-5,5),(b,-5,5),adaptive=True, color=rainbow(60, 'rgbtuple'))

(PS: please, activate latex on this stack exchange).