Gradient Descent and Backpropagation in Python

The concept in plain words

Training means adjusting the weights to reduce the loss. The gradient shows which direction makes the loss grow, so gradient descent takes small steps the opposite way. Backpropagation uses the chain rule to compute that gradient for every weight.

Train a neuron with gradient descent: use the chain rule to find how each weight affects the loss, then step downhill.

Key ideas

The gradient is the slope of the loss with respect to each weight.
The learning rate sets the step size: too big overshoots, too small is slow.
Backpropagation applies the chain rule from the output back to the first layer.
One epoch is one pass over the whole training data.

When should you use it?

Understand what happens inside neural networks and AI libraries.
Learn the idea before you implement it in the matching challenges.
Prepare for machine learning courses, interviews and projects.

See it in action

Press Play to train. The line moves toward the points while the orange path walks downhill on the loss landscape to its lowest point.

Example code

Run code →

main.py

# Train one neuron y = w * x + b to fit points on the line y = 2x + 1.
data = [(0, 1), (1, 3), (2, 5), (3, 7)]
w, b = 0.0, 0.0
learning_rate = 0.05

for epoch in range(1, 201):
    grad_w = grad_b = 0.0
    for x, y in data:
        error = (w * x + b) - y          # prediction minus target
        # Chain rule: d(error²)/dw = 2 * error * x, d(error²)/db = 2 * error
        grad_w += 2 * error * x
        grad_b += 2 * error
    # Step against the gradient to make the loss smaller.
    w -= learning_rate * grad_w / len(data)
    b -= learning_rate * grad_b / len(data)
    if epoch in (1, 10, 50, 200):
        loss = sum(((w * x + b) - y) ** 2 for x, y in data) / len(data)
        print(f"epoch {epoch:>3}: w={w:.3f} b={b:.3f} loss={loss:.4f}")

Expected output

epoch   1: w=0.850 b=0.400 loss=7.0587
epoch  10: w=2.009 b=0.958 loss=0.0009
epoch  50: w=2.005 b=0.989 loss=0.0000
epoch 200: w=2.000 b=1.000 loss=0.0000

How it works

For each point the error is the prediction minus the target. The chain rule gives the gradient of the squared error, 2·error·x for w and 2·error for b. Averaging over the data and stepping against the gradient moves w and b toward 2 and 1, and the loss drops to almost 0.

Change the values and run the program in the CodeUtility online Python compiler without installing Python locally.

Practice exercises

Modify the runnable example with the exercises below to build understanding beyond copying the result.

Change the inputs or weights and predict the output before running.
Print the intermediate values to follow each step of the calculation.
Solve the linked challenges below to implement the idea yourself.

Run in Python IDE →

Practice with a challenge

Use what this lesson covers to solve a real problem. Each challenge runs in your browser with instant tests and an AI review.

All challenges