How to Build Linear Regression From Scratch Using PyTorch Tensors

There is a particular kind of bug that does not crash anything. Your code runs. The loss goes down. The training log scrolls past looking exactly like every training log you have ever seen. And at the end, the model hands you an answer that is completely, confidently wrong, with no error message anywhere to tell you so. This tutorial walks you into one of those on purpose. You will build linear regression out of nothing but raw numbers, no ready made layers and no ready made loss function, and you will get it working. Then you will change a single character, watch the model learn the wrong thing entirely, and understand exactly why. That second half is worth more than the first.

What Linear Regression Is, and Why It Is the Right Thing to Build First

Linear regression fits a straight line through a cloud of points. That is the whole model. It has exactly two adjustable numbers, a slope and an intercept, which is small enough that you can watch both of them change and reason about whether the change makes sense.

Every neural network is, in a meaningful sense, this same idea repeated and stacked. A single nn.Linear layer is doing precisely what you build below, just with more inputs and outputs. Understanding the two parameter version completely is worth more than half understanding the million parameter version.

Why Build Something PyTorch Already Provides

You could write this model in four lines using nn.Linear and nn.MSELoss, and in real work you should. The reason to build it by hand once is diagnostic rather than practical.

When a model trains but produces poor results, the useful question is which stage is misbehaving. Engineers who have written a forward pass, a loss, and a parameter update themselves tend to reach for shape printing and gradient inspection. Engineers who have only ever assembled prebuilt layers tend to reach for hyperparameter tuning, which is usually the wrong tool and always the slower one.

The Failure This Tutorial Is Really About

There is a category of bug in numerical code that never raises an exception. The program runs, the loss decreases, the log looks ordinary, and the answer is wrong. These are far more expensive than crashes, because a crash tells you where to look and this tells you nothing.

PyTorch has a feature called broadcasting that quietly creates exactly this situation. Broadcasting lets you combine tensors of different shapes by automatically stretching the smaller one to fit. It is genuinely useful, and it is what lets you add one bias value to a hundred predictions without writing a loop.

The cost is that broadcasting will happily combine two shapes that were never meant to meet. A prediction shaped [100] and a target shaped [100, 1] are compatible under its rules, so instead of a shape error you get a [100, 100] grid comparing every prediction against every target. The loss is then an average over ten thousand meaningless pairs, and the only value that minimises it is the average of your data.

Step 1: Generate Data With a Known Answer

Create 100 points along the line y equals 3x plus 2, then add random noise so the fit is not trivially exact.

import torch
torch.manual_seed(42)

X = torch.linspace(0, 10, 100).unsqueeze(1)
y = 3 * X + 2 + torch.randn(X.size()) * 1.5

print("X shape:", X.shape, "| y shape:", y.shape)

X shape: torch.Size([100, 1]) | y shape: torch.Size([100, 1])

torch.linspace(0, 10, 100) produces 100 evenly spaced values and returns shape [100]. The .unsqueeze(1) call adds a second dimension, turning it into [100, 1], which is a column of 100 rows rather than a flat list.

That distinction looks pedantic. It is the entire subject of this tutorial.

Step 2: Create the Parameters to Learn

Linear regression has exactly two parameters: a slope and an intercept. Start both at zero and mark them as things PyTorch should track gradients for.

w = torch.zeros(1, 1, requires_grad=True)
b = torch.zeros(1, requires_grad=True)

requires_grad=True is the instruction that makes autograd record every operation these tensors take part in, so that gradients can be computed later. Tensors without that flag are treated as constant data.

Note the shape of w. It is [1, 1], not [1]. That choice is what keeps the matrix multiplication in the next step producing a column rather than a flat vector, and choosing [1] here is precisely the mistake explored further down.

Step 3: Write the Forward Pass and the Loss

No layers, no loss classes. Just arithmetic.

y_pred = X @ w + b
loss = ((y_pred - y) ** 2).mean()

print("y_pred shape:", y_pred.shape)
print("y shape     :", y.shape)
print("difference  :", (y_pred - y).shape)

y_pred shape: torch.Size([100, 1])
y shape     : torch.Size([100, 1])
difference  : torch.Size([100, 1])

The @ operator is matrix multiplication. Multiplying [100, 1] by [1, 1] gives [100, 1], then adding b broadcasts the single bias value across all 100 rows, which is exactly what you want.

Below that, mean squared error is written out in full: subtract prediction from target, square the result so positive and negative errors both count as error, then average across all examples. That is all nn.MSELoss does.

Those three printed shapes are the assertion that the model is wired correctly. All three read [100, 1]. Get used to checking that.

Step 4: Look at the Gradients Before Training

Before running a loop, look at what backward() actually produces on a single call.

w2 = torch.zeros(1, 1, requires_grad=True)
b2 = torch.zeros(1, requires_grad=True)
loss2 = ((X @ w2 + b2 - y) ** 2).mean()
loss2.backward()

print(f"dL/dw = {w2.grad.item():.4f}")
print(f"dL/db = {b2.grad.item():.4f}")

dL/dw = -221.7562
dL/db = -34.1793

Both gradients are negative, which means the loss decreases if both parameters increase. That matches intuition: starting from w equals 0 and b equals 0, the model predicts zero everywhere while the real data climbs from 2 up to 32, so both parameters need to rise.

The magnitudes also tell you something useful. The gradient on w is roughly six and a half times the gradient on b, because changing the slope moves every prediction proportionally to its x value while changing the intercept shifts everything by the same amount. Larger gradient means the optimizer moves that parameter further per step.

Step 5: Write the Training Loop

Three operations repeated a thousand times.

lr = 0.01

for epoch in range(1, 1001):
    y_pred = X @ w + b
    loss = ((y_pred - y) ** 2).mean()

    loss.backward()
    with torch.no_grad():
        w -= lr * w.grad
        b -= lr * b.grad
    w.grad.zero_()
    b.grad.zero_()

    if epoch == 1 or epoch % 200 == 0:
        print(f"epoch {epoch:4d} | loss {loss.item():8.4f} | w {w.item():.4f} | b {b.item():.4f}")

epoch    1 | loss 370.2874 | w 2.2176 | b 0.3418
epoch  200 | loss   2.2615 | w 3.0827 | b 1.5251
epoch  400 | loss   2.1795 | w 3.0250 | b 1.9089
epoch  600 | loss   2.1683 | w 3.0037 | b 2.0507
epoch  800 | loss   2.1668 | w 2.9958 | b 2.1031
epoch 1000 | loss   2.1666 | w 2.9929 | b 2.1225

The true values were w equals 3 and b equals 2. The model found 2.9929 and 2.1225. The remaining gap is the noise you added, and a perfect recovery would actually indicate the model had memorised the noise rather than found the underlying line.

Three details in that loop deserve attention.

with torch.no_grad() around the update is mandatory. You are modifying tensors that autograd is tracking. Without the context manager PyTorch tries to record the update itself as part of the graph, and raises a leaf Variable that requires grad is being used in an in-place operation.

Zeroing the gradients is not optional. PyTorch adds each new gradient to the existing value rather than replacing it. Skip w.grad.zero_() and by epoch 100 you are stepping on the sum of a hundred gradients.

The learning rate controls step size only. At 0.01 this problem converges comfortably. Raise it to 0.5 and the loss becomes nan within a few epochs as the parameters oscillate outward instead of settling.

Break It On Purpose

This is the part worth remembering. Change one character in the parameter definition and watch the model learn something completely different while giving no indication that anything is wrong.

epoch 2000 | loss 78.2313 | w 0.0001 | b 17.0888

Here is the mismatch made explicit:

w_bad = torch.zeros(1)
w_good = torch.zeros(1, 1)

print("bad :", (X @ w_bad).shape,  "-", y.shape, "->", (X @ w_bad - y).shape)
print("good:", (X @ w_good).shape, "-", y.shape, "->", (X @ w_good - y).shape)

bad : torch.Size([100]) - torch.Size([100, 1]) -> torch.Size([100, 100])
good: torch.Size([100, 1]) - torch.Size([100, 1]) -> torch.Size([100, 1])

Broadcasting is a genuinely useful feature. It is what lets you add a single bias value to 100 predictions without writing a loop. The cost is that it will happily do something you did not intend and never mention it.

The defence is one line, and it belongs in every model you write:

assert y_pred.shape == y.shape, f"shape mismatch: {y_pred.shape} vs {y.shape}"

The Same Model Using PyTorch Layers

Now that the mechanics are clear, here is the version you would actually write, which produces the same result.

from torch import nn

model = nn.Linear(1, 1)
loss_fn = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

for epoch in range(1000):
    loss = loss_fn(model(X), y)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

print(f"w = {model.weight.item():.4f}, b = {model.bias.item():.4f}")

The abstractions are worth using. The point of writing the left column once is that when a model misbehaves, you know what the right column is doing.

Error Decoder

Each message below was produced by running deliberately broken code on PyTorch 2.13.0.

What This Teaches You About Bigger Models

The failure in this tutorial scales badly. In a two parameter model you noticed because you knew the answer. In a network with a hundred million parameters trained on data you have never plotted, a silent broadcasting bug looks like a model that underperforms slightly, and teams spend weeks tuning architecture and learning rates chasing a shape mismatch.

Three habits prevent it entirely.

Print shapes at every boundary while developing. Between data loading and the model, between the model and the loss, and inside custom layers. Delete the prints once it works.

Assert that prediction shape equals target shape. One line, no runtime cost worth mentioning, and it converts a silent wrong answer into a loud failure.

Sanity check on data with a known answer. Before training on real data, run the pipeline on something small where you know what should come out. If the model cannot recover a line you drew yourself, it will not do better on anything harder.

Conclusion

You built a working regression model from raw tensors, recovered parameters within 0.01 of the truth, and then watched the same code fail silently because of a single missing dimension. The second half is the valuable part.

The mechanics are simple enough to hold in your head: a forward pass is arithmetic on tensors, a loss is a single number summarising how wrong that arithmetic was, and training is repeatedly nudging parameters against their gradients. Every framework abstraction is a convenience over those three ideas.

What does not simplify is shape discipline. Broadcasting is a feature that will silently do the wrong thing on your behalf, and no amount of experience fully immunises you. The engineers who lose the least time to it are the ones who print shapes constantly and assert them at boundaries. Add that assert line to your next model and it will eventually save you a day.

FAQS