Creating an automatic differentiation system from scratch is an excellent way to grasp how deep learning frameworks compute gradients. In this guide, we’ll build a simple scalar-based autograd engine similar to micrograd (by Andrej Karpathy), which focuses on the core mechanics without the complexity of multi-dimensional tensors.

Core Concept: The Computational Graph

Autograd works by building a Directed Acyclic Graph (DAG) during the “forward pass.”

• Nodes: Represent operands (values/scalars).
• Edges: Represent the operations that connect them.
• Backpropagation: We traverse this graph in reverse to apply the Chain Rule.

Step 1: The Value Class

This class will store a scalar value, its gradient, and a reference to the operations that created it.

python import math

class Value: def init(self, data, _children=(), _op=”): self.data = data self.grad = 0.0 # Hidden state: d(Result)/d(self) self._prev = set(_children) # Previous nodes in the graph self._op = _op # The operation that produced this node self._backward = lambda: None # Function to compute local gradient

def __repr__(self):
    return f"Value(data={self.data}, grad={self.grad})"

# Addition: c = a + b
def __add__(self, other):
    other = other if isinstance(other, Value) else Value(other)
    out = Value(self.data + other.data, (self, other), '+')

    def _backward():
        # Chain rule: dL/da = dL/dc * dc/da. For addition, dc/da = 1.
        self.grad += 1.0 * out.grad
        other.grad += 1.0 * out.grad
    out._backward = _backward

    return out

# Multiplication: c = a * b
def __mul__(self, other):
    other = other if isinstance(other, Value) else Value(other)
    out = Value(self.data * other.data, (self, other), '*')

    def _backward():
        # Chain rule: dL/da = dL/dc * dc/da. For multiplication, dc/da = b.
        self.grad += other.data * out.grad
        other.grad += self.data * out.grad
    out._backward = _backward

    return out

# Power: c = a^n
def __pow__(self, other):
    assert isinstance(other, (int, float)), "Supporting only int/float powers"
    out = Value(self.data**other, (self,), f'^{other}')

    def _backward():
        # Power rule: d(x^n)/dx = n * x^(n-1)
        self.grad += (other * (self.data**(other-1))) * out.grad
    out._backward = _backward

    return out

def relu(self):
    out = Value(0 if self.data < 0 else self.data, (self,), 'ReLU')

    def _backward():
        self.grad += (1.0 if out.data > 0 else 0) * out.grad
    out._backward = _backward
    return out

def backward(self):
    """Topological sort to ensure we compute gradients in the right order."""
    topo = []
    visited = set()
    def build_topo(v):
        if v not in visited:
            visited.add(v)
            for child in v._prev:
                build_topo(child)
            topo.append(v)

    build_topo(self)

    # Initial gradient is 1 (dOut/dOut)
    self.grad = 1.0
    for node in reversed(topo):
        node._backward()

# Boilerplate for operator overloading
def __neg__(self): return self * -1
def __sub__(self, other): return self + (-other)
def __rmul__(self, other): return self * other
def __truediv__(self, other): return self * (other**-1)

Step 2: Testing the Engine

Let’s see if we can calculate the gradient of a simple function: $f(x, y) = (x \cdot y) + y^2$.

python x = Value(2.0) y = Value(3.0)

f = (x * y) + y^2

f = (2 * 3) + 3^2 = 6 + 9 = 15

f = (x * y) + (y ** 2)

f.backward()

print(f”Result: {f.data}”) # Should be 15.0 print(f”df/dx: {x.grad}”) # df/dx = y = 3.0 print(f”df/dy: {y.grad}”) # df/dy = x + 2y = 2 + 6 = 8.0

Step 3: Minimal Neural Network Logic

Now that we have the autograd engine, we can build a Neuron and Layer.

python import random

class Neuron: def init(self, nin): self.w = [Value(random.uniform(-1, 1)) for _ in range(nin)] self.b = Value(random.uniform(-1, 1))

def __call__(self, x):
    # w * x + b
    act = sum((wi*xi for wi, xi in zip(self.w, x)), self.b)
    return act.relu()

def parameters(self):
    return self.w + [self.b]

Example Usage

n = Neuron(2) x = [Value(1.0), Value(-2.0)] y = n(x) y.backward()

print(f”Neuron Output: {y.data}”) print(f”Gradient of first weight: {n.w[0].grad}”)

Key Takeaways for Your Implementation

1. Gradient Accumulation (+=): We use += for self.grad because if a variable is used multiple times in a graph, its gradients must be summed (the Multivariate Chain Rule).
2. Topological Sort: In the backward() method, you cannot calculate a node’s gradient until you have processed all nodes that depend on it. A topological sort ensures we move from the output back to the inputs correctly.
3. Operator Overloading: Overloading __add__ and __mul__ allows you to write natural Python math code (a + b) while the Value class secretly builds the graph in the background.
4. Zero Grad: In a real training loop, you must call grad = 0 between iterations, otherwise gradients from the previous step will keep adding to the new ones.

This architecture is the fundamental “DNA” of PyTorch’s Variable and Function classes, scaled up to handle N-dimensional tensors and GPU acceleration.