De-Magicking AI: Writing a Neural Network in Pure C

Most "doing AI" is importing a Python library and calling a function. That's fine, but it never told me why it actually works. So I built a small neural network from scratch in C — no PyTorch, no TensorFlow, just math.h and a lot of pointer bugs.

Why bother

Use a high-level library and you take backprop and gradient descent on faith — it's a black box that just works. Writing it in C means there's no autograd to hide behind: it's just linear algebra and calculus, and you have to do all of it yourself.

What it does

Didn't want a hardcoded XOR toy, so I made it configurable:

• input/hidden/output sizes are all adjustable
• activations: sigmoid, ReLU, leaky ReLU, tanh, linear
• loss: MSE, binary cross-entropy, MAE
• momentum and learning rate decay

The network itself is just a struct full of double pointers for the weight matrices:

typedef struct {
    int input_size;
    int hidden_size;
    int output_size;

    // Weights and biases
    double** w1;
    double* b1;
    double** w2;
    double* b2;

    // Activations
    double* hidden;
    double* output;

    // Gradients & Momentum
    double** dw1;
    double** dw2;
    double momentum;
} NeuralNetwork;

The actual math

Forward pass: weighted sum plus bias, then squash it through an activation like sigmoid.

$$ Z = W \cdot X + b, \qquad A = \sigma(Z) = \frac{1}{1 + e^{-Z}} $$

Backward pass: chain rule, to figure out how much each weight contributed to the error.

$$ \frac{\partial L}{\partial W} = \frac{\partial L}{\partial A} \cdot \frac{\partial A}{\partial Z} \cdot \frac{\partial Z}{\partial W} $$

Then nudge every weight against its gradient, scaled by a learning rate:

$$ W_{new} = W_{old} - \eta \cdot \nabla L $$

In Python that's loss.backward(). In C it's a nested for loop over gradient arrays you allocated yourself.

The actual pain

It wasn't the math, it was memory. No lists, no garbage collector — every row of a dataset is a malloc you own, and every gradient array has to be freed correctly or you leak. Get sloppy and you get a segfault instead of a stack trace with a nice message.

Watching it learn

Built a CLI so I could tweak hyperparameters via flags and watch it train live:

$ ./nn -e 5000 -l 0.1 -h 8 -d xor -ha relu -oa sigmoid -loss bce -wd 0.001 -v

starting training...
epoch 0/5000 | train loss: 0.697071 | test loss: 0.713845
epoch 100: learning rate decayed to 0.099500
...
final results:
  training loss: 0.000590
  test loss: 0.001088
  training accuracy: 100.00% (80/80)
  test accuracy: 100.00% (20/20)

Loss goes from 0.69 to 0.0005, which is basically the network figuring out [0, 1] -> 1 and [0, 0] -> 0 on its own from the data.

Trying something harder

XOR is easy. So I generated points inside/outside a circle with some noise thrown in — not linearly separable, which is the whole point. It struggled at first, then converged once I bumped the hidden layer size and switched to ReLU:

$ ./nn -d circle_enhanced -n 2000 -ha relu -oa sigmoid -loss bce
...
input: [-0.281, -0.996] -> output: 0.4922 (expected: 0.0000) -> class: 0 ✓
input: [-0.192, 0.158]  -> output: 0.4922 (expected: 1.0000) -> class: 0 ✗

Still misses edge cases, which is fine — that's just ML being probabilistic, not a bug in my backprop.

So, is it magic?

No. It's forward pass (dot products + activations), loss (how wrong were we), backward pass (derivatives telling you which way to nudge the weights). Repeat a few thousand times.

Full source is on Codeberg.