A Character-Level RNN Without Autograd
A NumPy-only character language model that implements the recurrent forward pass, BPTT, gradient checking, Adam, checkpointing, and controlled sampling by hand.
- Python
- NumPy
- RNN
- BPTT
- Adam
- Pytest
Implements and numerically verifies the complete recurrent training path without PyTorch, TensorFlow, or autograd, then records measured validation loss and generated text.
Contents
The Problem
What does a recurrent language model actually compute when a framework is not hiding the state transitions or gradients? This project predicts the next character in Tiny Shakespeare and implements the entire training stack in NumPy.
Why This Approach?
A vanilla RNN is limited, but its simplicity makes recurrence inspectable. At each time step,
followed by a vocabulary projection and softmax. Reusing across time makes the model compact, but it also means gradients repeatedly multiply through the same recurrent dynamics.
Architecture
The system includes deterministic character vocabulary construction, one-hot inputs, a tanh recurrent state, a softmax output, truncated BPTT across adjacent windows, validation tracking, and temperature-controlled autoregressive sampling.
Implementation
The backward pass accumulates parameter gradients over every unrolled step because each recurrent weight participates at every position. Numerical gradient checking tests the analytical implementation. Gradient clipping limits exploding updates, while both SGD and Adam are available for optimization.
Checkpoint files include model parameters and metadata, allowing sampling without the original text file for current-format checkpoints. The repository reports 26 passing tests across preprocessing, recurrence, gradients, sampling, and training.
Experiments and Results
Random guessing over the 65-character vocabulary begins near loss per character. A 128-unit run reached 1.7915 training loss and 2.1922 validation loss at sequence length 30. Continuing with sequence length 50 reached 1.7158 training loss and 2.1348 validation loss.
At temperature 0.8, the published checkpoint produces text with learned dialogue formatting, punctuation rhythm, and word-like forms, but not consistently coherent language. That qualitative limitation agrees with the validation gap.
What Failed?
The model learns local texture more reliably than long-range structure. Clipping controls exploding gradients but does not create a durable memory path; repeated Jacobian products can still erase distant information. Generated words often look plausible while sentence-level meaning breaks down.
What I Learned
The evidence shows why gradient checking and validation tracking belong together. The former establishes that the implementation matches the intended derivatives; the latter shows whether those correct derivatives produce generalizable behavior.
Next Steps
- Compare sequence lengths and optimizers under matched budgets.
- Add an LSTM using the same interfaces for a controlled memory comparison.
- Record multiple seeds and confidence intervals rather than one continuation run.