Scratch Is All You Need: Building and Training a Transformer in Block Code

Most people learn transformers through high-level abstractions: torch.nn.MultiheadAttention, HuggingFace AutoModelForCausalLM, or a 50-line PyTorch script. These abstractions are convenient, but they hide the fundamental mechanical realities: memory layout, activation caching, stride arithmetic, arithmetic intensity, and the exact dataflow of the backward pass.

To test how minimal a substrate can be while still executing an end-to-end autoregressive language model, I built Scratch Is All You Need: a complete 29,665-parameter decoder-only transformer—with hand-derived backpropagation and AdamW optimization—implemented entirely in MIT Scratch 3.0 blocks.

No PyTorch. No C extensions. Just visual blocks and lists.

---

1. The Substrate Constraints

MIT Scratch was designed for teaching children to animate sprites and build basic 2D games. It is arguably the most hostile execution environment imaginable for training neural networks:

• No multidimensional arrays: There are no matrices or tensors. The only data structures are single, 1D flat lists with a hard cap of 200,000 elements.
• 1-indexed arrays: Array indexing starts at 1, making standard 0-indexed modular arithmetic error-prone without explicit stride calculation.
• No return values or local scopes: Custom procedures cannot return values or declare local variables. Intermediate state and register variables must be manually allocated and managed in a shared scratchpad.
• No power or exponent operators: Scratch lacks pow(x, y) or exp(x). Math is restricted to +, -, *, /, sqrt, sin, cos, and ln.
• Severe execution overhead: Vanilla Scratch runs in an interpreted JavaScript AST loop yielding to the browser frame budget, capping execution throughput unless executed under strict compiler optimization.

To make a 29,665-parameter model run within these boundaries, we had to treat Scratch like bare metal.

---

2. Model Architecture & Arena Memory Layout

The architecture is a micro-GPT configured to strike the Pareto frontier of expressiveness and Scratch VM limits:

• Vocabulary ($V$): 65 characters
• Context Length ($T$): 64 tokens
• Model Dimension ($d$): 32
• Layers ($L$): 2
• Attention Heads ($H$): 4 ($d_{head} = 8$)
• Feedforward Hidden Dimension ($d_{ff}$): 128 ($4 \times d$)
• Total Trainable Parameters: 29,665

Contiguous Arena Allocation

Dynamic memory allocation during training is a disaster in an environment with no garbage collection controls. We designed a contiguous arena allocator that statically partitions three global lists:

1. M (Parameters): 29,665 elements storing $W_{te}, W_{pe}, W_q, W_k, W_v, W_{proj}, W_{fc1}, W_{fc2}$, and LayerNorm scale/bias parameters.
2. A (Activations): 119,552 elements storing residual streams, queries, keys, values, attention weight matrices, GELU activations, and output logits for a batch size $B=1$ and context $T=64$.
3. dA (Adjoints): Exactly mirrors A to store gradients for the analytical backward pass.

Because Scratch lists have a ceiling of 200,000 items, the entire forward and backward activation footprint (119,552 elements each) fits within a single list without triggering memory overflow.

---

3. Hand-Derived Analytical Backpropagation

In PyTorch, loss.backward() triggers dynamic computation graph traversal via autograd. In Scratch, there is no tape and no runtime graph.

Every gradient step had to be derived analytically by hand and implemented as explicit unrolled custom blocks:

• Softmax Backward: Derived using the Jacobian vector product $\frac{\partial \mathcal{L}}{\partial z_i} = p_i \left( \frac{\partial \mathcal{L}}{\partial p_i} - \sum_j \frac{\partial \mathcal{L}}{\partial p_j} p_j \right)$.
• LayerNorm Backward: Explicit sensitivity propagation through sample mean $\mu$ and running variance $\sigma^2$ with epsilon stabilization.
• Causal Attention Backward: Gradient projection across transposed key and value slices with causal upper-triangular masking.
• GELU Backward: Approximated through the standard tanh-based formulation and its analytical derivative.

For optimization, we implemented full AdamW with first-moment ($m$) and second-moment ($v$) tracking, bias correction, weight decay, gradient clipping, and a cosine learning rate decay schedule.

---

4. Verification Against a 64-Bit NumPy Oracle

Claims of correctness in machine learning systems are meaningless without falsifiable baselines.

We built a Python toolchain that programmatically assembled the .sb3 project file and verified its execution headlessly against a reference 64-bit NumPy oracle using Node.js scratch-vm:

Region                    Offset   Count    Max Rel Err  Status
-----------------------------------------------------------------
x_emb                     0        2048     0.000e+00    PASS
l0.ln1_out                2048     2048     4.765e-14    PASS
l0.q                      4224     2048     2.428e-12    PASS
l0.att_p                  10368    16384    6.637e-16    PASS
l0.fc1_out                35072    8192     1.060e-12    PASS
l0.gelu_out               43264    8192     1.060e-12    PASS
logits                    111232   4160     8.751e-12    PASS
probs                     115392   4160     8.640e-16    PASS
-----------------------------------------------------------------
Overall: ALL 40 REGIONS PASS (rel_err < 1e-9)

• Forward Pass Precision: Across all 40 activation regions, the maximum relative error between Scratch's JavaScript floating-point runtime and NumPy was $< 1.4 \times 10^{-11}$.
• Gradient Precision: Comparing our hand-derived Scratch backward kernels to the analytical gradient oracle yielded a maximum relative error of $< 5.07 \times 10^{-12}$.
• Initial Loss: At initialization with uniform random weights, cross-entropy loss measured 4.1756 nats/token, matching the theoretical uniform prior $\ln(65) = 4.1744$.
• Overfitting Test: Under single-batch training in TurboWarp, the loss converged to 0.0494 nats/token at step 82, greedily reproducing the training sequence token-for-token.

---

5. What This Teaches About Engineering

Building this project reinforced three core engineering lessons that standard ML tutorials obscure:

1. Memory layout is everything. In low-resource environments (whether an educational visual VM or an embedded microchip), latency and correctness are determined by memory locality and contiguous stride calculation, not high-level abstractions.
2. Autograd is not magic. Deriving backpropagation for causal multi-head attention and LayerNorm by hand strips away the mystery of modern deep learning frameworks.
3. If you can't verify it with numbers, it doesn't work. An implementation that "looks right" often has subtle target misalignment or numerical instability. Having an automated gate suite (73 passing tests running headlessly) is the only reason this project works.

The full codebase, compiled .sb3 project, and LaTeX academic paper are available on GitHub:

• Paper: Read Paper PDF (GitHub)
• Code & Project: github.com/ayodejiades/scratch