microgpt

A GPT in 200 lines of Python

Computational Physics II · PHY4008 · Sogang University · November 19, 2026

BOS microgpt a n n a
1

Last week, this week

Ising classifiermicrogpt
Inputone spin configurationthe characters so far
Output2 class probabilities27 next-character probabilities
Losscross-entropycross-entropy
Gradientsbackpropagation by handautomatic (autograd)
Networkone hidden layerone transformer layer

The ingredients are the same as last week: a network, a cross-entropy loss, gradients. New today: the task of predicting the next character, and attention.

2

Language model

text so far model probability of the next token The capital of France is tokens: words language model billions of parameters Paris 0.92 the 0.02 a 0.01 located 0.01 in 0.01 + about 100,000 other tokens 01 probability (illustrative) BOS e m tokens: letters (BOS = start of a name) microgpt 4,192 parameters i 0.22 a 0.22 e 0.18 y 0.09 o 0.07 + 22 other tokens 01 probability (real)

A language model maps the text so far to a probability for every possible next token. A large model does this for words, microgpt for letters.

3

Generating text

The cat sat text so far language model on 0.60 down 0.20 in 0.10 by 0.05 probabilities (illustrative) on new token 1 predict 2 sample 3 append, then repeat The cat sat on loop 1 the loop 2 mat loop 3 . loop 4 prompt after 4 loops

Text is generated one token at a time: predict, sample, append, repeat. ChatGPT runs this same loop.

4

Transformer

BOS e m m tokens one vector per token attention tokens exchangeinformation MLP each token computeson its own updated vectors microgpt
\(N=1\)
tokens embedding probabilities 4,192 parameters GPT-2 small
\(N=12\)
tokens embedding probabilities 124 million parameters one block

A transformer repeats one block: attention lets tokens exchange information, the MLP processes each token separately. GPT is a stack of such blocks.

5

Scale

Models have grown by many orders of magnitude, with many improvements along the way. The core idea is unchanged: a transformer that predicts the next token. microgpt is that core alone.

Models have grown by many orders of magnitude, with many improvements along the way. The core idea is unchanged: a transformer that predicts the next token. microgpt is that core alone.

6

microgpt.py

1–12 imports os, math, random 14–21 Dataset 32,033 names 23–27 Tokenizer characters ↔ integers 29–72 Autograd chain rule on a graph 74–90 Parameters 4,192 numbers 92–144 Architecture token → 27 scores 146–184 Training loop 1,000 Adam steps 186–200 Inference sample new names 1 200

Seven short parts make a complete GPT.
“This file is the complete algorithm. Everything else is just efficiency.” (A. Karpathy)

7

Dataset

input.txt one document 1 emma 2 olivia 3 ava 4 isabella 5 sophia 6 charlotte 7 mia 8 amelia ⋮ ⋮ 32033 zzyzx samples 1 kamon 2 ann 3 karai 4 jaire 5 vialan 6 karia 7 yeran 8 anna ⋮ ⋮ microgpt train sample training data: 32,033 names model output: new names

The model sees nothing but a list of names. It learns their statistics and produces new strings that look like names; some coincide with real ones.

8

Tokenizer

a 0 b 1 c 2 d 3 e 4 f 5 g 6 h 7 i 8 j 9 k 10 l 11 m 12 n 13 o 14 p 15 q 16 r 17 s 18 t 19 u 20 v 21 w 22 x 23 y 24 z 25 BOS 26 vocabulary token id BOS 26 e e 4 m m 12 m m 12 a a 0 BOS 26 name tokenstoken ids look up the id

A tokenizer turns text into integers. Each letter is one token; the extra token BOS marks both ends of a name.

9

Next-token prediction

BOS e m m a BOS tokens target (next token) context (tokens so far) model
\(t=0\)
BOS gpt e
\(t=1\)
BOS e gpt m
\(t=2\)
BOS e m gpt m
\(t=3\)
BOS e m m gpt a
\(t=4\)
BOS e m m a gpt BOS

The model is trained to predict the next token from the tokens before it.
One four-letter name gives five training examples.

10

Value

\(a\)
data 2 grad 0
\(b\)
data 3 grad 0
\(c=a\cdot b\)
data the number grad gradient of the loss, filled in later _children the inputs it was made from _local_grads derivative w.r.t. each input 6
\(=2\cdot 3\)
0
\(\to\ \dfrac{\partial L}{\partial c}\)
\(a\)
\(b\)
3 2
\(\dfrac{\partial c}{\partial a}=b\)
\(\dfrac{\partial c}{\partial b}=a\)

Every number in the model is a Value: it remembers its inputs and the local derivatives, so the chain rule can be applied later.

11

Lego blocks

OperationForwardLocal gradients \(\partial/\partial a,\ \partial/\partial b\)
a + b\(a+b\)\(1,\quad 1\)
a * b\(ab\)\(b,\quad a\)
a ** n\(a^n\)\(n\,a^{n-1}\)
a.log()\(\ln a\)\(1/a\)
a.exp()\(e^{a}\)\(e^{a}\)
a.relu()\(\max(0,a)\)\(1\ \text{if}\ a>0,\ \text{else}\ 0\)

The whole model is built from these six operations. Each one knows only its own local derivative; the chain rule does the rest.

12

Forward pass on a graph

\(b\)
data 3 grad 0
\(a\)
data 2 grad 0 value (forward pass) gradient, not yet computed

The inputs are leaf nodes: each holds a value, and a gradient that starts at 0.

13

Forward pass on a graph

\(\times\)
\(b\)
data 3 grad 0
\(a\)
data 2 grad 0
\(c\)
data 6 grad 0
\(\dfrac{\partial c}{\partial b}=a=2\)
\(\dfrac{\partial c}{\partial a}=b=3\)
\(c=a\cdot b\)
value (forward pass) gradient, not yet computed

Each operation creates a new node and records the local derivative with respect to each input.

14

Forward pass on a graph

\(\times\)
\(+\)
\(b\)
data 3 grad 0
\(a\)
data 2 grad 0
\(c\)
data 6 grad 0
\(L\)
data 8 grad 0
\(\dfrac{\partial c}{\partial b}=a=2\)
\(\dfrac{\partial c}{\partial a}=b=3\)
\(\dfrac{\partial L}{\partial c}=1\)
\(\dfrac{\partial L}{\partial a}=1\)
\(c=a\cdot b\)
\(L=c+a\)
value (forward pass) gradient, not yet computed

The forward pass ends at \(L\). Every edge now carries a local derivative; no gradient has been computed yet.

15

Backward pass

\(\times\)
\(+\)
\(b\)
data 3 grad 0
\(a\)
data 2 grad 0
\(c\)
data 6 grad 0
\(L\)
data 8 grad 1
\(\dfrac{\partial c}{\partial b}=a=2\)
\(\dfrac{\partial c}{\partial a}=b=3\)
\(\dfrac{\partial L}{\partial c}=1\)
\(\dfrac{\partial L}{\partial a}=1\)
\(\dfrac{\partial L}{\partial L}=1\)
\(c=a\cdot b\)
\(L=c+a\)
value (forward pass) gradient
\(\partial L/\partial(\text{node})\)
node being processed

The backward pass starts at the output: the derivative of \(L\) with respect to itself is 1.

16

Backward pass

\(\times\)
\(+\)
\(b\)
data 3 grad 0
\(a\)
data 2 grad 1
\(c\)
data 6 grad 1
\(L\)
data 8 grad 1
\(\dfrac{\partial c}{\partial b}=a=2\)
\(\dfrac{\partial c}{\partial a}=b=3\)
\(\dfrac{\partial L}{\partial c}=1\)
\(\dfrac{\partial L}{\partial a}=1\)
\(\dfrac{\partial L}{\partial L}=1\)
\(1\cdot 1=1\)
\(1\cdot 1=1\)
\(c=a\cdot b\)
\(L=c+a\)
value (forward pass) gradient
\(\partial L/\partial(\text{node})\)
node being processed

Each node passes its gradient to its inputs, multiplied by the local derivative on the edge.

17

Backward pass

\(\times\)
\(+\)
\(b\)
data 3 grad 2
\(a\)
data 2 grad 4
\(c\)
data 6 grad 1
\(L\)
data 8 grad 1
\(\dfrac{\partial c}{\partial b}=a=2\)
\(\dfrac{\partial c}{\partial a}=b=3\)
\(\dfrac{\partial L}{\partial c}=1\)
\(\dfrac{\partial L}{\partial a}=1\)
\(\dfrac{\partial L}{\partial L}=1\)
\(1\cdot 1=1\)
\(1+3\cdot 1=4\)
\(2\cdot 1=2\)
\(c=a\cdot b\)
\(L=c+a\)
value (forward pass) gradient
\(\partial L/\partial(\text{node})\)
node being processed

A value used twice receives a gradient from each path, and the contributions add up.

18

Backward pass

\(\times\)
\(+\)
\(b\)
data 3 grad 2
\(a\)
data 2 grad 4
\(c\)
data 6 grad 1
\(L\)
data 8 grad 1
\(\dfrac{\partial c}{\partial b}=a=2\)
\(\dfrac{\partial c}{\partial a}=b=3\)
\(\dfrac{\partial L}{\partial c}=1\)
\(\dfrac{\partial L}{\partial a}=1\)
\(\dfrac{\partial L}{\partial L}=1\)
\(1\cdot 1=1\)
\(1+3\cdot 1=4\)
\(2\cdot 1=2\)
\(\dfrac{\partial L}{\partial a}=\dfrac{\partial L}{\partial c}\dfrac{\partial c}{\partial a}+1=1\cdot 3+1=4\)
value (forward pass) gradient
\(\partial L/\partial(\text{node})\)

One backward sweep gives the derivative of \(L\) with respect to every node: the chain rule, applied edge by edge.

19

Parameters

wte one row per token
\(27\times16=432\)
wpe one row per position
\(16\times16=256\)
attn_wq query
\(16\times16=256\)
attn_wk key
\(16\times16=256\)
attn_wv value
\(16\times16=256\)
attn_wo output
\(16\times16=256\)
mlp_fc1 expand
\(64\times16=1{,}024\)
mlp_fc2 project back
\(16\times64=1{,}024\)
lm_head 27 scores
\(27\times16=432\)
embedding attention MLP output 4,192 parameters 1 cell = 1 number

The model is nothing but these 4,192 numbers, initially random. Training adjusts them; the architecture only fixes how they are combined.

20

Forward pass

mtokenwtewpe3positionrmsnorm
\(x\)
16Attentionrmsnorm
\(W_q\)
\(W_k\)
\(W_v\)
4 heads
concat \(\;\to\, W_o\)
MLPrmsnormfc1 16 → 64ReLUfc2 64 → 16residualresidual1616lm_headlogits27softmaxi0.25a0.23e0.17y0.08o0.05
\(p(\text{next token})\)
looks at earlier tokensprocesses this token27 scores → probabilities

One token and its position become a vector of 16 numbers. Attention and the MLP each add a correction to it; the result is turned into 27 next-token probabilities.

21

Vector space

manwomankingqueenapplebananaorangegenderroyalty
\(\vec{v}_{\text{king}}-\vec{v}_{\text{man}}+\vec{v}_{\text{woman}}\approx\vec{v}_{\text{queen}}\)
similar words are close2D sketch of a space with hundreds of dimensionsMikolov et al., 2013 (word2vec)

An embedding gives every token a position in a vector space. After training, nearby vectors mean similar things and directions carry meaning; microgpt does the same for letters, in 16 dimensions.

22

Embedding

wtetoken embeddings
\(27\times16\)
abcdefghijklmnopqrstuvwxyzBOSwpeposition embeddings
\(16\times16\)
0123456789101112131415token mposition 3sumrmsnorm
\(x\)
the vector that enters the model−0.50+0.5parameter value

Every token and every position owns one row of 16 learned numbers. Their sum, rescaled, is the vector \(x\) that represents “m at position 3”.

23

Query, key, value

current token m
\(t=3\)
\(x\)
its embedding (16 numbers) attn_wq linear map, 16 × 16 query
\(q_3\)
“what I look for” attn_wk linear map, 16 × 16 key
\(k_3\)
“what I contain” attn_wv linear map, 16 × 16 value
\(v_3\)
“what I pass on” append to keys append to values m
\(t=3\)
m
\(t=2\)
e
\(t=1\)
BOS
\(t=0\)
new stored by earlier tokens used now, not stored

Each token is turned into three vectors: a query (what it looks for), a key (what it offers) and a value (what it passes on). Keys and values are stored for later tokens.

24

softmax

\[p_i=\frac{e^{\,l_i}}{\sum_{j} e^{\,l_j}}\]

\(l_i\): logits, any real numbers  ·  \(p_i\): probabilities, positive and summing to 1

softmax turns scores into probabilities. It is the Boltzmann distribution with \(E_i=-l_i\) and \(k_BT=1\): a higher score means a more probable state.

25

One attention head

token key BOS
\(k_0\)
e
\(k_1\)
m
\(k_2\)
m
\(k_3\)
a not yet seen

Every token seen so far has left a key in the cache: a short vector that says what that token contains.

26

One attention head

token key BOS
\(k_0\)
e
\(k_1\)
m
\(k_2\)
m
\(k_3\)
\(q_3\)
query what the current token looks for a not yet seen

The current token makes a query: a vector that says what it is looking for.

27

One attention head

token key score
\(q_3\cdot k_t/\sqrt{4}\)
dot product BOS
\(k_0\)
0.14 e
\(k_1\)
0.15 m
\(k_2\)
−2.06 m
\(k_3\)
−0.68
\(q_3\)
query what the current token looks for a not yet seen

The dot product of the query with each key gives one score per token. A higher score means a better match.

28

One attention head

token key score weight
\(q_3\cdot k_t/\sqrt{4}\)
\(\alpha_t\)
dot product BOS
\(k_0\)
0.14 0.39 e
\(k_1\)
0.15 0.39 m
\(k_2\)
−2.06 0.04 m
\(k_3\)
−0.68 0.17 softmax: weights are positive and add up to 1
\(q_3\)
query what the current token looks for a not yet seen

Softmax turns the scores into weights that are positive and add up to 1. The best-matching tokens get the largest weights.

29

One attention head

token key score weight value
\(q_3\cdot k_t/\sqrt{4}\)
\(\alpha_t\)
dot product BOS
\(k_0\)
0.14 0.39
\(\times\)
\(v_0\)
e
\(k_1\)
0.15 0.39
\(\times\)
\(v_1\)
m
\(k_2\)
−2.06 0.04
\(\times\)
\(v_2\)
m
\(k_3\)
−0.68 0.17
\(\times\)
\(v_3\)
softmax: weights are positive and add up to 1
\(q_3\)
query what the current token looks for a not yet seen

Every token has also stored a value: the information it passes on. Each value is multiplied by the weight of its token.

30

One attention head

token key score weight value
\(q_3\cdot k_t/\sqrt{4}\)
\(\alpha_t\)
dot product BOS
\(k_0\)
0.14 0.39
\(\times\)
\(v_0\)
\(+\)
e
\(k_1\)
0.15 0.39
\(\times\)
\(v_1\)
\(+\)
m
\(k_2\)
−2.06 0.04
\(\times\)
\(v_2\)
\(+\)
m
\(k_3\)
−0.68 0.17
\(\times\)
\(v_3\)
\(=\)
softmax: weights are positive and add up to 1
\(q_3\)
query what the current token looks for a not yet seen head output
\(\sum_t \alpha_t\, v_t\)
(−0.06, 0.13, −0.19, 0.09)

The head output is the weighted average of the values: the current token collects information mostly from the tokens it attends to.

31

Attention

\[\alpha_t=\frac{\exp\!\left(q\cdot k_t/\sqrt{d}\right)}{\sum_{s\le \mathrm{pos}}\exp\!\left(q\cdot k_s/\sqrt{d}\right)},\qquad \mathrm{out}=\sum_{t\le \mathrm{pos}}\alpha_t\,v_t\]

\(q\): query of the current token  ·  \(k_t, v_t\): key and value of token \(t\)  ·  \(d=4\): dimension of one head

The previous figure in one line: a softmax over the tokens seen so far gives the weights \(\alpha_t\), and the output is the weighted average of their values.

32

Four heads

query
\(q\)
4 heads × 4 numbers; keys, values likewise head 0 BOS e m m a BOS 1 e .47 .53 m .43 .35 .22 m .28 .31 .18 .23 a .22 .26 .18 .16 .17 head 1 BOS e m m a BOS 1 e .49 .51 m .41 .30 .29 m .30 .23 .22 .24 a .16 .10 .09 .22 .43 head 2 BOS e m m a BOS 1 e .14 .86 m .39 .57 .04 m .39 .39 .04 .17 a .02 .10 .25 .56 .07 head 3 BOS e m m a BOS 1 e .96 .04 m .27 .46 .27 m .22 .39 .21 .17 a .22 .02 .16 .50 .11 output joined: 16 attn_wo MLP
\(x\)
before attention row: current token column: token it attends to

Each head works on its own 4 numbers and learns its own attention pattern. The four outputs are joined and mixed by one linear map.

33

KV cache

\(t=0\)
BOS e m m a query
\(q_0\)
key
\(k_0\)
value
\(v_0\)
BOS keys values
\(t=2\)
BOS e m m a query
\(q_2\)
key
\(k_2\)
value
\(v_2\)
BOS e m keys values
\(t=4\)
BOS e m m a query
\(q_4\)
key
\(k_4\)
value
\(v_4\)
BOS e m m a keys values

A token can attend only to what is already stored: itself and the earlier tokens. The future is not in the cache, so no mask is needed.

34

MLP and residual

16
\(x\)
rmsnormmlp_fc116 → 6464ReLU
\(\max(0,z)\)
\(z\)
64mlp_fc264 → 161616
\(x\)
residual: the input is kept−1.50+1.5value of each entryinput +correction

Attention moves information between tokens; the MLP computes on one token alone. It widens the vector to 64, keeps only the positive entries, and adds the result to its input.

35

rmsnorm

\[\mathrm{rmsnorm}(x)_i=\frac{x_i}{\sqrt{\frac{1}{n}\sum_{j=1}^{n}x_j^{2}+\varepsilon}}\]

\(n=16\): vector dimension  ·  \(\varepsilon=10^{-5}\)

The vector keeps its direction and its root-mean-square is rescaled to 1. This keeps the numbers from growing or vanishing as they pass through the blocks.

36

Training loop

1 One name tokens BOS 26 e 4 m 12 m 12 a 0 BOS 26 2 Forward pass gpt() tokens model predictions
\(L\)
loss 3 Backward pass loss.backward()
\(L\)
loss
\(\partial L/\partial p_i\)
4,192 gradients 4 Adam update params 4,192 parameters
\(p \leftarrow p-\eta_t\,\dfrac{\hat m}{\sqrt{\hat v}+\epsilon}\)
small step × 1,000 steps loss 3.37 → 2.32

One step = one name, one forward pass, one backward pass, one small change to all 4,192 parameters. Repeating it 1,000 times is the whole training.

37

Loss

input token correct next token BOS e e m m m m a a BOS probability the model gave it
\(p_t\)
0.05 0.06 0.01 0.23 0.17
\(1/27=0.037\)
uniform guess loss at this position
\(-\ln p_t\)
2.97 2.77 4.62 1.48 1.80 2.73
\(\ln 27=3.30\)
uniform guess mean
\(L=\frac{1}{5}\sum_{t=0}^{4}\bigl(-\ln p_t\bigr)=2.73\)
loss of the whole name

At each position the loss is \(-\ln p\) of the correct next token: high probability costs little, low probability costs a lot. The loss of the name is the mean, 2.73.

38

Cross-entropy

\[L=\frac{1}{n}\sum_{t=0}^{n-1}\Bigl[-\ln p_t(\mathrm{target}_t)\Bigr]\]

\(n\): number of positions in the name  ·  \(p_t(\mathrm{target}_t)\): probability the model gave to the correct next token

The loss is low when the correct next character gets high probability. A uniform guess gives \(\ln 27\approx 3.30\); anything below that has been learned.

39

Adam

\[\begin{aligned} m &\leftarrow \beta_1 m+(1-\beta_1)\,g, &\qquad \hat m &= \frac{m}{1-\beta_1^{\,t}}\\[6pt] v &\leftarrow \beta_2 v+(1-\beta_2)\,g^{2}, &\qquad \hat v &= \frac{v}{1-\beta_2^{\,t}}\\[6pt] p &\leftarrow p-\eta_t\,\frac{\hat m}{\sqrt{\hat v}+\epsilon} \end{aligned}\]

\(g=\partial L/\partial p\)  ·  \(\beta_1=0.85\), \(\beta_2=0.99\)  ·  \(\eta_t=0.01\,(1-t/1000)\)

Gradient descent with two additions: \(m\) averages recent gradients (momentum), and \(\sqrt{v}\) gives each parameter its own step size.

40

Loss curve

The average loss falls from \(\ln 27\approx 3.3\) to about 2.3 in 1,000 steps. It fluctuates because each step uses a different name.

The average loss falls from \(\ln 27\approx 3.3\) to about 2.3 in 1,000 steps. It fluctuates because each step uses a different name.

41

Inference

input token model probability of each next letter temperature = 0.5 draw one at random random.choices BOS gpt a k j m k 0.12 k gpt a e i o a 0.77 a gpt r n l y m m 0.03 m gpt i a e y o o 0.02 o gpt n r l BOS n 0.83 n gpt BOS a n e BOS 0.84 stop its probability new name: kamon

Generation is prediction in a loop: predict, draw one letter at random, feed it back as the next input. The name ends when BOS is drawn.

42

Temperature

\(T=0.5\)
sharper a b c d e f g h i j k l m n o p q r s t u v w x y z BOS 0.31 first letter of the name
\(T=1\)
as trained a b c d e f g h i j k l m n o p q r s t u v w x y z BOS 0.14 first letter of the name
\(T=2\)
flatter a b c d e f g h i j k l m n o p q r s t u v w x y z BOS 0.08 first letter of the name 0 0.1 0.2 0.3 probability
\(p_i=\frac{e^{\,l_i/T}}{\sum_j e^{\,l_j/T}}\)
logits
\(l_i\)
from the model

Low \(T\) concentrates probability on the most likely letters; high \(T\) flattens it. This is the Boltzmann distribution with energies \(E_i=-l_i\).

43

Generated names

kamonannkaraijairevialan
kariayeranannaarelikaina
konnakeylenliolealerinearan
lennekanalaraalelaanton

Eleven of these twenty are real names from the data; nine are inventions that only look like names. The model knows what is probable, not what exists: the same effect as a chatbot stating a plausible but false fact.

44

What it learned

Given only BOS, the trained model reproduces how often each letter starts a name in the data. The parameters have absorbed the statistics.

Given only BOS, the trained model reproduces how often each letter starts a name in the data. The parameters have absorbed the statistics.

45

Progression

FileWhat it adds
train0.pybigram count table, no neural network
train1.pyMLP, manual and numerical gradients, SGD
train2.pyautograd: the Value class
train3.pyposition embedding, single-head attention, rmsnorm, residuals
train4.pymulti-head attention and the layer loop: the full GPT
train5.pyAdam: this is microgpt.py

Karpathy builds the file in six steps, each adding one idea. Reading them in order is a good way to study the code.

46

microgpt and production LLMs

microgptProduction LLM
Data32,033 namestrillions of tokens of text
Tokenizer27 charactersabout 100,000 subword tokens
Autogradone scalar at a time, in Pythontensors on GPUs
Architecture4,192 parameters, 1 layer10¹¹ parameters, 100+ layers, RoPE, MoE, …
Trainingone name per step, 1,000 stepsmillions of tokens per step, for months
Post-trainingnonefine-tuning, reinforcement learning

Real LLMs are far larger and add many improvements. The core loop is the same: predict the next token, compute the loss, update the parameters.

47

Next week

  • We read microgpt.py line by line in the notebook and train it (about two minutes)
  • Experiments: temperature, first-letter statistics, attention maps
  • Before class: read the post at karpathy.github.io/2026/02/12/microgpt

Today was the map. Next week every box in these diagrams becomes a few lines of Python.

48

References

  • A. Karpathy, microgpt (2026): karpathy.github.io/2026/02/12/microgpt
  • J. Alammar, The Illustrated Transformer; The Illustrated GPT-2: jalammar.github.io
  • Transformer Explainer: poloclub.github.io/transformer-explainer
  • B. Bycroft, LLM Visualization: bbycroft.net/llm
  • 3Blue1Brown, Transformers and Attention: 3blue1brown.com/lessons/gpt
49