Highlights of key papers and progression of Neural Networks over the last decade (ish). This isn't learning material -- its a reference!

Last Updated: 2026-8-15

We'll Start with the Recurrent Neural Net

Because we have to start somewhere.

A Neural Probabilistic Language Model ("Curse of Dimensionality") establishes why predicting next token can be effective but difficult to scale due to the sheer number of dimensions needed when one-hot encoding tokens in a vocabulary. The authors propose adding an Embedding layer as the first action in a network, which is learned during the training process. The length (size) of an Embedding vector remains a hyper-parameter. Through this process tokens naturally organize themselves in space, so similar tokens cluster together based on the training data. Embeddings being smaller than the entire vocabulary helps solve dimensionality while encoding more information.

The Unreasonable Effectiveness of Recurrent Neural Networks + Attention & RNNs establish the importance of prior tokens needing to inform latter tokens through building up an internal context matrix. This proves to be very effective but also limiting -- an internal context only holds so much information and becomes "lossy". When we combine this concept with Attention we find even greater success, leading to Attention is all You Need (and the transformer)

The Transformer

The Illustrated Transformer demonstrates the structure of a Transformer. This is not the same as 'modern' GPTs. A transformer has two halves: an Encoder and a Decoder. Each side is made of a stacks of transformer layers, where each item has:

Self Attention & The Decoder Calculation

How is self-attention implemented?

Inputs Our token embeddings for all tokens in the input thus-far and their positional encodings (relative to each other/the start of input - one-hot vectors). The input+position are combined to yield a single vector of dimensionality width. Inputs go through "layer normalization" before entering each transformer block, where each token row gets rescaled to handle high skew.

Attention Input is fed through a self-attention layer. This layer has 3 parameter matricies learned during training: Query (Wq), Key (Wk), and Value (Wv). At inference time this means:

  1. Each token's Embedding is mat-muled against each Wq, Wk, Wv to produce a Query/Key/Value vector per token. These get cached across iterations, so only the latest token really computes something new here. This calculation can be batched: if we concatenate the Wq, Wk, Wv together horizontally, then mat-mul against a vertical embedding vector yields a concatenated vector Vq, Vk, Vv
  2. The current token's query vector is multiplied by all the key vectors (dot product) of all the prior tokens. This tells us how well each token matches against the current one, generating a "score".
  3. This score is divided by the sqrt of the dimensionality of the Vk. If we're training, then any token after this token's position is also zero'd (-inf) to prevent revealing the true next tokens. Softmax is then applied.
  4. Each value vector is then multiplied against this score and the entire list summed, yielding a new result vector Z

Multi-head attention (MHA) has proven valuable, which trades computational cost for even better performance in finding what to attend to. There's different ways to implement this.

After obtaining each head's Z vector, heads are merged by:

Performing self-attention as Matrix Operations -- the above demonstrated vector-wize operations but much of this can be made matrix ops. This will be more relevant for training side of the equation.

  1. Concatenate the embedding tokens & rebuild K/Q/V using Wk, Wq, Wv. More specifically the KV cache will be used to restore some of the prior token's K/Q/V when rebuilding these matricies.
  2. The rest of the steps are consolidated: Softmax(Q*K^T / dk) * V = Z. (where dk is that dimensionality of K mentioned earlier -- 768).
  3. In the case of multiple attention heads, the same applies: The Z matricies are concatenated from the result of each head and then multiplied by a matrix to get the result back down to the size our FFN expects.

Residuals -- before feeding this result into the FFN, the orignial input embedding is element-wise added back to the self-attention result above forming a residual connection. This is also done after each FFN result before passing to the next decoder in the stack.

The Feed Forward Network

This part gives the transformer it's space to learn/encode information after the attention context has been baked into the input. The sizing of this is entirely a hyper-parameter. In GPT2 this was a 2 layer network where:

(Bigger Picture)

Stepping back, the transformer takes the effectiveness of an RNNs internal representation but solve the lossy problem by just looking up the prior context "on the fly" through the Key & Query vectors, and extract the learned information about each token through the V vector. Phrased another way:

These modern models (to put this in perspective) don't make bigger and bigger individual layers but instead stack them deeper and deeper. GPT-3 had 175B parameters: 96 Decoder layer, 96 attention heads, 12,288 dimensions in each layer.

Faster Attention & Caching

Attention heads directly make the entire computation slower. One Write-Head is All You need & GQA: Grouped Query Attention discuss the implications of this. While training these models can parallelize / batch feed forward, inference has to do one token at a time making the inference side carry a different cost profile than the training side (a token is predicted and then the entire phrase is re-fed through).

This is where the KV cache concept comes in -- there's no need to recompute the KV vectors of each prior token on each future token. But it does mean the context size ("Sequence Length") worth of tokens (and specifically their KV vectors) must be held onto. Key Value matricies (& the KV cache) scale with 2 * L * SequenceLength * model dimensionality:

These articles find that having many heads sharing fewer KV matricies is effective -- we can still gain the performance of additional Heads without the overhead of storing (and thus loading) multiple different KV matricies. In practice, having some number of KV matricies 1 < Num(KV) < Heads is best: "Grouped Query Attention" proposes having groups of heads share the same KV balancing the performance of 1 KV per head and the performance of 1 KV for all heads.

Bigger Data & Training

Training becomes its entirely own data engineering problem space. Rather than speaking in terms of epochs (eg running the same training dataset through the network multiple times) we're now looking at how many tokens we can run through the model. The laws of large data apply: a sufficiently large dataset will observe all possible trends and distributions. Chinchilla ('Training Compute Optimal Large Language Models') affirms this: a bigger model (more parameters) actually performs worse than a smaller model with a much higher ratio of tokens fed through during training. The rule of thumb has become 20 tokens / parameter is needed in training.

A series of other papers teach us about harvesting data for the initial "pre-" training:

When it comes to post-training (or tuning the model's outputs to match human expectations) the majority of discussion is around "Alignment". Most of this wasn't very interesting to me. Two concepts stood out:

Reasoning

Back to model design, we arrive at Reasoning. Chain-of-Thought was a large breakthrough in improving logical thought processing, like math word problems -- it didn't require any changes to the model itself but instead how you prompt the model. Providing the model with examples of "thinking through the problem aloud" and then asking it to then solve a similar problem would have it mimic your example approaches and arrive at correct answers far more often. This concept could actually be trained into the network, to such a degree that this prompting technique isn't needed in the latest GPTs.

Self Consistency provides another framework for improving model results at the harness level by having it process the same question multiple times, using different avenues for getting to the final answer. The harness would then sample the provided responses to find which was the most frequent and treat that as the final, correct result. This, combined with Chain-of-Thought pushes performance even higher.

(TODO): Continue! ReAct DeepSeek-R1