GPT-2

The GPT series follows the decoder route, adhering to the autoregressive language model objective, and has revealed the phenomenon of emergent capabilities through scaling.

When OpenAI released GPT-2 in 2018, it staged the public release of the model weights on the grounds that its capabilities were too powerful and could be misused.

GPT-series models use the Byte Pair Encoding (BPE) algorithm to segment text into tokens.

GPT series models use byte-level BPE (Byte-level BPE), which takes raw bytes as the basic unit to eliminate the out-of-vocabulary character problem.

Karpathy's earlier nanoGPT implemented GPT-2 training in approximately 600 lines of code.

Dario Amodei previously served as VP of Research at OpenAI, leading the research work on key models such as GPT-2 and GPT-3.

OpenAI has used a variant of BPE since GPT-2, and GPT-4's cl100k_base vocabulary contains approximately 100,000 tokens.

Karpathy的llm.c项目用纯C语言实现了GPT-2训练，完全不依赖PyTorch等框架

Few-shot learning capability was almost unusable in the era of GPT-2 (1.5 billion parameters) and suddenly became practical with GPT-3 (175 billion parameters).

GPT-2's largest version had approximately 1.5 billion parameters

Transformer Explainer is an interactive tool developed by a team at Georgia Tech that uses GPT-2 as a basis to demonstrate the computation flow of each Transformer layer.

In work such as GPT-2 and GPT-3, OpenAI systematically validated scaling laws and used them as the basis for planning its model R&D roadmap.

With exactly the same GPUs and the same data, the cost of training GPT-2 has dropped from the initial level of several hundred dollars to about one hundred dollars, thanks to technical improvements at the algorithm, data, and kernel levels.

Training from scratch on the nanochat, Llama and GPT-2 model families verified the effectiveness of low-bit n-gram tables.

Experiments are conducted on synthetic problems, data hyper-cleaning, and GPT-2 data reweighting tasks.

GPT-2 uses learnable positional embeddings added to token embeddings; if the K/V ordering in the cache is misaligned with the positions assumed by the query vector, then when the sequence length is greater than 1 a mismatch occurs between query positions and cache positions, changing the visible set under the causal mask and thereby changing the softmax weights and output.

An incorrect KV cache ordering can pass a single-token test, because when only one token is generated the cache has only one history entry and any ordering degenerates to the same result; the output deviation is only exposed once the sequence grows longer and the cache has multiple entries.

LoRA performs comparably to or even better than full fine-tuning on RoBERTa, DeBERTa, GPT-2 and GPT-3, for example reaching 74.0% on WikiSQL and 91.7% on MultiNLI.

In controlled experiments attaching newly initialized modules to a frozen GPT-2, all experimental groups reproduced the results of the 50 completed training runs across three sampling seeds, indicating that basic memory writing itself is achievable.

In the controlled GPT-2 experiments, the two experimental groups that updated only row vectors achieved an average hit rate of 5.3% for the target string under natural language questioning.