Understanding AI Large Models in Ten Minutes: Tokens, Vectors, and the Transformer Principle

AI large models have permeated scenarios like daily conversation, content creation, and code generation, yet most people still treat them as a "black box." Mike, co-founder of the Bilibili channel Gupao Technology, used a ten-minute video to break down the complete pipeline of large models from training to inference—from tokens and vectorization to the Transformer attention mechanism. This article builds on that video's content and further organizes the technical details.

The Industrial Shift Toward Generative AI Seen Through the ChatGPT Explosion

On November 30, 2022, OpenAI released ChatGPT. In under two months, its daily active users surpassed 100 million, breaking the previous nine-month growth record held by TikTok and making it the fastest-growing app in history. This product directly accelerated the AI industry's evolution from decision-based AI to generative AI, with companies at home and abroad rapidly entering the large model race. According to data cited in the video, within just two years, 63% of domestic enterprises were building large model applications. General-purpose foundation models such as Pangu, Tongyi, Wenxin, Hunyuan, and iFlytek Spark form the industrial foundation, while the application layer covers industries including education, automotive, and finance.

Conversational products rapidly proliferated after ChatGPT: overseas offerings like Anthropic Claude, Microsoft Bing AI, and Google Bard, and domestic ones like Wenxin Yiyan, Tongyi Qianwen, and Tencent Yuanbao, forming a dense product matrix. The prerequisite for understanding this industrial shift is distinguishing between two AI paradigms—decision-based AI makes judgments and predictions based on historical data, for example training a model on housing price data to predict rises and falls, commonly used in intelligent recommendation and decision support; generative AI, by contrast, learns and generalizes from existing data and then imitates and creates, generating entirely new content. The latter is closer to the ideal form of intelligence in human imagination and is therefore regarded as a new milestone in AI capability.

Why Large Models "Spit Out One Word at a Time"

First-time ChatGPT users often notice that answers stream out word by word, as if someone were typing in real time. This behavior is not a product design choice for "premium feel" but is determined by the model's principle: a large model is essentially reasoning about what the next piece of text is based on context.

Taking the Doubao model as an example, when generating content it returns information word by word or phrase by phrase, which the frontend then concatenates and displays. More specifically, given the sentence "Today's weather is very" to an LLM, the model infers based on context that the next character is "good"; this character is then fed back into the context, and it continues generating subsequent content like "sunny," "temperature," and "moderate." Through this iterative prediction process, short phrases gradually expand into long texts. This is the fundamental principle of large model text generation understood from surface phenomena.

Pretraining, Fine-Tuning, and Inference: The Three-Stage Pipeline of Large Models

Understanding large models requires first establishing a macro framework: from training to inference, the entire pipeline includes three stages—pretraining, fine-tuning, and inference.

Pretraining: Unsupervised Learning Builds the Foundation Model

Pretraining uses unsupervised learning, autonomously learning from massive knowledge bases. Taking GPT-3 as an example, its corpus covers books, news articles, scientific papers, Wikipedia, social media posts, and more. The officially published data annotates token counts for each corpus, including clear statistics for Common Crawl web crawler data, paper text, internet book corpora, and the Wikipedia knowledge base, with the overall training using 300 billion tokens.

A token is the basic unit by which models process and understand text. The model splits user input text into tokens, then performs computation and reasoning based on those tokens. You may not have noticed that token segmentation does not strictly correspond to English word boundaries—for example, "Tokenization" is split into the two tokens "Token" and "ization," and "Newer" is split into "New" and "er," with the space before a word usually forming a token together with the word. Different models use different tokenizers, so the same sentence may have different tokenization results across models, because tokenization algorithms may split by character, symbol, word, or subword, or may treat frequently co-occurring character combinations as a single unit.

Before training, text is first converted to token IDs, because computers cannot directly store text, and any character must ultimately be represented by numbers. During inference computation, the model performs a series of transformations based on the unique ID of each token, completing processing through the mapping between token text and IDs.

The difference between unsupervised and supervised learning can be illustrated with an intuitive example: in supervised learning, training data has both features and labels—for instance, knowing the cat is named Tom and the mouse is named Jerry—and the model learns the correspondence between features and labels, so when faced with new data containing only features it can still determine the label; unsupervised learning, by contrast, finds hidden structure in unlabeled data—the model doesn't know which is Tom and which is Jerry, but through machine learning it can identify that cat features belong to one type and mouse features to another, completing clustering. The result of pretraining is a foundation model that can predict the next token based on context—for example, given "The capital of Hunan is," it generates "Changsha"—but the foundation model is not good at conversation.

Tokenization algorithms are key to understanding token segmentation. Common tokenization methods include BPE (Byte Pair Encoding), WordPiece, and SentencePiece. BPE starts at the character level and repeatedly merges the most frequent adjacent character pairs, ultimately forming a subword vocabulary that balances vocabulary size against the out-of-vocabulary problem; WordPiece is similar but selects merges based on likelihood probability; SentencePiece directly treats raw text as a character sequence without relying on pre-tokenization. Different models choose different tokenizers, resulting in different token counts and ID mappings for the same text, which affects the model's context window utilization and inference cost.

Fine-Tuning: Supervised Learning Teaches the Model to Converse

To make the foundation model achieve ChatGPT-style conversational performance, further fine-tuning is needed. Fine-tuning is further training on an existing foundation model, changing the model's internal parameters to make it better suited to specific tasks. At this stage the model no longer learns from massive text but from high-quality conversation data provided by humans—equivalent to providing both the question and a reference answer, which is supervised fine-tuning. The fine-tuned model performs noticeably better on conversational questions.

To further enhance capabilities, the fine-tuned model can also undergo reinforcement learning—taking actions in an environment and receiving result feedback, continuously learning from that feedback to take optimal actions in specific situations. After pretraining and fine-tuning, the model can hold normal conversations with users, and the quality of its output is already considerable. For example, the model can generate answers like "Changsha is a vibrant city, combining historical culture with modern urban charm"—essentially a reasoning result obtained by the large model after training on massive text.

RLHF (Reinforcement Learning from Human Feedback) is a key technique for further improving model alignment after fine-tuning, and much of ChatGPT's success is attributed to it. RLHF typically proceeds in three steps: first, collect preference ranking data from humans on model outputs, and train a reward model to simulate human evaluation; then, using the fine-tuned model as the policy, optimize the policy with a reinforcement learning algorithm (such as PPO) so that its generated answers receive higher rewards; at the same time, add a KL divergence constraint to prevent the model from drifting too far from the original language model. RLHF makes the model not only "able to talk" but also more aligned with human intent and values.

Probability Tables and Hallucinations: Why Large Models "Spout Nonsense with a Straight Face"

Large model text generation is essentially a probability problem. Take "I like bananas or apples" as an example—the choice of the next word depends on the probability model learned during pretraining. Suppose bananas has a probability of 80%; the generated text is then "I like bananas."

How is this probability obtained? A simplified example illustrates it: take three sets of training data, convert them to tokens, and the training task is to predict the token that follows other tokens. A simple approach is to take all consecutive token pairs in the training set and build a probability table. For example, in data such as "I like apples" and "I like bananas," "like" is followed by "apples" once and by "bananas" twice, so the probability that apples follows like is about 33.3% and bananas about 66.7%. Actual large model training uses massive text and does not suffer from such sample-sparsity gaps.

The hallucination problem of early large models "spouting nonsense with a straight face" stems precisely from this: the model itself cannot understand the meaning of text in the human world and can only generate content based on contextual probability. The generated text satisfies requirements in grammatical form, but how relevant it is to the current historical context and question is something the model cannot judge. As training parameters increase and model capabilities strengthen, such low-level problems gradually diminish.

Vectorization: Letting the Model Understand That "This Apple Is Not That Apple"

Human language is semantically rich. How does the model know that Apples and Bananas are both fruits and, in the context of "eating," can associate and substitute Orange? This involves vector embeddings.

A token ID is just a single integer with limited information expression capacity. Take man and woman as an example: both describe humans but are opposite in gender. If represented by a single number alone, the magnitude of the distance cannot simultaneously carry the two layers of information "both belong to humans" and "opposite genders." Vectorization converts token IDs into a string of numbers, bringing two benefits: first, high-dimensional vectors can represent richer syntactic and semantic information, with similar words having closer coordinates in vector space and unrelated words farther apart, allowing the model to capture complex relationships between words through mathematical computation; second, vectors have both direction and magnitude, and can represent relationships between objects in space. GPT-3's vector length reaches 12,288, with information capacity far exceeding a single number.

Another characteristic of vector representations is that similarity is measurable: cosine similarity measures the angle between vectors, or Euclidean distance measures the straight-line distance in space. In a three-dimensional illustration, fruit-category coordinates are close to each other, and animal-category coordinates are close to each other. In practical applications, "How much is this," "What's the price," and "Give me a quote" are all price-related questions with high vector coordinate similarity; "I want this" and "Give me this" also have high similarity; while the distance between these two categories of questions is relatively large. This solves the problem of text similarity judgment.

Cosine similarity and Euclidean distance are two common measures of vector similarity. Cosine similarity computes the cosine of the angle between two vectors, ranging from -1 to 1, with values closer to 1 indicating more consistent direction; it considers only direction and is unaffected by vector length, making it suitable for comparing text semantics. Euclidean distance computes the straight-line distance in vector space, with smaller values indicating greater similarity, but it is affected by vector magnitude. In practice, vectors are often normalized before using cosine similarity, making the two equivalent. Which measure to choose depends on the specific task and whether vectors are normalized.

Transformer and Self-Attention: The Key to Understanding Context

Vectorization still doesn't solve all problems. For example, in "I bought the latest model Apple on JD.com" and "I bought an Aksu apple on JD.com," the two "apples" refer to different objects, yet large models can understand text meaning like humans and generate semantically appropriate answers—the key is the Transformer.

The Transformer was proposed in the 2017 paper "Attention is All You Need" and rapidly became the mainstream architecture in natural language processing, widely used in tasks such as machine translation and text generation. Its core innovation is the self-attention mechanism, which enables the model to effectively capture relationships among elements within a sequence when processing sequence data, unrestricted by positional distance.

Take "The animal didn't cross the street because it was too tired" as an example—the "it" in the sentence could refer to the animal or the street. The Transformer uses the attention mechanism to compute relevance weights between each word and other words, identifying that "it" refers to the animal and writing the animal's hidden state into the vector space where it resides. GPT-3's vector length is 12,288, and representing a single word's contextual information in over ten thousand dimensions is more than enough. The higher the attention weight, the stronger the relevance between two words, and the model focuses on the most important parts of the input accordingly.

The Transformer is not a single-layer structure but may contain dozens of layers or more, each with its own attention mechanism and different focuses: some focus on verbs, some on modifiers, some on sentiment. The more layers, the deeper the understanding of text content. These layers are not computed serially but processed in parallel. GPT-3 contains 96 Transformer layers; the first few layers focus on sentence grammar and ambiguity resolution, while later layers work on high-level understanding of entire paragraphs—for example, tracking information such as gender, age, relationships, location, personality, and goals when reading a short story. The goal of a 96-layer LLM is to output a final word hidden state containing all necessary information, thereby accurately predicting the next word.

The computation process of the self-attention mechanism can be broken down into three vectors: Query, Key, and Value. For each token in the sequence, the model multiplies its vector by three learnable matrices to obtain Q, K, and V; it then takes the dot product of the current token's Q with the K of all tokens and divides by a scaling factor (usually the square root of the dimension), normalizing via Softmax to obtain attention weights; finally, it computes a weighted sum of V using the weights to obtain the token's new representation. This process enables each token to aggregate global contextual information and allows parallel computation, breaking through the sequential processing bottleneck of RNNs.

Summary

From tokenization and vector embeddings to the Transformer's multi-layer attention, the text generation chain of large models can be summarized as: pretraining learns the probability of the next token on massive text, fine-tuning teaches the foundation model to converse, reinforcement learning further enhances capabilities, and inference generates word by word and concatenates the output. Understanding this chain helps explain why large models hallucinate and why their output deviates from expectations, and it also provides a cognitive foundation for further tuning and optimizing models.

One-line Summary

From tokens and vectorization to the Transformer, break down the full pipeline of large model training and inference.

Paragraph Summary

Based on a video by the Bilibili channel Gupao Technology, this article systematically organizes the complete technical pipeline of large models from training to inference. It first reviews the background of ChatGPT igniting the generative AI industrial shift and distinguishes decision-based AI from generative AI. It then explains the principle of word-by-word output—predicting the next token based on context. It then details the three stages of pretraining, fine-tuning, and inference: pretraining uses unsupervised learning to build a foundation model on massive tokens, while fine-tuning uses supervised learning to teach the model to converse. The article also analyzes the mechanism by which probability tables lead to hallucinations, how vectorization lets the model capture semantic similarity, and finally delves into how the Transformer's self-attention mechanism understands contextual reference relationships, noting that GPT-3 contains 96 Transformer layers.