In-Context Learning

The in-context learning capability of Few-Shot prompting was first systematically described in the GPT-3 paper.

Simply stacking parameters without sufficient training data results in a model that is large but not capable; a leap in parameters must be accompanied by a proportionally expanded training corpus and computing power to realize capability gains.

The effectiveness of prompt engineering is rooted in the large language model's In-Context Learning capability, which enables task switching without modifying model weights.

TabPFN (Tabular Prior-Data Fitted Network) was proposed by Hollmann et al. in 2023 and pre-trains a Transformer on a large number of synthetic tabular datasets to learn the meta-task of tabular prediction.

In-Context Learning (ICL) refers to the ability of large language models to perform new tasks without updating weights, solely by providing examples or instructions in the input prompt; this ability was first systematically described in the GPT-3 paper.

Few-shot examples are "user question—ideal answer" pairs embedded in the prompt to demonstrate the expected answer style, format, and reasoning pattern, achieving in-context learning without fine-tuning.

In-context learning in the tabular domain typically relies on attention mechanisms that let the prediction sample interact directly with the training samples in the context, equivalent to soft retrieval weighting at inference time.

After switching to frontier LLMs, the team gained two additional benefits, in-context learning and continual learning: the robot can quickly understand new tasks through a small number of examples without retraining; when encountering new situations, the large model intervenes to adjust the entire pipeline, enabling the system to continuously evolve over deployment time.

The training of deep learning models is not a fully interpretable process, and researchers often discover capabilities that were not explicitly designed during post-training evaluations, such as in-context learning, chain-of-thought reasoning, tool-calling tendencies, and so on.

A user's negative tone in a conversation may trigger a more direct or blunt response style from the model through in-context learning mechanisms, rather than the model developing a long-term "personality."

Eric Bigelow's broad definition of in-context learning is: any process that does not occur inside the model's weights but dynamically changes the model's behavior at runtime.

The inference paradigm adopted by TabFM is In-Context Learning (ICL), a concept first systematically demonstrated in GPT-3.

During the pretraining phase, the model uses labeled contexts along with their true thresholds, while during inference it infers entirely new thresholds from context alone

Frank Hutter believes that transfer in tabular data should occur at the pattern level rather than the row level, and that in-context learning is the key technique for achieving this transfer.

TabPFN is a new-generation tabular foundation model centered on in-context learning, pre-trained on a massive number of diverse synthetic tabular tasks.

TabICL (Tabular In-Context Learning) transforms tabular prediction problems into in-context learning tasks, requiring only a single inference in practice, eliminating the need for training and hyperparameter search.

The continuation of an anthropomorphic style across a multi-agent framework is essentially a manifestation of in-context learning at the stylistic level, rather than genuine intent transmission or emotional resonance between agents.

Induction heads are an important clue for understanding the capability of in-context learning.

GPT-3之后研究者发现通过精心设计的提示，单一大模型即可零样本或少样本完成多类任务，这一范式被称为In-Context Learning

少样本示例（Few-shot Prompting）的工作原理基于Transformer注意力机制，无需更新模型权重，被称为上下文学习（In-context Learning）