BERT
BERT (Bidirectional Encoder Representations from Transformers) is a pre-trained language representation model proposed by Google, based on a bidirectional Transformer encoder architecture. Its core pre-training tasks include Masked Language Modeling (MLM) and Next Sentence Prediction (NSP), using self-supervised learning on large-scale unlabeled text to obtain deep bidirectional semantic representations. After fine-tuning, BERT can be applied to a wide range of downstream NLP tasks including text classification, named entity recognition, and question answering.
Core Facts
Large language models such as GPT and BERT are based on the self-supervised learning paradigm
BERT was introduced by Google in 2018, a pioneering work in pre-trained language models, and was the first to adopt a bidirectional encoder architecture, setting new records on 11 NLP benchmark tests.
当前所有主流大模型（GPT、BERT、LLaMA等）都基于Transformer架构
Timeline (last 90 days)
Implementing negation detection typically requires syntactic analysis or a semantic classifier based on a pretrained model (such as BERT), rather than simple string matching.
Legora's initial idea was born during the period when Google's BERT model was openly accessible, but the early use cases were too difficult and it only worked once GPT-3.5 appeared.
BERT is essentially the Transformer encoder with a few small modifications: sentence pairs are packed into a single input sequence, segment embeddings are added to distinguish the two sentences, and two small output heads are added on top for masked-word prediction and next-sentence prediction respectively.
When fine-tuning BERT, if the task does not involve sentence pairs, segment embeddings are usually all set to zero.
Segment embeddings are special vectors in BERT used to distinguish sentence pairs, added to token embeddings and position embeddings before being fed into the encoder.
Jev's Encoder implementation route uses a BERT-style encoder backbone, with the question first and the context after in token order, inserting a mask token before each option and taking its hidden state, then applying a dot product with the forward pass and softmax to obtain option probabilities; with zero training the results are inaccurate—a question that should be "yes" is given only about a 30% probability.
Decision models are typically based on BERT-like encoders or smaller specialized networks and perform only a single forward computation, so latency and inference cost are far lower than LLM calls on the same task.
The Jev-style System One model is a class of discriminative architectures that do not rely on autoregressive generation but instead directly compute the conditional probability of each candidate answer and take the maximum, typically based on an encoder structure (such as BERT-style).
Unlike BERT, which usually makes only one prediction, a dLLM starts from full masking or a high masking rate and iterates repeatedly, gradually reducing the mask until the entire sentence is filled in.
The backbone of a diffusion language model (dLLM) is highly similar to that of a BERT-style masked language model: during training, a portion of the tokens is randomly masked, and the model restores them based on the remaining context.
40 more timeline events
All Facts (20)
Large language models such as GPT and BERT are based on the self-supervised learning paradigm
90%VerifiedBERT was introduced by Google in 2018, a pioneering work in pre-trained language models, and was the first to adopt a bidirectional encoder architecture, setting new records on 11 NLP benchmark tests.
80%Verified当前所有主流大模型（GPT、BERT、LLaMA等）都基于Transformer架构
80%VerifiedGLM采用了独特的自回归填空预训练范式，与GPT的纯自回归和BERT的纯掩码语言模型都有所不同
80%VerifiedThe Transformer architecture proposed by Google in 2017 laid the foundation for large language models.
80%Verified大语言模型的预训练阶段本质上属于自监督学习，是一种特殊的非监督学习，模型通过预测下一个词来从海量文本中自动学习语言规律
80%VerifiedBERT was proposed by Google in 2018 and is a pre-trained language model based on the Transformer architecture.
80%VerifiedBERT (Bidirectional Encoder Representations from Transformers) is a pre-trained language model released by Google in 2018.
75%VerifiedGLM uses an autoregressive blank-filling training objective, randomly masking text segments and requiring the model to autoregressively restore them.
75%VerifiedModern LLMs (such as the GPT series, Claude, and Llama) are all based on the Transformer architecture, and their decoding process is essentially a Markov chain.
75%VerifiedDistilBERT reduces BERT's parameter count by 40% through distillation, increases inference speed by 60%, and incurs only approximately 3% performance loss
75%VerifiedBoth the GPT series and the Claude series employ a tokenization mechanism similar to Byte-Pair Encoding (BPE).
75%VerifiedBERT uses an encoder architecture, the GPT series uses a decoder architecture, and T5 and BART use an encoder-decoder architecture
75%VerifiedBERT, the GPT series, LLaMA, and Claude are all built on the Transformer architecture.
70%VerifiedThe core innovation of the Transformer is the self-attention mechanism, and mainstream large models such as GPT, Claude, and Gemini are all built on the Transformer.
70%VerifiedBERT was proposed by Google in 2018, and its bidirectional attention mechanism makes it excel at tasks such as text classification and semantic understanding.
70%VerifiedModernBERT is an improved version of the BERT architecture released in late 2024, introducing modern designs such as Rotary Position Embedding (RoPE) and alternating local/global attention.
65%VerifiedThe original Transformer consists of an Encoder and a Decoder; BERT-type models use only the Encoder and excel at understanding and classification, while GPT-type models use only the Decoder and focus on text generation.
65%VerifiedThe Transformer architecture has diverged into two major lineages: BERT-style (Encoder-only, suited for understanding tasks) and GPT-style (Decoder-only, suited for generation tasks)
65%VerifiedBERT was proposed by Google in 2018 and is a bidirectional encoder; GPT was proposed by OpenAI in 2018 and is an autoregressive decoder.
65%