ModernBERT
An improved encoder model in the BERT architecture family released in late 2024. While retaining the advantages of bidirectional attention, it incorporates modern design elements such as Rotary Position Embeddings (RoPE) and alternating local/global attention, making it suitable for lightweight encoder tasks.
Core Facts
Timeline (last 90 days)
In the course, the routing layer does not use LLM calls but is built on small encoders (ModernBERT, SetFit), runs on CPU, and keeps latency under 15 milliseconds.
GitHub replaced the general-purpose LLM in secret scanning with a fine-tuned ModernBERT.
Compared with the old LLM pipeline, GitHub's new secret scanning approach improves offline Precision by 13% and Recall by 19%; online A/B experiments show that median post-push scan execution time drops by 86%, model serving cost drops by more than 98%, and the repository-owner-weighted alert acceptance metric improves by 11.5%.
ModernBERT's NFC path already skips ASCII on the main branch, so its English results gain only 1.06x from the optimizations on that branch.
ModernBERT replaces absolute position encoding with rotary position encoding (RoPE), uses FlashAttention for acceleration, and extends the maximum context window from BERT's 512 tokens to 8192 tokens.
A Reddit user fine-tuned ModernBERT with 138,000 financial news headlines, and the training loss kept decreasing but the test set AUC stayed between 0.49 and 0.51.
ModernBERT is an improved version of the BERT architecture released in late 2024, introducing modern designs such as Rotary Position Embedding (RoPE) and alternating local/global attention.
All Facts (8)
ModernBERT is an improved version of the BERT architecture released in late 2024, introducing modern designs such as Rotary Position Embedding (RoPE) and alternating local/global attention.
65%UnverifiedSparky的创造者Alexis之前的工作是训练大语言模型（encoder-only架构，类似ModernBERT）
85%UnverifiedIn the course, the routing layer does not use LLM calls but is built on small encoders (ModernBERT, SetFit), runs on CPU, and keeps latency under 15 milliseconds.
50%UnverifiedGitHub replaced the general-purpose LLM in secret scanning with a fine-tuned ModernBERT.
50%UnverifiedCompared with the old LLM pipeline, GitHub's new secret scanning approach improves offline Precision by 13% and Recall by 19%; online A/B experiments show that median post-push scan execution time drops by 86%, model serving cost drops by more than 98%, and the repository-owner-weighted alert acceptance metric improves by 11.5%.
50%UnverifiedModernBERT's NFC path already skips ASCII on the main branch, so its English results gain only 1.06x from the optimizations on that branch.
50%UnverifiedModernBERT replaces absolute position encoding with rotary position encoding (RoPE), uses FlashAttention for acceleration, and extends the maximum context window from BERT's 512 tokens to 8192 tokens.
50%UnverifiedA Reddit user fine-tuned ModernBERT with 138,000 financial news headlines, and the training loss kept decreasing but the test set AUC stayed between 0.49 and 0.51.
50%