MLM
A self-supervised pretraining objective that randomly masks part of the tokens in an input sequence and has the model reconstruct the masked portions from context. It is widely used in the pretraining of Transformer encoders.
Timeline (last 90 days)
On multiple downstream evaluation tasks in the study, none of the JEPA variants outperformed standard MLM.
Reconstructing individual events may cause MLM to overemphasize noisy or low-information observations and waste its capacity on low-value signals.
MLM does not explicitly encourage the model to learn high-level patient states (such as overall disease progression or risk stratification), which are the abstractions most needed for clinical downstream tasks.
For practitioners, in the short term MLM remains a reliable baseline for longitudinal EHR pretraining.
BERT's MLM performs one-shot prediction and is primarily used for representation learning, whereas discrete diffusion models treat the masking ratio as a continuous variable from 0% to 100% to construct a complete Markov chain, achieving progressive generation through multi-step iteration.
All Facts (5)
On multiple downstream evaluation tasks in the study, none of the JEPA variants outperformed standard MLM.
50%UnverifiedReconstructing individual events may cause MLM to overemphasize noisy or low-information observations and waste its capacity on low-value signals.
50%UnverifiedMLM does not explicitly encourage the model to learn high-level patient states (such as overall disease progression or risk stratification), which are the abstractions most needed for clinical downstream tasks.
50%UnverifiedBERT's MLM performs one-shot prediction and is primarily used for representation learning, whereas discrete diffusion models treat the masking ratio as a continuous variable from 0% to 100% to construct a complete Markov chain, achieving progressive generation through multi-step iteration.
50%UnverifiedFor practitioners, in the short term MLM remains a reliable baseline for longitudinal EHR pretraining.
50%