Self-supervised Learning

大语言模型的预训练阶段本质上属于自监督学习，是一种特殊的非监督学习，模型通过预测下一个词来从海量文本中自动学习语言规律

In-context learning was first observed in large language models, and its capability fundamentally stems from the diversity of pretraining data.

Foundation models learn by predicting the 'next word' on massive amounts of unlabeled text using a self-supervised learning paradigm, represented by models such as the GPT series and LLaMA

Abandoning text supervision means the model can only rely on the structure of the acoustic signal itself for self-supervised learning, with common approaches including masked prediction, contrastive learning and prediction targets based on dialogue turns.

Examples of technical-route debates include: in model architecture, the debate between Transformer and hybrid architectures that alternate convolution and attention; in training methods, the trade-off between self-supervised learning and supervised learning; in deployment strategy, the weighing of cloud large models against on-device small models.

Conventional self-supervision relies on externally constructed proxy tasks (such as masked prediction and contrastive learning), whereas MetaPred's self-supervision signal comes from measurements of the state itself.

The current framework's feature set is still manually designed, and whether it can be combined with self-supervised learning in the future to improve performance while maintaining interpretability is an open question worth attention in this direction.

In the AI field, ideas similar to "play-learn" are embodied in unsupervised pretraining and self-supervised learning: a model first learns general representations on unlabeled data, then transfers to downstream tasks.

Transfer learning or self-supervised learning can improve model robustness when labeled samples are limited, but it also means model interpretability is compromised—a real constraint for products that need to give clear agricultural recommendations.

Due to the lack of large-scale labeled data, researchers often rely on transfer learning or self-supervised learning, first pretraining representations on massive unlabeled signals and then fine-tuning classifiers with a small amount of controlled experimental data.

Self-supervised learning does not require manual annotation; instead, it constructs supervision signals by predicting the next token from the given preceding context, which is the key to scaling up LLM training.

Common methods in academia and industry for addressing few-shot defect detection include anomaly detection (such as autoencoder reconstruction error methods), data augmentation and synthetic defect generation, transfer learning, and semi-supervised/self-supervised learning.

The study adopts a self-supervised approach that automatically generates training labels for wheel tracks by back-projecting the vehicle's future trajectory onto the current image frame using GNSS data.

Using 'future trajectories' as labels is based on the assumption that the vehicle's current driving path is safely traversable, which holds in the scenario of following existing wheel tracks but may introduce noisy labels when breaking new ground.

Unsupervised or self-supervised learning paradigms are particularly critical for studying non-human communication that lacks parallel translations.

The mainstream approaches to depth estimation fall into two main categories: geometry-based multi-view stereo (MVS) and deep-learning-based monocular depth estimation.

This approach is essentially a self-supervised learning application that uses the generation process itself to produce perfectly aligned supervision signals.

Self-supervised TTT work can be traced back to around 2020, enabling models to adapt to the current input by constructing auxiliary self-supervised tasks (such as rotation prediction) on test samples.

Current mainstream large language models and AI systems are based on a training approach that combines self-supervised learning on massive datasets with reinforcement learning

By constructing supervisory signals from the data itself, self-supervised learning has become the core training strategy for large-scale pre-trained models such as BERT, GPT, and MAE.