Contrastive Learning

The CLIP architecture was proposed by OpenAI in 2021 and maps images and text to the same high-dimensional vector space through contrastive learning.

The core loss functions of contrastive learning (such as InfoNCE and NT-Xent) essentially pull the representations of similar samples closer together while pushing the representations of dissimilar samples further apart.

During fine-tuning, hard negatives (e.g., positions in the same field but at a different seniority level) should be used instead of random negatives to obtain maximum gains.

Embedding model fine-tuning usually adopts a contrastive learning framework: semantically similar sample pairs serve as positive examples and unrelated ones as negative examples, optimizing retrieval ranking ability by pulling positive examples closer and pushing negative examples farther apart.

Contrastive learning is the core paradigm for fine-tuning dense retrieval models, pulling a query and its positive document closer in vector space while pushing the query away from other documents in the batch (negatives).

The key technologies for achieving multimodal fusion include cross-modal attention mechanisms and contrastive learning, enabling the model to learn the associations between different data sources.

The paper "SimOPD: Relational Knowledge Transfer for On-Policy Distillation" proposes the SimOPD method, drawing on the use of inter-instance relations in contrastive learning to transfer the teacher's internal knowledge through the similarity of student-generated responses.

Contrastive signals are more robust to thresholds and scales, and are easier to read directly from the editor's existing feature space, without having to additionally train an independent scoring network.

The typical pipeline of font embedding includes: data preparation (collecting TTF/OTF font files and rendering characters), feature extraction (encoding into vectors with a CNN or VAE), training the embedding space (contrastive learning or autoencoder reconstruction loss), dimensionality reduction visualization (t-SNE, UMAP or PCA), and observing the structure.

The Transformer architecture, contrastive learning, and reinforcement learning with human feedback (RLHF) are paradigm-level innovations.

The paper connects three mathematical tools—contrastive learning, exponential family distributions, and information geometry—to establish a rigorous mapping between probability distributions over anchor points and Bregman geometry over the representation space.

The paper proposes that representations obtained through contrastive learning inherently contain a family of geometries rather than a single geometry, and that the same set of fixed vector representations can be specialized to different semantic structures without retraining the model.

Transformer architecture and contrastive learning have been used to create embedding representations of sperm whale coda sequences.

Embedding model fine-tuning typically adopts a contrastive learning framework, training the model to bring positive sample pairs closer together and push negative sample pairs apart by constructing positive and negative sample pairs.

Geospatial foundation models typically use the Transformer architecture as their core, employing self-supervised learning methods for pre-training on large volumes of remote sensing imagery, with typical methods including masked autoencoders (MAE) and contrastive learning.

Contrastive learning naturally encourages models to capture the most discriminative semantic features in an image, rather than fine-grained spatial and quantitative information.

Contrastive learning is the core paradigm for implementing negative sample learning, which learns features by pulling the representation distance of positive pairs closer and pushing the representation distance of negative pairs farther apart.

Contrastive learning typically uses the InfoNCE loss function, which originates from the theory of Noise Contrastive Estimation.

The essential difference between JEPA and contrastive learning is that JEPA learns by predicting the representations of missing parts, without relying on explicit data augmentation strategies.

Representation collapse is a widely studied phenomenon in deep learning, and is especially common in self-supervised learning and contrastive learning frameworks.