CLAP
Contrastive Language-Audio Pretraining (CLAP) is a framework that aligns audio and text representations through contrastive learning. It supports zero-shot audio classification and audio-text retrieval tasks.
Timeline (last 90 days)
The embedding spaces of multimodal contrastive models (such as CLIP and CLAP) exhibit a modality gap: image and text embedding vectors each occupy their own region on the unit sphere and are difficult to align.
The method is purely geometric and model-agnostic, and can be plugged into existing CLIP/CLAP-type models without relying on a specific contrastive loss or network architecture.
Audio conditioning techniques typically rely on pretrained audio encoders (such as CLAP or EnCodec) to compress reference audio into embedding vectors, which are then injected into the generative network via cross-attention or AdaLN mechanisms.
Sound Captioning and Event Localization are two core subtasks in the field of audio understanding, typically relying on multimodal models pretrained on large-scale audio-text pairs.
All Facts (4)
The embedding spaces of multimodal contrastive models (such as CLIP and CLAP) exhibit a modality gap: image and text embedding vectors each occupy their own region on the unit sphere and are difficult to align.
50%UnverifiedThe method is purely geometric and model-agnostic, and can be plugged into existing CLIP/CLAP-type models without relying on a specific contrastive loss or network architecture.
50%UnverifiedAudio conditioning techniques typically rely on pretrained audio encoders (such as CLAP or EnCodec) to compress reference audio into embedding vectors, which are then injected into the generative network via cross-attention or AdaLN mechanisms.
50%UnverifiedSound Captioning and Event Localization are two core subtasks in the field of audio understanding, typically relying on multimodal models pretrained on large-scale audio-text pairs.
50%