clap-htsat-unfused

by laion

CLAP: contrastive language-audio pretraining for text-to-audio search

462Kdl/month

78likes

HuggingFace Run on your data

Identifiers

Model ID

laion/clap-htsat-unfused

Feature URI

mixpeek://audio_extractor@v1/laion_clap_unfused_v1

Overview

CLAP is CLIP for sound. It trains an audio encoder and a text encoder into one shared embedding space, so you can search an audio library with a natural language description and get back matching clips with no labels and no transcript.

That distinction matters. Speech-to-text makes spoken words searchable and tells you nothing about a door slam, a smoke alarm, applause, or an engine note. Those are acoustic events, not language, and a transcript never contains them. CLAP embeddings capture them, which makes a query like glass breaking or crowd cheering in a stadium possible over raw audio.

On Mixpeek this complements transcription rather than replacing it: transcripts cover what was said, CLAP covers what was heard.

Architecture

HTSAT (hierarchical token-semantic audio transformer) audio encoder paired with a RoBERTa text encoder, projected into a shared embedding space by a contrastive objective. The unfused variant uses the audio encoder alone, without the feature-fusion mechanism, which is simpler and faster for fixed-length clips.

Mixpeek SDK Integration

import { Mixpeek } from "mixpeek";

const mx = new Mixpeek({ apiKey: "API_KEY" });

// Managed: create a collection over a bucket; Mixpeek runs this model's extractor
const collection = await mx.collections.create({
  namespace_id: "my-namespace",
  collection_name: "my-collection",
  source: { type: "bucket", bucket_ids: ["bkt_your_bucket"] },
  feature_extractor: {
    feature_extractor_name: "audio_embedding",
    version: "v1",
    parameters: { model_id: "laion/clap-htsat-unfused" },
  },
});