clap-htsat-unfused

Name: clap-htsat-unfused
Author: laion

by laion

CLAP: contrastive language-audio pretraining for text-to-audio search

792Kdl/month

79likes

HuggingFace Run it on your own data, free

Identifiers

Model ID

laion/clap-htsat-unfused

Feature URI

mixpeek://audio_extractor@v1/laion_clap_unfused_v1

Overview

CLAP is CLIP for sound. It trains an audio encoder and a text encoder into one shared embedding space, so you can search an audio library with a natural language description and get back matching clips with no labels and no transcript.

That distinction matters. Speech-to-text makes spoken words searchable and tells you nothing about a door slam, a smoke alarm, applause, or an engine note. Those are acoustic events, not language, and a transcript never contains them. CLAP embeddings capture them, which makes a query like glass breaking or crowd cheering in a stadium possible over raw audio.

On Mixpeek this complements transcription rather than replacing it: transcripts cover what was said, CLAP covers what was heard.

Architecture

HTSAT (hierarchical token-semantic audio transformer) audio encoder paired with a RoBERTa text encoder, projected into a shared embedding space by a contrastive objective. The unfused variant uses the audio encoder alone, without the feature-fusion mechanism, which is simpler and faster for fixed-length clips.

Mixpeek SDK Integration

// No extractor parameter takes a Hugging Face model id (checked against
// GET /v1/discovery/extractors, which returns 13), so clap-htsat-unfused runs
// on your side and the output is upserted through POST
// /v1/namespaces/{namespace_id}/documents/upsert. On Enterprise the other
// path is to upload the weights instead: POST /v1/namespaces/{id}/models
// accepts the huggingface format and a custom plugin loads them.
const res = await fetch(
  "https://api.mixpeek.com/v1/namespaces/ns_your_namespace/documents/upsert",
  {
    method: "POST",
    headers: {
      Authorization: "Bearer API_KEY",
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      collection_id: "col_your_collection",
      documents: [
        {
          document_id: "asset-00412",
          // The vector name has to match a vector index on the collection.
          vectors: { "audio-embedding": yourVector },
          payload: { source_key: "archive/2026/asset-00412" },
        },
      ],
    }),
  },
);

// Managed alternative, if this exact model is not the requirement:
// audio_fingerprint_extractor@v1 runs laion/clap-htsat-tiny
// (512-d) over a bucket, with no inference of your own.