SenseVoiceSmall

Name: SenseVoiceSmall
Author: FunAudioLLM

by FunAudioLLM

Non-autoregressive speech understanding: transcription, language ID, emotion and audio events in one pass

4.0Kdl/month

Not published as a number; the model card states a parameter count similar to Whisper-Smallparams

HuggingFace Run it on your own data, free

Identifiers

Model ID

FunAudioLLM/SenseVoiceSmall

Feature URI

mixpeek://transcription@v1/alibaba_sensevoice_small_v1

Overview

SenseVoice is a speech foundation model that does four things in one forward pass: automatic speech recognition, spoken language identification, speech emotion recognition, and audio event detection. The Small variant is non-autoregressive, which is where its latency comes from: the model card reports 70ms to process 10 seconds of audio and describes that as 15 times faster than Whisper-Large.

The reason to reach for it over Whisper is the extra channels rather than the transcript. A Whisper transcript tells you what was said. SenseVoice additionally labels how it was said and what else was audible, tagging events like background music, applause, laughter, crying, coughing and sneezing. For a retrieval corpus that turns 'find the part where the audience laughs' into a filter rather than a guess.

It ships through FunASR rather than transformers, with ONNX and libtorch exports and a GGUF build that runs as a self-contained binary under llama.cpp, so CPU-only and edge deployments do not need a Python runtime.

Architecture

Non-autoregressive end-to-end speech foundation model, trained on over 400,000 hours of audio across more than 50 languages. A single forward pass emits the transcript plus language, emotion and audio-event labels, rather than decoding token by token, which is what removes the autoregressive latency. Distributed via the FunASR toolkit; ONNX, libtorch and GGUF runtimes are published alongside the PyTorch weights.

Mixpeek SDK Integration

// No extractor parameter takes a Hugging Face model id (checked against
// GET /v1/discovery/extractors, which returns 13), so SenseVoiceSmall runs
// on your side and the output is upserted through POST
// /v1/namespaces/{namespace_id}/documents/upsert. On Enterprise the other
// path is to upload the weights instead: POST /v1/namespaces/{id}/models
// accepts the huggingface format and a custom plugin loads them.
const res = await fetch(
  "https://api.mixpeek.com/v1/namespaces/ns_your_namespace/documents/upsert",
  {
    method: "POST",
    headers: {
      Authorization: "Bearer API_KEY",
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      collection_id: "col_your_collection",
      documents: [
        {
          document_id: "call-00412",
          // The extra channels are the point: keep emotion, language and the
          // event labels as payload fields so they are filterable, and embed
          // the transcript so it is searchable by meaning.
          payload: {
            transcript: modelOutput.text,
            language: modelOutput.language,
            emotion: modelOutput.emotion,
            audio_events: modelOutput.events,
          },
          vectors: { "text-embedding": embeddingOfTranscript },
        },
      ],
    }),
  },
);

// Managed alternative, if this exact model is not the requirement:
// multimodal_extractor@v2 runs Whisper transcription over a bucket with
// run_transcription, with no inference of your own. It produces the
// transcript but not the emotion or audio-event channels.

Capabilities

Transcription across 50+ languages, with Mandarin, Cantonese, English, Japanese and Korean called out on the model card
Spoken language identification without being told the language
Speech emotion recognition on the same pass as the transcript
Audio event detection: background music, applause, laughter, crying, coughing, sneezing
70ms per 10s of audio, reported by the authors as 15x faster than Whisper-Large
CPU and edge deployment via a GGUF build under llama.cpp, no Python runtime required

Use Cases on Mixpeek

Search a call archive by emotion as well as by words

Find audience reaction moments in recorded events by the event label rather than the transcript

Transcribe a mixed-language corpus where the language is not known per file

Cheap first-pass transcription on CPU where a GPU budget for Whisper is not available

Performance

Input Size10s audio

GPU LatencyInput dependent

GPU ThroughputBatch dependent

GPU MemoryModel dependent

70ms per 10s of audio, reported on the model card as 15x faster than Whisper-Large. Non-autoregressive, so latency does not scale with transcript length the way a decoder does. We have not published our own measurement on Mixpeek infrastructure, so no figure is quoted here.