Audio Sentiment Extractor

​

Pipeline Steps

1. Filter Dataset (if collection_id provided)
   • Filter to specified collection
2. Apply Input Mappings
   • Resolve audio/video field from source (e.g., payload.audio_url, payload.webcast_url)
3. Audio Extraction (conditional: if video input)
   • FFmpeg strips audio track from MP4/MOV; supports AAC, MP3, FLAC output
4. Voice Activity Detection + Segmentation
   • split_method: time — fixed-length windows (default 30s)
   • split_method: silence — split at natural speech pauses (VAD threshold configurable)
   • split_method: speaker — one segment per speaker turn (requires run_diarization=true)
5. Speaker Diarization (conditional: if run_diarization=true)
   • pyannote.audio 3.x pipeline separates speakers (CEO, CFO, Analyst_1, etc.)
   • Assigns speaker_id and optionally maps to speaker_role via role manifest
6. Transcription (conditional: if run_transcription=true)
   • Whisper large-v3-turbo speech-to-text with financial vocabulary prompt
   • Per-segment timestamps aligned to diarization boundaries
7. FinBERT Text Sentiment (conditional: if run_finbert=true)
   • ProsusAI/FinBERT financial-domain sentiment classifier
   • Outputs sentiment_label (positive/negative/neutral), sentiment_score (-1 to +1), confidence
   • Generates 768D FinBERT CLS embedding for semantic search
8. Prosodic Feature Extraction (conditional: if run_prosodics=true)
   • LibROSA + Parselmouth extract 5 features per segment:
     • Pitch variability (F0 standard deviation, Hz) — hesitation and stress indicator
     • Speech rate (words per minute) — confidence and urgency signal
     • Vocal energy (RMS dB) — assertiveness and emotional weight
     • Pause ratio (fraction of silence) — cognitive load and evasiveness marker
     • Vocal tremor (jitter + shimmer) — anxiety and deception indicator
   • Normalized into a 128D prosodic embedding for similarity search
9. Audio-Text Alignment Score (conditional: if run_alignment=true)
   • Cosine similarity between FinBERT sentiment direction and prosodic valence
   • Low alignment = voice contradicts words (high-value deception/stress signal)
10. LLM Structured Enrichment (conditional: if run_llm_enrichment=true or response_shape set)
    • Gemini/GPT-4o processes transcription with custom prompt
    • Extracts structured signals: guidance confidence, topic classification, hedging language
11. Output
    • Per-segment documents with both embedding types, raw prosodic features, sentiment scores, speaker metadata, and computed alpha signals

​

When to Use

​

When NOT to Use

​

Supported Input Types

​

Input Schema

{
  "audio": "s3://bucket/earnings/AAPL_Q4_2024.mp3"
}

{
  "video": "s3://bucket/investor-day/msft-2024-ceo-keynote.mp4"
}

{
  "url": "https://edge.media-server.com/mmc/p/xyz/earnings-call.mp3"
}

​

Output Schema

{
  "start_time": 245.0,
  "end_time": 275.0,
  "speaker_id": "SPEAKER_00",
  "speaker_role": "CEO",
  "transcription": "We're very confident in our Q1 guidance range of twelve to fourteen dollars per share...",
  "sentiment_label": "positive",
  "sentiment_score": 0.71,
  "sentiment_confidence": 0.89,
  "pitch_variability_hz": 38.2,
  "speech_rate_wpm": 142,
  "vocal_energy_db": -18.4,
  "pause_ratio": 0.21,
  "vocal_tremor": 0.14,
  "audio_text_alignment": 0.63,
  "stress_index": 0.31,
  "audio_sentiment_extractor_v1_text_embedding": [0.041, -0.018, ...],
  "audio_sentiment_extractor_v1_prosodic_embedding": [0.72, 0.34, ...]
}

​

Parameters

​

Audio Segmentation

{
  "split_method": "time",
  "time_split_interval": 30
}

{
  "split_method": "silence",
  "silence_db_threshold": -40,
  "min_silence_duration_ms": 500
}

{
  "split_method": "speaker",
  "run_diarization": true
}

​

Feature Extraction Parameters

​

Speaker Role Manifest

{
  "speaker_role_manifest": {
    "SPEAKER_00": "CEO",
    "SPEAKER_01": "CFO",
    "SPEAKER_02": "Analyst"
  }
}

​

LLM Structured Extraction

{
  "response_shape": "Extract: forward guidance confidence level (1-5), number of hedging phrases, primary topic discussed, and any mentioned risk factors"
}

{
  "response_shape": {
    "type": "object",
    "properties": {
      "guidance_confidence": { "type": "integer", "minimum": 1, "maximum": 5 },
      "hedging_phrase_count": { "type": "integer" },
      "topic": { "type": "string", "enum": ["revenue", "margins", "guidance", "macro", "capex", "other"] },
      "risk_factors": { "type": "array", "items": { "type": "string" } },
      "quantitative_claims": { "type": "array", "items": { "type": "string" } }
    }
  }
}

​

Configuration Examples

{
  "feature_extractor": {
    "feature_extractor_name": "audio_sentiment_extractor",
    "version": "v1",
    "input_mappings": {
      "audio": "audio_url"
    },
    "field_passthrough": [
      { "source_path": "metadata.ticker" },
      { "source_path": "metadata.fiscal_quarter" },
      { "source_path": "metadata.call_section" }
    ],
    "parameters": {
      "split_method": "speaker",
      "run_transcription": true,
      "transcription_language": "en",
      "run_finbert": true,
      "run_prosodics": true,
      "run_alignment": true,
      "run_diarization": true,
      "num_speakers": 6,
      "speaker_role_manifest": {
        "SPEAKER_00": "CEO",
        "SPEAKER_01": "CFO",
        "SPEAKER_02": "IR"
      }
    }
  }
}

​

Performance & Costs

​

Processing Speed

​

Cost Estimates (per hour of audio)

​

Vector Indexes

​

Text Embedding (FinBERT)

​

Prosodic Embedding

​

Alpha Signal Guide

​

Composite Stress Index

stress_index = normalize(
  0.25 * pitch_variability_z +
  0.20 * speech_rate_z +      # inverted: lower rate = higher stress
  0.20 * (1 - vocal_energy_z) +
  0.20 * pause_ratio_z +
  0.15 * vocal_tremor_z
)

​

Recommended Factor Construction

# Example: CEO guidance confidence factor
import mixpeek

client = mixpeek.Client(api_key="YOUR_KEY")

# Query for CEO guidance segments with sentiment divergence
results = client.retrievers.run(
    retriever_id="earnings-sentiment-retriever",
    inputs={
        "query": "revenue guidance outlook fiscal year",
        "filters": {
            "speaker_role": "CEO",
            "forward_looking": True,
            "audio_text_alignment": {"$lt": 0.3},   # divergence signal
            "stress_index": {"$gt": 0.6}             # high stress
        }
    }
)

​

Limitations

• Speaker diarization accuracy: pyannote achieves ~90% DER on clean 2-speaker recordings; accuracy degrades with > 8 speakers or poor audio quality
• Non-English: Whisper transcription supports 99 languages; FinBERT is English-only — for non-English calls, disable run_finbert and use multilingual sentiment models
• Audio quality: Prosodic features require 16kHz+ audio; compressed phone audio (8kHz) reduces pitch extraction accuracy by ~30%
• Baseline dependency: stress_index Z-score normalization requires at least 4 prior segments from the same speaker_id to be meaningful
• Segment length: Prosodic features are unreliable for segments < 5 seconds; short interjections are best excluded
• LLM enrichment latency: run_llm_enrichment=true adds 1–2s per segment; disable for batch throughput

​

Related

• Feature Extractors Overview
• Text Extractor
• Multimodal Extractor
• Document Extractor