Training
Scalable Multimodal Processing
Sharded collections + batch pipelines + manifest tracking. This recipe is operational, not algorithmic—it's about infrastructure.
video
image
audio
text
Production
89.0K runs
Deploy Recipe"Process 10 million images from S3 manifest with automatic sharding, retries, and idempotency"
Why This Matters
Scale is an infrastructure problem. Mixpeek handles sharding, retries, and idempotency so you can focus on queries, not operations.
from mixpeek import Mixpeekclient = Mixpeek(api_key="your-api-key")# Create sharded collection for scalecollection = client.collections.create(collection_name="production_media",feature_extractor={"feature_extractor_name": "multimodal_extractor","version": "v1"},sharding={"enabled": True,"strategy": "hash","num_shards": 32})# Batch ingest from manifestmanifest = client.manifests.create(manifest_name="q4_uploads",source_uri="s3://bucket/manifest.json")# Process at scale with retriesjob = client.jobs.create(collection_id=collection.id,manifest_id=manifest.id,batch_size=500,max_retries=3,idempotency_key="q4_uploads_v1")# Monitor progressstatus = client.jobs.get_status(job_id=job.id)print(f"Processed: {status['completed']}/{status['total']}")
Feature Extractors
Feature Extractors
Image Embedding
Generate visual embeddings for similarity search and clustering
752K runs
Video Embedding
Generate vector embeddings for video content
610K runs
Audio Transcription
Transcribe audio content to text
450K runs
Text Embedding
Extract semantic embeddings from documents, transcripts and text content
827K runs