Buckets
Schema-backed containers for organizing multimodal objects with automatic validation and lineage tracking
Video Overview
Why do anything?
Organizing multimodal data (videos, images, documents, audio) requires consistent structure and validation before processing. Without buckets, data pipelines break when inputs don't match expected formats.
Why now?
AI applications need reliable data staging areas. Manual validation is error-prone and doesn't scale. Collections need structured inputs to process data correctly.
Why this feature?
Schema-backed buckets validate blob types (text, image, video, audio, JSON) before storage, ensure metadata consistency, and maintain complete lineage from source to processed documents. Works with S3, MinIO, LocalStack, and direct uploads.
How It Works
Buckets are schema-backed containers that organize raw multimodal inputs before processing. They enforce validation, maintain metadata, and provide a staging area for collections to transform objects into searchable documents.
Schema Definition
Define blob properties (text, image, video, audio, JSON) with validation rules (required, enum, description)
Object Registration
Upload objects with blobs that match schema properties, include metadata for downstream use
Blob Validation
Validate each blob against schema: check type, required fields, and content format
Storage & Indexing
Store blob metadata in MongoDB, save blob content to S3/MinIO/LocalStack, index by key_prefix
Lineage Tracking
Assign object_id, track root_object_id and root_bucket_id for complete decomposition history
Collection Integration
Feed validated objects to collections for feature extraction and document creation
Why This Approach
Schema validation at ingestion prevents downstream processing failures. Separation of storage (buckets) from processing (collections) enables reusing the same data across multiple feature extractors without re-uploading. Lineage tracking maintains provenance from raw input to final enriched document.
Integration
from mixpeek import Mixpeek
client = Mixpeek(api_key="YOUR_API_KEY")
# Create bucket with schema
bucket = client.buckets.create(
bucket_name="product_catalog",
description="E-commerce product data",
bucket_schema={
"properties": {
"product_text": {
"type": "text",
"required": True,
"description": "Product description"
},
"hero_image": {
"type": "image",
"required": True
},
"spec_sheet": {
"type": "json"
}
}
}
)
# Upload object to bucket
object_response = client.buckets.objects.create(
bucket_id=bucket.bucket_id,
key_prefix="/products/red-sneaker",
metadata={
"category": "footwear",
"brand": "Acme",
"sku": "SNK-001"
},
blobs=[
{
"property": "product_text",
"type": "text",
"data": "Comfortable red sneaker with foam sole."
},
{
"property": "hero_image",
"type": "image",
"data": "https://cdn.example.com/images/red-sneaker.jpg"
},
{
"property": "spec_sheet",
"type": "json",
"data": {
"size_range": "6-13",
"colors": ["red", "blue"],
"weight_oz": 12
}
}
]
)
# List objects with filters
objects = client.buckets.objects.list(
bucket_id=bucket.bucket_id,
filters={
"AND": [
{"field": "metadata.category", "operator": "eq", "value": "footwear"}
]
},
page_size=50
)Comparisons & Alternatives
vs. Direct S3/GCS uploads
Key Differences
- Built-in schema validation vs. manual validation
- Automatic lineage tracking vs. custom metadata management
- Integration with Mixpeek collections vs. separate processing setup
- Unified API for multimodal types vs. format-specific handling
When to Use
When you need validated, schema-backed storage that feeds directly into ML pipelines
When NOT to Use
If you only need raw file storage without processing pipelines
vs. Database tables (PostgreSQL, MongoDB)
Key Differences
- Blob-optimized storage (S3) vs. database storage
- Multimodal support (video, image, audio) vs. structured data only
- Feature extraction integration vs. manual processing
- Presigned URL access vs. direct database queries
When to Use
For multimodal data that needs ML processing and feature extraction
When NOT to Use
For purely structured relational data without media files
vs. Data lakes (Databricks, Snowflake)
Key Differences
- Purpose-built for multimodal AI vs. general analytics
- Real-time feature extraction vs. batch ETL
- Direct collection integration vs. external connector setup
- Embedded lineage tracking vs. separate data catalog
When to Use
When building AI applications that need instant multimodal processing
When NOT to Use
For business intelligence dashboards and reporting use cases
Resources
This capability is referenced and used across the following resources:
Recipes (2)
Solutions (3)
Tutorials (2)
University Modules (1)
Recent updates
Full changelog- Sep 26, 2026An object deleted during a batch stays deletedWhen an object was deleted while its batch was still queued, the delete removed the object's row from the bucket's system collection and the batch then wrote it back, so the deleted object showed up again in the bucket's listing. When a tier finishes, the rows of any batch objects deleted in the meantime are now removed.
- Jul 25, 2026Create Bucket points at the schema templates instead of asking for a propertyStep 2 of the Create Bucket wizard blocks until the object schema has at least one property, and the block read `Please define at least one property in the object schema`. One-click templates for Videos, Images, Documents, and Products sit directly above that step, and nothing connected the two. The message now reads `Add at least one field, or pick a template above (Videos, Images, Documents) to fill in a starting schema you can tweak`.