Best S3-Compatible Object Storage for AI Workloads in 2026

Real-World Use Cases

• •Storing and serving 100M+ embedding vectors via MVS with sub-$1K/mo storage costs
• •Backing up model checkpoints during distributed training with free CDN egress for model serving
• •Hosting large multimodal datasets (images, video, audio) for feature extraction pipelines
• •Building a cost-effective RAG knowledge base with vector search layered on top via Mixpeek

Integration Example

import boto3

b2 = boto3.client(
    "s3",
    endpoint_url="https://s3.us-west-004.backblazeb2.com",
    aws_access_key_id="YOUR_B2_KEY_ID",
    aws_secret_access_key="YOUR_B2_APP_KEY",
)

# Upload embeddings file to B2
b2.upload_file("my-ai-bucket", "embeddings/batch_001.parquet", "/tmp/batch_001.parquet")

# Generate presigned URL for model weight download
url = b2.generate_presigned_url(
    "get_object",
    Params={"Bucket": "my-ai-bucket", "Key": "models/v2/weights.safetensors"},
    ExpiresIn=3600,
)

Real-World Use Cases

• •Serving model weights to inference endpoints across multiple regions without egress penalties
• •Hosting a RAG document store where every retrieval query pulls data — zero egress keeps costs flat
• •Running edge AI inference with Cloudflare Workers reading model artifacts from R2
• •Building a global image search API where search results serve original media from R2 at zero transfer cost

Integration Example

import boto3

r2 = boto3.client(
    "s3",
    endpoint_url="https://YOUR_ACCOUNT_ID.r2.cloudflarestorage.com",
    aws_access_key_id="YOUR_R2_ACCESS_KEY",
    aws_secret_access_key="YOUR_R2_SECRET_KEY",
)

# Upload training data — zero egress when models pull it later
r2.upload_file("ai-data", "datasets/train.parquet", "/tmp/train.parquet")

# List model artifacts
response = r2.list_objects_v2(Bucket="ai-data", Prefix="models/v3/")
for obj in response.get("Contents", []):
    print(f"{obj['Key']} — {obj['Size'] / 1e9:.1f} GB")

Real-World Use Cases

• •End-to-end ML pipelines with SageMaker reading training data from S3 and writing model artifacts back
• •Event-driven feature extraction using S3 event notifications triggering Lambda or Step Functions
• •Using S3 Vectors for lightweight vector similarity search without deploying a separate vector database
• •Multi-tier storage for ML artifacts — hot models on Standard, archived checkpoints on Glacier

Integration Example

import boto3

s3 = boto3.client("s3")

# Upload training dataset with Intelligent Tiering
s3.upload_file(
    "/tmp/dataset.parquet",
    "ml-pipeline",
    "datasets/v2/train.parquet",
    ExtraArgs={"StorageClass": "INTELLIGENT_TIERING"},
)

# Set up event notification for new uploads
s3.put_bucket_notification_configuration(
    Bucket="ml-pipeline",
    NotificationConfiguration={
        "LambdaFunctionConfigurations": [{
            "LambdaFunctionArn": "arn:aws:lambda:us-east-1:123:function:process-upload",
            "Events": ["s3:ObjectCreated:*"],
            "Filter": {"Key": {"FilterRules": [{"Name": "prefix", "Value": "datasets/"}]}},
        }]
    },
)

Real-World Use Cases

• •Serving ML model weights globally with automatic edge caching — no manual replication to each region
• •Multi-region RAG deployments where embedding retrieval must be low-latency regardless of user location
• •Distributed training pipelines that need strongly consistent access to shared datasets across regions

Integration Example

import boto3

tigris = boto3.client(
    "s3",
    endpoint_url="https://fly.storage.tigris.dev",
    aws_access_key_id="YOUR_TIGRIS_ACCESS_KEY",
    aws_secret_access_key="YOUR_TIGRIS_SECRET_KEY",
)

# Upload model weights — auto-replicated to nearest edge
tigris.upload_file("ai-models", "llm/v3/weights.safetensors", "/tmp/weights.safetensors")

# Data is automatically cached at the edge closest to readers
obj = tigris.get_object(Bucket="ai-models", Key="llm/v3/weights.safetensors")
print(f"Content-Length: {obj['ContentLength'] / 1e9:.1f} GB")

Real-World Use Cases

• •Long-term archival of training datasets that rarely change — the 90-day minimum is irrelevant for static data
• •Storing pre-computed embedding collections that are written once and read many times
• •Backing up model checkpoints from training runs where predictable cost matters more than retrieval speed

Integration Example

import boto3

wasabi = boto3.client(
    "s3",
    endpoint_url="https://s3.wasabisys.com",
    aws_access_key_id="YOUR_WASABI_KEY",
    aws_secret_access_key="YOUR_WASABI_SECRET",
)

# Upload large training dataset — flat rate, no egress fees on reads
wasabi.upload_file("ml-datasets", "imagenet/train.tar", "/tmp/imagenet_train.tar")

# Verify upload
head = wasabi.head_object(Bucket="ml-datasets", Key="imagenet/train.tar")
print(f"Uploaded: {head['ContentLength'] / 1e9:.1f} GB")

Real-World Use Cases

• •Air-gapped ML environments in defense or healthcare where training data cannot leave the network perimeter
• •On-prem GPU cluster with local MinIO storing training data for zero-network-hop data loading
• •Self-hosted MVS deployment for private vector search over sensitive embeddings
• •Local development and CI/CD environments that need S3-compatible storage without cloud dependencies

Integration Example

import boto3

minio = boto3.client(
    "s3",
    endpoint_url="http://minio.internal:9000",
    aws_access_key_id="minioadmin",
    aws_secret_access_key="minioadmin",
)

# Create bucket for ML artifacts
minio.create_bucket(Bucket="ml-artifacts")

# Upload model weights to self-hosted storage
minio.upload_file("ml-artifacts", "models/resnet50.pt", "/tmp/resnet50.pt")

# Set up event notification for new data ingestion
minio.put_bucket_notification_configuration(
    Bucket="ml-artifacts",
    NotificationConfiguration={
        "QueueConfigurations": [{
            "QueueArn": "arn:minio:sqs::1:webhook",
            "Events": ["s3:ObjectCreated:*"],
        }]
    },
)

Real-World Use Cases

• •Vertex AI training pipelines reading datasets directly from GCS with native integration
• •BigQuery ML workflows that query structured data alongside unstructured media stored in GCS
• •Autoclass-managed storage for mixed-access ML artifacts — hot models tier up, cold checkpoints tier down automatically

Integration Example

from google.cloud import storage

client = storage.Client()
bucket = client.bucket("ml-training-data")

# Upload training dataset with Autoclass tiering
blob = bucket.blob("datasets/v3/train.parquet")
blob.upload_from_filename("/tmp/train.parquet")

# Generate signed URL for model download
url = blob.generate_signed_url(
    version="v4",
    expiration=3600,
    method="GET",
)
print(f"Download URL: {url[:80]}...")

Real-World Use Cases

• •GDPR-compliant storage for medical imaging or biometric datasets that must remain within the EU
• •Budget ML training data storage for European research labs and universities
• •Hosting embedding collections for EU-deployed RAG applications with strict data residency requirements

Integration Example

import boto3

hetzner = boto3.client(
    "s3",
    endpoint_url="https://fsn1.your-objectstorage.com",
    aws_access_key_id="YOUR_HETZNER_KEY",
    aws_secret_access_key="YOUR_HETZNER_SECRET",
)

# Upload dataset with versioning enabled
hetzner.put_bucket_versioning(
    Bucket="eu-ml-data",
    VersioningConfiguration={"Status": "Enabled"},
)
hetzner.upload_file("eu-ml-data", "datasets/train.parquet", "/tmp/train.parquet")

Real-World Use Cases

• •Archiving petabytes of training data at minimal cost with inherent geographic redundancy
• •Storing encrypted embedding backups where security and cost matter more than retrieval speed
• •Distributing large open-source ML datasets with built-in content delivery

Integration Example

import boto3

storj = boto3.client(
    "s3",
    endpoint_url="https://gateway.storjshare.io",
    aws_access_key_id="YOUR_STORJ_ACCESS_KEY",
    aws_secret_access_key="YOUR_STORJ_SECRET_KEY",
)

# Upload training data — encrypted and distributed automatically
storj.upload_file("ml-archive", "datasets/train_v5.tar.gz", "/tmp/train_v5.tar.gz")

# Data is erasure-coded across 80+ nodes for durability
head = storj.head_object(Bucket="ml-archive", Key="datasets/train_v5.tar.gz")
print(f"Stored: {head['ContentLength'] / 1e9:.1f} GB (encrypted, distributed)")

Real-World Use Cases

• •Storing iterating-quickly datasets that get replaced frequently — no early deletion penalties
• •Budget embedding storage for research teams and startups that cannot afford $15-23/TB/mo
• •Backing up ML experiment artifacts where cost predictability matters but data churn is high

Integration Example

import boto3

idrive = boto3.client(
    "s3",
    endpoint_url="https://YOUR_REGION.idrivee2.com",
    aws_access_key_id="YOUR_IDRIVE_KEY",
    aws_secret_access_key="YOUR_IDRIVE_SECRET",
)

# Upload dataset — no minimum retention, delete anytime
idrive.upload_file("experiments", "run-42/embeddings.npy", "/tmp/embeddings.npy")

# Clean up old experiment data without early-deletion penalties
idrive.delete_object(Bucket="experiments", Key="run-41/embeddings.npy")

Real-World Use Cases

• •Co-locating model artifacts with Vultr A100 GPU instances for zero-hop model loading
• •Storing inference results and logs from Vultr-hosted ML services
• •Small-to-medium dataset hosting for GPU-powered training runs on Vultr infrastructure

Integration Example

import boto3

vultr = boto3.client(
    "s3",
    endpoint_url="https://ewr1.vultrobjects.com",
    aws_access_key_id="YOUR_VULTR_KEY",
    aws_secret_access_key="YOUR_VULTR_SECRET",
)

# Upload model to co-located storage near GPU instance
vultr.upload_file("ml-models", "yolo/v8/weights.pt", "/tmp/yolov8.pt")

# Download to GPU instance on same network — minimal latency
vultr.download_file("ml-models", "yolo/v8/weights.pt", "/tmp/yolov8_local.pt")