AWS S3

Mixpeek reads your S3 bucket through an IAM role you create and Mixpeek assumes. A sync lists the bucket on a schedule, or when an S3 event triggers it, and brings new and changed objects into a Mixpeek bucket. Your own code can also push objects through the API.

Choose how files arrive

Option	Available	Time until the object exists	You run code
Sync on a schedule (recommended)	Yes	Up to one polling interval (30 s minimum)	No
Sync triggered by S3 events	Yes	Seconds, plus one listing of the prefix	No
Push from your code	Yes	Seconds	Yes
Upload into a Mixpeek bucket	Yes	Seconds	Yes
One-time backfill	Yes	Not for ongoing uploads	No
S3 events through an SQS queue Mixpeek reads	No

Processing comes on top of every row. On production, a short text file was searchable 3 to 6 minutes after it landed in S3, with the object itself created 2 to 7 seconds after upload. Larger files and video take longer. For new uploads, start with a scheduled sync. If a few minutes is too slow, set polling_interval_seconds between 30 and 60, or have S3 events trigger the sync. If you need the object within seconds and already run code on upload, push from your code. Use a one-time backfill for an existing library. Sync on a schedule. You create one read-only IAM role, which can be limited to a prefix, and every access shows up in CloudTrail. Nothing runs on your side. A new upload waits up to one interval. Each pass is a LIST over the prefix, so a very large prefix makes each pass slower and adds request charges, and a short interval multiplies both. Sync triggered by S3 events. The sync starts within seconds of an upload, and a trigger during a run queues one follow-up run however many uploads arrive. On your side you add an EventBridge rule, a connection that holds your API key, and an API destination. Each triggered run still lists the prefix, about 4 seconds for 20,000 objects, so events suit prefixes up to about that size. Each trigger counts against your plan’s data rate limit; a 429 is retried by EventBridge, so a burst is delayed. Keep the schedule, for example hourly, to catch dropped events and to find deletes. Push from your code. Fields go in the request body, so the 2 KB limit on S3 user metadata does not apply, and you decide the order and priority of what gets processed. You write, run and pay for that code, including retries when a call fails, and a Lambda invocation per upload gets expensive at high volume. Mixpeek fetches the file from the URL you send and does not use your storage connection’s role for it. Upload into a Mixpeek bucket. Mixpeek never gets access to your bucket. If you also keep the file in S3, it is stored twice, and you pay AWS to transfer it out. S3 events through an SQS queue and sidecar JSON files (a clip_1234.json beside each clip_1234.mp4) are not available for S3. Use the EventBridge trigger for events. Today only the iconik connector pairs companion files.

Where metadata comes from

Source	Read today	Size limit	Can be changed in place
User metadata (`x-amz-meta-*`)	Yes, by a sync	2 KB per object	No, copy the object onto itself
Request body	Yes, when you push from your code	None	Yes, with a `PATCH` on the object
Object tags	Yes, by a sync	10 tags per object	Yes, but a sync sees the edit only when the object next changes
Sidecar JSON files	No, for S3

Push from your code

Any code that runs when a file is ready can push it: your upload service, a queue worker, or an AWS Lambda function on s3:ObjectCreated. Send a URL Mixpeek can fetch, such as a presigned GetObject URL, with one blob per bucket field:

import boto3, requests

s3 = boto3.client("s3")
url = s3.generate_presigned_url(
    "get_object", Params={"Bucket": "YOUR_BUCKET_NAME", "Key": "videos/clip_1234.mp4"}, ExpiresIn=3600
)
headers = {"Authorization": f"Bearer {MIXPEEK_API_KEY}", "X-Namespace": NAMESPACE_ID}

obj = requests.post(
    f"https://api.mixpeek.com/v1/buckets/{BUCKET_ID}/objects",
    headers=headers,
    json={"blobs": [
        {"property": "video", "type": "video", "data": url},
        {"property": "video_id", "type": "text", "data": "1234"},
        {"property": "publisher", "type": "text", "data": "acme-news"},
    ]},
).json()

requests.post(
    f"https://api.mixpeek.com/v1/buckets/{BUCKET_ID}/batches",
    headers=headers,
    json={"object_ids": [obj["object_id"]], "auto_submit": True},
)

A presigned URL must stay valid until Mixpeek has fetched the file, so give it at least an hour. To send the bytes instead of a URL, create a presigned upload, PUT the file to its presigned_url, confirm it, and reference the upload_id in the blob. See Register objects for every blob form.

Prerequisites

An S3 bucket, or a prefix in one, holding the files to process.
Permission to create IAM policies and roles in that AWS account.
A Mixpeek API key with the admin permission, which creating a storage connection needs.

Configuration Steps

Create the read policy

Mixpeek needs s3:ListBucket on the bucket and s3:GetObject on the objects. Replace YOUR_BUCKET_NAME.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": ["s3:ListBucket"],
            "Resource": ["arn:aws:s3:::YOUR_BUCKET_NAME"]
        },
        {
            "Effect": "Allow",
            "Action": ["s3:GetObject"],
            "Resource": ["arn:aws:s3:::YOUR_BUCKET_NAME/*"]
        }
    ]
}

To limit Mixpeek to one prefix, scope both statements to it. The s3:prefix condition restricts what Mixpeek can list, and the object ARN restricts what it can read. Replace videos/ with your prefix and keep the trailing slash.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": ["s3:ListBucket"],
            "Resource": ["arn:aws:s3:::YOUR_BUCKET_NAME"],
            "Condition": {"StringLike": {"s3:prefix": ["videos/*"]}}
        },
        {
            "Effect": "Allow",
            "Action": ["s3:GetObject"],
            "Resource": ["arn:aws:s3:::YOUR_BUCKET_NAME/videos/*"]
        }
    ]
}

A separate test bucket works the same way with the first policy. Save the policy as, for example, MixpeekS3ReadAccessPolicy.

Optional: allow writes for export destinations

Read access is all Mixpeek needs to ingest from your bucket. Add write access only if you want to use this connection as an export destination, where Mixpeek writes a dataset export and its manifest.json into your bucket.Add one statement to the policy above:

{
    "Effect": "Allow",
    "Action": [
        "s3:PutObject"
    ],
    "Resource": [
        "arn:aws:s3:::YOUR_BUCKET_NAME/*"
    ]
}

Then turn the capability on for that connection, which is off by default:

curl -X PATCH https://api.mixpeek.com/v1/organizations/connections/conn_xxx \
  -H "Authorization: Bearer $MIXPEEK_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"write_enabled": true}'

Both are required, and they are independent: write_enabled is your permission for Mixpeek to try, and s3:PutObject is AWS letting it succeed. An export naming a connection without the flag is refused before anything is written, and the error names the permission to add. Connections you never enable stay read-only.

Create a role Mixpeek can assume

Create an IAM role with this trust policy and attach the read policy to it. Mixpeek’s production API runs as arn:aws:iam::931340738373:user/mixpeek-platform-prod, in AWS account 931340738373.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {"AWS": "arn:aws:iam::931340738373:root"},
            "Action": "sts:AssumeRole",
            "Condition": {"StringEquals": {"sts:ExternalId": "YOUR_EXTERNAL_ID"}}
        }
    ]
}

The external ID is a value you choose, 8 to 128 characters, such as mixpeek- followed by your organization ID. Keep it private. The role accepts Mixpeek only when Mixpeek presents the same external ID, which stops another Mixpeek customer from naming your role ARN. To trust only the Mixpeek API user, set the principal to arn:aws:iam::931340738373:user/mixpeek-platform-prod.Mixpeek requests a 1-hour session on the role each time a sync connects. Every access appears in your CloudTrail under a session named mixpeek-storage-sync-<timestamp>.Long-lived access keys also work: pass {"type": "access_key", "access_key_id": "...", "secret_access_key": "..."} as credentials in the next step, from an IAM user that has the read policy. The role is the safer choice, because no secret leaves your account.

Create the storage connection

curl -X POST "https://api.mixpeek.com/v1/organizations/connections" \
  -H "Authorization: Bearer $MIXPEEK_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "video-library",
    "provider_type": "s3",
    "provider_config": {
      "credentials": {
        "type": "iam_role",
        "role_arn": "arn:aws:iam::YOUR_ACCOUNT_ID:role/mixpeek-storage-sync-role",
        "external_id": "YOUR_EXTERNAL_ID"
      },
      "region": "us-east-1"
    },
    "test_before_save": true
  }'

The response carries the connection_id for the sync.With test_before_save: true, Mixpeek assumes the role and calls ListBuckets before it saves, so a wrong trust policy fails here with the AWS error. ListBuckets needs s3:ListAllMyBuckets on *, which the read policy above does not grant. Either add that action to the policy or send "test_before_save": false, in which case the first sync run is the test.

Sync the bucket

Create a sync on a Mixpeek bucket. source_path names the S3 bucket and, optionally, the prefix.

curl -X POST "https://api.mixpeek.com/v1/buckets/$BUCKET_ID/syncs" \
  -H "Authorization: Bearer $MIXPEEK_API_KEY" \
  -H "X-Namespace: $NAMESPACE_ID" \
  -H "Content-Type: application/json" \
  -d '{
    "connection_id": "conn_abc123",
    "source_path": "s3://YOUR_BUCKET_NAME/videos/",
    "sync_mode": "continuous",
    "polling_interval_seconds": 60,
    "file_filters": {"include_patterns": ["*.mp4"]}
  }'

How new and changed objects arrive

A continuous sync polls: it lists the prefix every polling_interval_seconds (default 300, minimum 30, maximum 86400) and takes every object whose LastModified is newer than the last pass. A new upload is picked up within one interval, then submitted for processing. To start the sync within seconds of an upload, trigger it from EventBridge.

Change in S3	What the next poll does
New object	Creates a Mixpeek object and processes it
Object overwritten with new content	Deletes the Mixpeek object and its documents, then creates a new object from the new content and processes it. The new object has a new `object_id`, so a stored `object_id` no longer resolves. A copy that changes only metadata keeps the object and its `object_id` (next row)
User metadata changed by a copy onto itself	Updates the mapped fields and re-processes, while `reconcile.on_update` is `true` (the default)
Object deleted	Deletes the Mixpeek object and its documents, while `reconcile.on_delete` is `true` (the default)
Object tags edited	Nothing, because a tag edit leaves `LastModified` unchanged. A `tag` mapping picks up the new tags the next time the object itself changes

Object metadata

Send each object’s metadata as S3 user metadata, the x-amz-meta-* headers set at upload. Map each key to a bucket field with a metadata source in the sync’s schema_mapping. The key is the header name without x-amz-meta-, and S3 stores it in lowercase.

aws s3 cp clip_1234.mp4 s3://YOUR_BUCKET_NAME/videos/clip_1234.mp4 \
  --metadata video_id=1234,publisher=acme-news,breaking=true

{
  "schema_mapping": {
    "mappings": {
      "content":   { "target_type": "blob",  "source": { "type": "file" }, "blob_type": "video" },
      "video_id":  { "target_type": "field", "source": { "type": "metadata", "key": "video_id" } },
      "publisher": { "target_type": "field", "source": { "type": "metadata", "key": "publisher" } },
      "breaking":  { "target_type": "field", "source": { "type": "metadata", "key": "breaking" } }
    }
  }
}

S3 user metadata is fixed once an object is written. To change it, copy the object onto itself with the new values. The copy gets a new LastModified, so the next poll applies it:

aws s3 cp s3://YOUR_BUCKET_NAME/videos/clip_1234.mp4 s3://YOUR_BUCKET_NAME/videos/clip_1234.mp4 \
  --metadata video_id=1234,publisher=acme-news,breaking=false --metadata-directive REPLACE

REPLACE writes only the metadata you pass, so include every key, not only the changed one. Each changed object is re-processed by default. To update fields without re-running extraction, set reconcile.re_extract_on_update to false, or list the fields that should trigger it in reconcile.re_extract_fields. Mixpeek reads user metadata with one HeadObject per object, and only when the mapping has a metadata source.

Object tags

A tag source maps an S3 object tag to a bucket field. Grant the role s3:GetObjectTagging on the objects as well as s3:GetObject; without it, each object fails with an error naming that permission.

"region": { "target_type": "field", "source": { "type": "tag", "key": "region" } }

Mixpeek reads tags with one GetObjectTagging per object, and only when the mapping has a tag source. S3 allows 10 tags per object. A tag edit does not change LastModified, so the sync applies new tags when the object is next overwritten or copied onto itself. For values that change often, use user metadata with a copy in place, or the API.

One-time backfill

For a library already in S3, run a sync with sync_mode: "initial_only", which lists the prefix once and stops, or create objects in batches. Then submit them with bulk-submit, which splits a large bucket into batches. Batch ingestion at scale covers chunk sizes, concurrency and priority. A backfill suits tens of thousands to millions of existing objects and does not pick up later uploads, so pair it with one of the options above.

Run the sync on every upload

S3 reports each upload to Amazon EventBridge, and an EventBridge API destination calls the sync’s trigger:

POST https://api.mixpeek.com/v1/buckets/{bucket_id}/syncs/{sync_config_id}/trigger?queue_follow_up=true

With queue_follow_up=true, a trigger that arrives while the sync is running queues one follow-up run behind it and answers queued_follow_up. Any number of triggers during one run queue a single follow-up, so a burst of uploads costs one extra run. The trigger ignores the event body. A sync sharded across several bucket_prefixes cannot queue a follow-up and answers 400. Events suit a prefix of up to about 20,000 objects: each triggered run still lists the prefix, which takes about 4 seconds at that size.

Send the bucket's events to EventBridge

aws s3api put-bucket-notification-configuration --bucket YOUR_BUCKET_NAME \
  --notification-configuration '{"EventBridgeConfiguration": {}}'

Create a connection that carries your API key

Create a Mixpeek API key for this purpose alone, with write on the namespace, so you can rotate it without touching other callers.

aws events create-connection --name mixpeek-sync-trigger --authorization-type API_KEY \
  --auth-parameters '{
    "ApiKeyAuthParameters": {"ApiKeyName": "Authorization", "ApiKeyValue": "Bearer YOUR_API_KEY"},
    "InvocationHttpParameters": {"HeaderParameters": [{"Key": "X-Namespace", "Value": "YOUR_NAMESPACE_ID", "IsValueSecret": false}]}
  }'

Create the API destination

Keep --invocation-rate-limit-per-second at or under your plan’s data rate limit divided by 60, for example 2 on Build (120 a minute). That budget is shared with your other object and document calls. See Rate limits and quotas.

aws events create-api-destination --name mixpeek-sync-trigger \
  --connection-arn CONNECTION_ARN \
  --invocation-endpoint "https://api.mixpeek.com/v1/buckets/BUCKET_ID/syncs/SYNC_CONFIG_ID/trigger?queue_follow_up=true" \
  --http-method POST --invocation-rate-limit-per-second 2

Add a rule for new objects

Filter by bucket and key prefix, then target the API destination with a retry policy and a dead-letter queue. The target’s role needs events:InvokeApiDestination.

aws events put-rule --name mixpeek-new-videos --event-pattern '{
  "source": ["aws.s3"],
  "detail-type": ["Object Created"],
  "detail": {"bucket": {"name": ["YOUR_BUCKET_NAME"]}, "object": {"key": [{"prefix": "videos/"}]}}
}'

aws events put-targets --rule mixpeek-new-videos --targets '[{
  "Id": "mixpeek",
  "Arn": "API_DESTINATION_ARN",
  "RoleArn": "arn:aws:iam::YOUR_ACCOUNT_ID:role/YOUR_EVENTBRIDGE_ROLE",
  "RetryPolicy": {"MaximumRetryAttempts": 185, "MaximumEventAgeInSeconds": 86400},
  "DeadLetterConfig": {"Arn": "arn:aws:sqs:REGION:YOUR_ACCOUNT_ID:YOUR_DLQ"}
}]'

Keep the schedule as a safety net

EventBridge can drop or repeat an event. Leave the sync continuous and raise polling_interval_seconds to 3600, so an hourly pass catches anything missed. Deletes are found by that pass too, since the rule sends only new objects.

A 429 from the trigger is retried by EventBridge, so a burst over the rate limit is delayed. Each run’s job records metadata.triggered_by: manual_api_trigger for a trigger, queued_follow_up for a follow-up.

Troubleshooting

AccessDenied on the connection test: check that the trust policy names the principal above and the same external ID you sent, and that the policy grants s3:ListAllMyBuckets (or send "test_before_save": false).
Objects outside the prefix fail to list: the prefix in source_path must sit inside the s3:prefix condition of the policy.
A mapped field is empty: the key must match the header name without x-amz-meta-, in lowercase.

Get started

Connect your data

Extract features

Build retrievers

Enrich & organize

Integrate & operate

Resources

Choose how files arrive

Where metadata comes from

Push from your code

Prerequisites

Configuration Steps

How new and changed objects arrive

Object metadata

Object tags

One-time backfill

Run the sync on every upload

Troubleshooting

​Choose how files arrive

​Where metadata comes from

​Push from your code

​Prerequisites

​Configuration Steps

​How new and changed objects arrive

​Object metadata

​Object tags

​One-time backfill

​Run the sync on every upload

​Troubleshooting

Choose how files arrive

Where metadata comes from

Push from your code

Prerequisites

Configuration Steps

How new and changed objects arrive

Object metadata

Object tags

One-time backfill

Run the sync on every upload

Troubleshooting