How do I make files searchable as soon as they land in S3?
Have S3 tell your indexer when an object is created, through an event notification to SQS, Lambda or EventBridge, and keep a slower scheduled listing of the bucket as a backstop. S3 delivers events at least once, usually within seconds but sometimes a minute or more later, so the indexer has to drop duplicates and catch what the events miss. Search freshness is then the event delay plus the time to fetch, embed and index the file, and for video the processing time dominates.
What are the ways to detect new files in an S3 bucket?
There are four, and most production setups combine two of them.
| Approach | How a new upload is found | Typical delay | What you run | Main risk |
|---|---|---|---|---|
| Scheduled listing (polling) | LIST the prefix on a timer and compare LastModified with the last pass | Up to one interval | A scheduler | Large prefixes make each pass slow and add request charges |
| Event notification to SQS | S3 puts a message on a queue, a worker reads it | Seconds | A queue and a worker | Duplicate or late events |
| Event notification to Lambda | S3 invokes a function per event | Seconds | A function | One invocation per upload gets costly at high volume |
| EventBridge rule | S3 sends events to EventBridge, a rule filters by prefix or suffix and calls a target, such as an HTTPS API destination | Seconds | A rule and a target | A target that answers success but did nothing loses the event |
S3 supports SNS, SQS, Lambda and EventBridge as notification destinations (AWS docs). EventBridge adds content filtering and lets one bucket feed several consumers without changing the bucket's notification configuration (Using EventBridge).
Can I rely on S3 events alone?
No. AWS states that S3 event notifications "are designed to be delivered at least once" and are "typically delivered in seconds but can sometimes take a minute or longer" (AWS docs). That gives three rules for the consumer:
1. Make processing idempotent. Key each file on bucket, key and version id (or ETag), so a second delivery of the same event is a no-op. 2. Expect late events. An event can arrive after the file was overwritten again, so read the object's current state instead of trusting the event body. 3. Reconcile on a schedule. A periodic listing finds anything the events missed, including uploads made while the consumer was down. It can run every few minutes or hourly, since it only catches stragglers.
S3 has had strong read-after-write consistency for PUT and LIST on all objects (Amazon S3 consistency), so a listing taken after an upload completes includes it.
Why does an upload get skipped while an indexing job is running?
Because the event arrived during a run that had already listed past the new key, and the trigger was dropped as a duplicate run. This is the most common gap in event-driven indexing. A typical indexer holds a lock so only one run per bucket happens at a time. If a second trigger arrives during a run and is answered "already running", the new file waits for the next scheduled pass.
EventBridge makes this worse in one specific way: it retries a target only on retriable errors, by default for up to 24 hours and 185 attempts (retry policy). A target that answers 200 "skipped" has accepted the event, so nothing retries it.
The fix is to coalesce triggers: when a trigger arrives during a run, set a pending flag, and when the run releases its lock, start one more incremental run if the flag is set. Any number of events during one run then cost exactly one follow-up run, and none is lost. Configure a dead-letter queue on the EventBridge target as well, so events that exhaust their retries are kept.
How long does an incremental listing take?
It grows with the number of keys under the prefix, because S3 returns at most 1,000 keys per LIST page and the indexer has to page through all of them to compare timestamps. In our measurements through Mixpeek's S3 sync, an incremental listing cost about 0.2 seconds per 1,000 keys: 36 to 42 seconds on a prefix of 200,000 keys. So an event that triggers a full-prefix listing stays fast up to a prefix of about 20,000 keys. Past that, split the bucket into prefixes, or process the event's own key directly.
LIST requests are billed at the PUT, COPY and POST rate, which is higher than the GET rate (S3 pricing), so a short polling interval over a large prefix also adds up on the AWS bill.
How do I carry metadata with the file?
Pick the source by how big the metadata is and whether it changes after upload.
| Source | Limit | Can change after upload | Notes |
|---|---|---|---|
| User metadata (x-amz-meta-* headers) | 2 KB per object (AWS docs) | Only by copying the object onto itself | Read with one HeadObject; the copy changes LastModified, so a poll notices it |
| Object tags | 10 per object (AWS docs) | Yes, in place | Read with GetObjectTagging; a tag edit leaves LastModified unchanged, so only a tagging event reveals it |
| S3 annotations | 1 MB of UTF-8 text per annotation (AWS docs) | Yes, without changing the object | A newer S3 feature for rich payloads such as AI-generated labels; check that your indexer reads them before you rely on it |
| The indexing API request body | Set by the API | Yes | Your code sends the fields when it registers the file |
| A sidecar JSON file | None | Yes | The file and its sidecar can arrive in either order, so pairing has to wait for both |
Can I index a private S3 object from a presigned URL?
Yes, if the indexer fetches it with the method the URL was signed for. A presigned URL is signed for one HTTP method, so a presigned GET URL answers a HEAD request with 403. An indexer that validates URLs with HEAD will reject every presigned GET URL unless it falls back to a ranged GET of the first byte, which the GET signature allows. Give the URL an expiry long enough to cover the time until the file is fetched, which for a queued backlog can be longer than you expect.
How Mixpeek does this
Mixpeek is a multimodal data warehouse: it indexes video, images, audio and documents where they already live in object storage and makes them searchable through retrievers. Three of the patterns above are available for S3:
S3 ingest patterns in the docs compares them, and AWS S3 covers the IAM role. Once files are indexed, the vector store runs on the same object storage (pricing).
Frequently Asked Questions
How do I trigger indexing when a file is uploaded to S3?
Send S3 event notifications for object-created events to SQS, Lambda or EventBridge, and have the consumer register or process the file. Because events are delivered at least once and can arrive a minute or more late, make the processing idempotent on bucket, key and version, and keep a scheduled listing of the bucket to catch missed events.
Are S3 event notifications guaranteed to arrive?
They are designed to be delivered at least once, so you can get duplicates, and AWS says they usually arrive in seconds but sometimes take a minute or longer. Pair them with a periodic listing of the bucket as a backstop, and configure a dead-letter queue on EventBridge targets for events that exhaust their retries.
Should I poll S3 or use events?
Use both. Polling alone is simplest and works when a delay of one interval is fine; a 30 to 60 second interval suits many search use cases. Events cut the delay to seconds, and they need idempotent processing and a reconciliation pass behind them.
Why did my file upload not get indexed?
The most common cause is an event that arrived while an indexing run was already in progress and was dropped as a duplicate run. Coalesce triggers into one follow-up run instead. Other causes are a prefix or suffix filter that did not match the key, a missing s3:GetObject or s3:GetObjectTagging permission, and a presigned URL that expired before the indexer fetched it.
How fast can a video in S3 become searchable?
Detection takes seconds with events, or up to one interval with polling. After that the file has to be downloaded, split into scenes or frames, embedded and indexed, which for a long video takes longer than detection, so measure end-to-end time on your own files.