What is Mixpeek?

[Back to Blog](/blog)

[](https://twitter.com/intent/tweet?text=How%20to%20Process%20Thousands%20of%20Videos%20in%20Parallel%20with%20Celery%2C%20FastAPI%20and%20S3&url=https%3A%2F%2Fmixpeek.com%2Fblog%2Fusing-celery-to-process-thousands-of-videos)[](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Fmixpeek.com%2Fblog%2Fusing-celery-to-process-thousands-of-videos&title=How%20to%20Process%20Thousands%20of%20Videos%20in%20Parallel%20with%20Celery%2C%20FastAPI%20and%20S3)[](https://www.reddit.com/submit?url=https%3A%2F%2Fmixpeek.com%2Fblog%2Fusing-celery-to-process-thousands-of-videos&title=How%20to%20Process%20Thousands%20of%20Videos%20in%20Parallel%20with%20Celery%2C%20FastAPI%20and%20S3)

This was originally in our newsletter.

Get Updates

Aug 2, 2024•7 min read

# How to Process Thousands of Videos in Parallel with Celery, FastAPI and S3

Split each video into chunks, queue one Celery task per chunk on Redis, and add workers to scale. The architecture, code and failure handling we used to process thousands of videos at once.

![How to Process Thousands of Videos in Parallel with Celery, FastAPI and S3](https://blog.mixpeek.com/content/images/2025/06/ChatGPT-Image-Jun-5--2025--04_20_50-PM.png)

Data Processing

TABLE OF CONTENTS

**How do you process thousands of videos in parallel?** Put each video in object storage, split it into fixed-length chunks, and queue one task per chunk on a broker such as Redis. A pool of Celery workers pulls the tasks, and you add workers to go faster. Make every task idempotent and acknowledge it late, so a worker that dies mid-video hands its chunk to another worker.

Below is the pipeline we ran on Celery, FastAPI, S3 and Render, with its code. Updated October 2026: Mixpeek's production pipeline now runs on [Ray](https://docs.ray.io/en/latest/data/data.html?ref=blog.mixpeek.com), and the chunk-per-task design and failure handling described here carried over.

At Mixpeek, we've developed a highly scalable and fault-tolerant video understanding pipeline capable of processing thousands of videos concurrently. Our system leverages Celery workers, FastAPI, AWS S3, AWS Sagemaker, and Render for deployment to create a distributed processing architecture that ensures reliability and efficiency. This article will dive into the technical details of our implementation, providing insights and code snippets to help you replicate our setup.

## System Architecture Overview

Our video processing pipeline consists of the following key components:

1.  FastAPI endpoint for video upload
2.  S3 storage for uploaded videos
3.  Celery tasks for distributed processing
4.  Custom video chunking and embedding services
5.  Database storage for processed video segments
6.  Render for deployment and scaling
7.  Redis for caching and as a message broker

Here's a visual representation of our system architecture:

.diagram-container { display: flex; justify-content: center; width: 100%; margin: 0 auto; } .mermaid { max-width: 100%; }

graph TD A\[Client\] --> B\[FastAPI Endpoint\] B --> C\[S3 Storage\] B --> H\[Redis\] H -->|Message Broker| D\[Celery Tasks\] subgraph Celery Workers D --> E\[Video Chunking Service\] D --> F\[Video Embedding Service\] end E --> G\[Database Storage\] F --> G H -->|Caching| D subgraph Render Deployment B D H end classDef renderService fill:#f9f,stroke:#333,stroke-width:2px; class B,D,H renderService; classDef externalService fill:#fff,stroke:#333,stroke-width:2px; class C,G externalService;

This architecture allows us to efficiently handle video uploads, process them in a distributed manner, and store the results for further use.

## Deployment Stack on Render

Our production deployment on Render consists of the following components:

**Redis Cache**: Serves as a message broker for Celery and provides caching capabilities.

![](https://blog.mixpeek.com/content/images/2024/08/image-3.png)

Free tier tends to work just fine

**API Service**: Handles API requests, including the video upload endpoint.

![](https://blog.mixpeek.com/content/images/2024/08/image-6.png)

**Celery Workers**: Two separate background worker services for processing tasks.

![](https://blog.mixpeek.com/content/images/2024/08/image-2.png)

This setup allows us to scale our video processing pipeline horizontally by adding more Celery workers as needed.

## FastAPI Upload Endpoint

The entry point of our pipeline is a FastAPI route that handles video uploads:

```python
@router.post("/upload")
async def index_upload(
    request: Request,
    file: UploadFile = File(...),
    collection: str = Form(...),
    metadata: str = Form(default=None)
):
    # ... (file handling and metadata extraction)

    s3_response = upload_file_to_s3(content_io, file_id, request.index_id, collection)

    input_data = ProcessInputData(**{
        "index_id": request.index_id,
        "source": "upload",
        "file_id": file_id,
        "file_ext": file_extension,
        "s3_response": {
            **s3_response.dict()
        },
        "collection": collection,
        "metadata": metadata_dict
    })

    task = process_file.delay(input_data.dict())
    
    return {"message": "File upload processing started", "task_id": task.id}
```

This endpoint:

1.  Accepts video uploads and metadata
2.  Uploads the video to S3
3.  Creates a ProcessInputData object with relevant information
4.  Queues a Celery task for processing

## Celery Task Definition

We use a shared Celery task to process the uploaded videos:

```python
@shared_task(
    bind=True,
    name="managed.tasks.process_file",
    max_retries=3,
    default_retry_delay=5,
)
def process_file(self, input_data: dict):
    processor = FileProcessor(input_data)
    processor.task_id = self.request.id
    processor.process()
```

The `@shared_task` decorator offers several advantages over `@task`:

1.  It allows the task to be used by multiple Celery apps
2.  It provides better integration with Django (if used)
3.  It offers more flexibility in task registration and discovery

The task is configured with:

*   `max_retries=3`: Automatically retries the task up to 3 times in case of failure
*   `default_retry_delay=5`: Waits 5 seconds between retry attempts

## Video Processing Pipeline

The core of our video processing is handled by the `FileProcessor` class:

```python
class FileProcessor:
    def process(self):
        try:
            self.initialize_file()
            self.handle_file_source()
            self.process_file()
            self.finalize_file()
        except Exception as e:
            self.handle_error(e)
            raise
```

This class orchestrates the entire processing pipeline, including error handling and cleanup.

## Video Chunking and Embedding

The heart of our video understanding pipeline lies in the `process_video` function:

```python
def process_video(tmp_file_path, s3_url: AnyUrl, input_data: ProcessInputData):
    video_chunker = ChunkService(
        file_path=tmp_file_path,
        target_size_mb=5,
        auto_adjust=False
    )
    embed_service = VideoEmbedService(model="vuse-generic-v1")
    video_segments_db = VideoSegmentsDBService(input_data.index_id)

    batch_size = 10  # Process 10 chunks at a time
    temp_dir = tempfile.mkdtemp()

    try:
        batch = []
        for i, video_chunk in enumerate(video_chunker.process_video()):
            # Process each chunk (embedding, metadata extraction, etc.)
            # ...

            if len(batch) >= batch_size:
                process_batch(batch, video_segments_db, temp_dir, i // batch_size)
                batch = []

        # Process any remaining chunks
        if batch:
            process_batch(batch, video_segments_db, temp_dir, (i + 1) // batch_size)

    finally:
        # Clean up temporary files
        # ...
```

Key components:

1.  `ChunkService`: Splits videos into manageable chunks
2.  `VideoEmbedService`: Generates embeddings for each chunk, using our [vuse-generic-v1 embedding model](https://blog.mixpeek.com/vuse-v1-release/) hosted on AWS Sagemaker
3.  `VideoSegmentsDBService`: Stores processed video segments
4.  Batch processing: Processes chunks in batches for efficiency

The `ChunkService` is responsible for splitting the video into smaller, processable chunks:

## Run this on your own content

Point Mixpeek at the storage you already have and search your video, images, audio, and documents the way this post describes. Build starts at $25/mo for up to 1M vectors.

[Search your own files](https://studio.mixpeek.com/namespaces/create?mode=managed&utm_source=homepage&utm_medium=blog_onramp_mid&utm_campaign=mvs_onramp)[Talk to us](/demo)

```python
def process_video(self):
    self.initialize_video()
    chunk_count = 0
    while True:
        temp_video_path = os.path.join(self.temp_dir, f"chunk_{chunk_count}.mp4")
        frames_written = self._write_chunk_to_disk(temp_video_path)
        
        if frames_written == 0:
            break

        base64_string = self._create_base64_from_file(temp_video_path)
        
        start_time, end_time = self._calculate_time_intervals(frames_written)
        chunk_object = {
            "base64": base64_string,
            "start_time": start_time,
            "end_time": end_time
        }

        yield chunk_object

        os.remove(temp_video_path)
        chunk_count += 1

    self.cap.release()
```

This generator-based approach allows for efficient memory usage when processing large videos.

## Scaling with Celery on Render

To scale our video processing pipeline, we use Celery workers deployed on Render. Here's how we've set up and run our Celery workers:

*   `-A db.service.celery_app`: Specifies the Celery app instance
*   `--loglevel=info`: Sets the logging level
*   `-n worker1@%h`: Names the worker (replace `worker1` with a unique name for each worker)
*   `-c 4`: Sets the number of concurrent worker processes to 4

2.  **Redis as Message Broker and Result Backend**:  
    We use a Redis instance on Render as both the message broker and result backend for Celery. This setup ensures efficient task distribution and result storage.
3.  **Additional Disk Space**:  
    Our Celery workers are configured with additional disk space on Render to handle temporary storage of video chunks and processing artifacts.
4.  **Scaling Workers**:  
    By using multiple Celery worker instances (celery\_1 and celery\_2), we can easily scale our processing capacity. We can add more worker instances on Render as needed to handle increased load.
5.  **Monitoring**:  
    Our dashboard service, also deployed on Render, allows us to monitor the performance and status of our Celery workers and overall pipeline.

**Celery Worker Configuration**:  
We've deployed two Celery worker instances (celery\_1 and celery\_2) as Background Worker services on Render. Each worker is configured with the following command:

```
celery -A db.service.celery_app worker --loglevel=info -n worker1@%h -c 4
```

Key components of this command:

## Celery Configuration

To use Redis as the message broker and result backend, our Celery configuration (celeryconfig.py) looks like this:

```python
broker_url = 'redis://your-redis-url:6379/0'
result_backend = 'redis://your-redis-url:6379/0'
task_serializer = 'json'
result_serializer = 'json'
accept_content = ['json']
timezone = 'UTC'
enable_utc = True
```

## End Result: Video Understanding Demo for Advertisers

The true power of Mixpeek's video processing pipeline becomes evident when we look at its practical applications. Let's explore how our system processes thousands of social media videos for scene understanding, particularly benefiting the advertising sector.

![](https://blog.mixpeek.com/content/images/2024/08/image-8.png)

### Video Intelligence for Advertising

Our demo showcases the processing of numerous social media videos, extracting key concepts and scenes that are valuable for advertisers.

Here's what the end result looks like:

1.  **Concept Extraction**: The system identifies popular concepts within videos, such as:
    *   Crowd celebrating
    *   Person dancing
    *   Breaking the ice
    *   Human connection
2.  **Scene Matching**: For each concept, the system provides:
    *   The number of matching scenes
    *   A relevance score (e.g., Score: 0.63)
    *   The collection the video belongs to (e.g., Media and Advertising)

### Integration and Accessibility

Advertisers can easily integrate this video intelligence into their existing workflows:

1.  **One-Line Integration**: Integrate your S3 bucket directly or bake the Mixpeek SDK into your code
2.  **Real-Time Updates**: The system keeps embeddings fresh by syncing with the client's database, ensuring up-to-date video understanding.
3.  **Exploration Interface**: A user-friendly interface allows advertisers to explore processed videos, search for concepts, and visualize matches.

## Conclusion

Mixpeek's video processing pipeline demonstrates how to leverage Celery, FastAPI, and custom services to create a scalable and robust system for handling thousands of videos. Our deployment on Render, utilizing multiple Celery workers and a Redis cache, allows for efficient scaling and management of the processing load.

Key takeaways:

1.  Use FastAPI for efficient upload handling
2.  Leverage Celery for distributed task processing
3.  Implement custom services for video chunking and embedding
4.  Process videos in batches for improved efficiency
5.  Use S3 for reliable storage of uploaded videos
6.  Implement proper error handling and retries
7.  Deploy on a scalable platform like Render
8.  Utilize Redis for efficient task queuing and result storage
9.  Scale horizontally by adding more Celery workers as needed

By following these principles and adapting the provided architecture and code snippets, you can build a similar system capable of processing large volumes of video content without failure or droppage, while maintaining the flexibility to scale as your processing needs grow.

## Frequently asked questions

### How do I process thousands of videos in parallel with Celery?

Queue one task per video chunk, so a long file spreads across workers. Run several worker services against the same Redis broker and scale them horizontally. A FastAPI endpoint uploads each video to S3 and calls `process_file.delay(...)`, which returns a task id at once.

### What Celery settings should I use for long video tasks?

Celery's own optimizing guide recommends `task_acks_late = True` and `worker_prefetch_multiplier = 1` for long-running tasks. A worker then reserves one task at a time, and a task whose worker is killed is redelivered. Because of that redelivery, the task must be idempotent: writing the same chunk twice must leave one result.

### Should I use task.retry() or acks\_late for failed video tasks?

Both, for different failures. Use `task.retry()` with `max_retries` for errors your code catches, such as a timeout calling an embedding model. `acks_late` covers what no exception reports: a worker that runs out of memory or loses its machine. See [Celery's task guide](https://docs.celeryq.dev/en/stable/userguide/tasks.html?ref=blog.mixpeek.com).

### Why split videos into chunks before embedding them?

A chunk is the unit you search and the unit you retry. A 10-second segment can be embedded, stored and returned as a search result on its own, and a failure costs only that chunk.

### Can I skip running this pipeline myself?

If the goal is search over your videos, yes. Mixpeek reads your files where they already live, in S3, GCS or any bucket, and turns what's in them into data your software can search, classify and moderate, with chunking, embedding, retries and scaling handled for you. See [how Mixpeek works](https://mixpeek.com/docs/overview/concepts?ref=blog.mixpeek.com) and [pricing](https://mixpeek.com/pricing?ref=blog.mixpeek.com).

[ES

Ethan Steininger

Former lead of MongoDB's Search Team, Ethan noticed the most common problem customers faced was building indexing and search infrastructure on their S3 buckets. Mixpeek was born.



](/blog/authors/ethan)

## Run this on your own content

Point Mixpeek at the storage you already have and search your video, images, audio, and documents the way this post describes. Build starts at $25/mo for up to 1M vectors.

[Search your own files](https://studio.mixpeek.com/namespaces/create?mode=managed&utm_source=homepage&utm_medium=blog_onramp_end&utm_campaign=mvs_onramp)[Talk to us](/demo)

## Continue Learning

[

![Video Segmentation: Unlocking Structure for Search and Analytics](https://blog.mixpeek.com/content/images/2025/06/ChatGPT-Image-Jun-5--2025--07_46_19-PM.png)

### Video Segmentation: Unlocking Structure for Search and Analytics

Segmentation turns raw video into searchable chunks—objects, actions, scenes—boosting precision in multimodal search. It bridges unstructured content with structured queries like “man falling in warehouse,” enabling faster, more accurate retrieval across large datasets.

Read More





](/blog/video-segmentation-unlocking-structure-for-search-and-analytics)[

![Automatic Speech Recognition: Build vs. Buy](https://blog.mixpeek.com/content/images/2025/06/ChatGPT-Image-Jun-6--2025--03_48_43-PM.png)

### Automatic Speech Recognition: Build vs. Buy

Learn how to build a scalable ASR pipeline using Ray and Whisper, with batching, GPU optimization, and real-world tips from production deployments

Read More





](/blog/whisper-ray-huggingface)

### Author

ES

Ethan Steininger

![Mixpeek Icon](/brand/logo-dark-icon.webp)

#### Multimodal Search, On Demand

Index, retrieve, and reason over videos, images, audio, and docs, all from a single API.

[Start Building](/start)

### Share

[](https://twitter.com/intent/tweet?text=How%20to%20Process%20Thousands%20of%20Videos%20in%20Parallel%20with%20Celery%2C%20FastAPI%20and%20S3&url=https%3A%2F%2Fmixpeek.com%2Fblog%2Fusing-celery-to-process-thousands-of-videos)[](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Fmixpeek.com%2Fblog%2Fusing-celery-to-process-thousands-of-videos&title=How%20to%20Process%20Thousands%20of%20Videos%20in%20Parallel%20with%20Celery%2C%20FastAPI%20and%20S3)[](https://www.reddit.com/submit?url=https%3A%2F%2Fmixpeek.com%2Fblog%2Fusing-celery-to-process-thousands-of-videos&title=How%20to%20Process%20Thousands%20of%20Videos%20in%20Parallel%20with%20Celery%2C%20FastAPI%20and%20S3)