NEWVectors or files. Pick a path.Start →
    data pipeline

    Apache Kafka Connector

    Real-time event streaming for multimodal pipelines

    Stream unstructured data through Mixpeek's multimodal processing engine using Apache Kafka. This connector enables real-time ingestion and enrichment of text, images, video, and audio events at scale, turning high-throughput Kafka topics into structured, searchable intelligence.

    kafka
    event streaming
    real-time
    data pipeline
    message queue
    stream processing
    Quick Install:
    npm install @mixpeek/kafka

    Use Cases

    1

    Real-time content enrichment from event streams

    2

    Streaming media ingestion for search indexing

    3

    Event-driven taxonomy classification at scale

    4

    Continuous data pipeline for multimodal analytics

    Features

    Consumer group integration for parallel processing
    Automatic schema detection for multimodal payloads
    Dead-letter queue support for failed enrichments
    Configurable batch size and flush intervals
    Exactly-once semantics with offset management
    Built-in back-pressure handling

    Get Started

    Integrations

    Apache Kafka
    Confluent Cloud
    Amazon MSK
    Redpanda

    Details

    LicenseApache 2.0
    Categorydata pipeline
    Registrynpm

    Resources

    Documentation

    Learn how to use this connector

    Quick Info

    LicenseApache 2.0
    Categorydata pipeline
    Registrynpm

    Frequently Asked Questions

    How does the Apache Kafka Connector work with Mixpeek?

    The connector subscribes to one or more Kafka topics and forwards incoming messages to Mixpeek for multimodal processing. It automatically detects content types (text, image URLs, video references) and routes them through the appropriate enrichment pipeline. Processed results can be published back to an output topic or stored directly in Mixpeek namespaces.

    Does it support Confluent Cloud and Amazon MSK?

    Yes. The connector works with any Kafka-compatible broker including Apache Kafka, Confluent Cloud, Amazon MSK, and Redpanda. Authentication supports SASL/PLAIN, SASL/SCRAM, and mTLS depending on your deployment.

    How does it handle processing failures?

    Failed messages are routed to a configurable dead-letter topic with full error context. The connector supports exactly-once semantics by committing offsets only after successful processing, so no data is lost during transient failures.

    What throughput can I expect?

    Throughput depends on message size and enrichment complexity. For text-only payloads, the connector comfortably handles tens of thousands of messages per second per partition. Image and video payloads are bounded by Mixpeek API processing time, typically 200-500ms per item.

    Can I filter or transform messages before processing?

    Yes. The connector supports configurable message filters (by header, key pattern, or payload fields) and lightweight transforms (field extraction, renaming) before sending data to Mixpeek. This lets you reduce API calls and focus enrichment on relevant events.

    Related connectors

    Apache Airflow OperatorWorkflow orchestration for Mixpeek pipelinesApache Spark ConnectorDistributed multimodal processing at warehouse scaleSnowflake ConnectorCloud data warehouse integration for multimodal enrichmentDatabricks ConnectorLakehouse AI integration for multimodal data

    Ready to integrate?

    Get started with Apache Kafka Connector in minutes. Check out the documentation or explore the source code on GitHub.