--- title: "How we built a production Kafka connector for ClickHouse®" excerpt: "How we built a scalable, reliable Kafka connector that processes billions of events with minimal operational overhead" authors: "Alberto Romeu, Jordi Vilaseca" categories: "Engineering Excellence" createdOn: "2026-01-22 00:00:00" publishedOn: "2026-01-22 00:00:00" updatedOn: "2026-01-22 00:00:00" status: "published" --- At Tinybird, we've spent years evolving our Kafka to ClickHouse® connector from a simple consumer script into a battle tested service that processes billions of events daily. This post covers why we built our own connector instead of using ClickHouse's built-in Kafka engine, the architecture decisions that make it scale and the production challenges we've solved along the way. ## Why build our own Kafka connector? Tinybird is a managed ClickHouse with a focus on the best developer experience. Our users expect analytics over Kafka to just work: connect, point to a topic and start querying. Building our own connector lets us deliver that experience. ClickHouse's built-in Kafka engine has limitations: partial Schema Registry support (Avro only), manual offset management and scaling challenges. The Kafka ecosystem adds complexity too, with Confluent, Redpanda, MSK, Event Hubs and others each having protocol quirks that break assumptions from vanilla Apache Kafka. Even with our own ClickHouse fork, coupling the connector to the database creates operational headaches. Cluster upgrades are intricate and tying connector fixes to database releases slows iteration. A separate service lets us move fast on Kafka support while keeping ClickHouse upgrades independent. Using an independent service to load data also brings additional benefits: - **Offloads work from ClickHouse**: Parsing Avro, converting to RowBinary and handling inserts happen in the connector service, reducing load on the database cluster - **Simpler private link configuration**: Only the connector service needs to be added to a private link, rather than the entire ClickHouse cluster {% image src="tinybird-kafka-connector.png" alt="Tinybird Kafka Connector for ClickHouse" caption="Tinybird Kafka Connector for ClickHouse" /%} ## Architecture of a scalable Kafka connector The Tinybird Kafka connector is a Python service deployed as Kubernetes StatefulSets across multiple regions. It consumes from Kafka topics, transforms messages and inserts them into ClickHouse. **Wait, Python for a high throughput Kafka connector?** Yes, and for good reasons: 1. **I/O bound vs CPU bound**: The bottleneck isn't Python's interpreter, it's network I/O and ClickHouse's ingestion capacity 2. **Native performance where it matters**: Performance-critical components use native libraries that release the GIL during I/O operations: - `confluent_kafka` (C++ bindings via librdkafka) for Kafka interactions - `orjson` (Rust) for JSON parsing, 2-3x faster than Python's standard library - `fastavro` for Avro deserialization - Tinybird's in-house C++ library for optimized RowBinary encoding (30% less overhead when inserting to ClickHouse) - `pycurl` (C) for HTTP requests 3. **Developer expertise**: Those are good technical reasons but our engineering team's experience matters too. Other services in our backend are already written in Python, which allows for rapid iteration on customer needs and reusing existing ClickHouse tooling. The data flows through the system in a clear pipeline: ```sh Kafka Topics → Kafka Consumer → Schema Deserialization → RowBinary Encoding → Ingestion proxy → ClickHouse tables ``` ### Controller/Worker Pattern One of our key architectural decisions was implementing a controller/worker pattern. Instead of a monolithic consumer, we use a controller that manages multiple independent workers. One Kafka topic or datasource can be handled by one or more workers, which is how we achieve horizontal scaling. This design brings several benefits: - **Fault isolation**: One customer's Kafka issues don't impact others - **Resource efficiency**: New datasources start consuming without requiring extra resources. We only scale when workers are using all their CPU, reducing the cost per datasource - **Independent scalability**: Allocate resources per customer workload at topic level The controller listens to Redis pub/sub channels for configuration updates, enabling zero downtime changes. When you create a new Kafka datasource in Tinybird, the backend publishes a configuration update and the controller automatically spins up a new worker to consume that topic. ## Five design decisions that make it production ready Several design decisions differentiate a production ready connector from a simple Kafka consumer script: ### 1. Rendezvous Hashing for Topic Assignment We use [rendezvous hashing](https://www.tinybird.co/blog/kafka-horizontal-scaling) to assign topics to workers: - Each worker calculates a hash based score for each topic - Topics are assigned to the worker with the highest score. When a topic requires N workers, we select the N workers with the highest scores - Kafka's native assignment strategy distributes partitions among the workers handling the same topic (we set `group.instance.id` using the pod name to ensure stable assignments) Combined with stable pod names from StatefulSets, rendezvous hashing minimizes consumer group rebalances during pod restarts. Note that this approach helps with restarts but not with scaling up or down, which would require a different strategy like consistent hashing. {% image src="rendezvous-hashing-kafka.jpeg" alt="Rendezvous hashing reduces the number of connections significantly, especially with a large number of workers and/or topics" caption="Rendezvous hashing reduces the number of connections significantly, especially with a large number of workers and/or topics" /%} ### 2. Circuit Breaker Pattern for Reliability To prevent cascading failures, we implemented the circuit breaker pattern: - Monitors error rates and failure patterns - Opens the circuit when error thresholds are exceeded - Stops consuming when the circuit is open - Automatically retries with backoff periods We classify errors into four types to enable intelligent handling: - **FATAL**: Requires worker restart (configuration errors) - **RECOVERABLE**: Transient errors (network timeouts, broker unavailable) - **USER**: User configuration errors (authentication, authorization). Users are notified via email to fix these issues - **INTERNAL**: Internal processing errors Only recoverable errors trigger the circuit breaker, preventing unnecessary service disruption from user errors or fatal conditions. ### 3. RowBinary Encoding and Ingestion Proxy Messages are encoded into RowBinary format, a compact binary format optimized for ClickHouse. Smaller payload sizes, faster serialization and native type support deliver significant performance benefits over JSON. Our in house C++ library handles type conversion, null handling and binary serialization at native speed, [making ingestion 30% faster](https://www.tinybird.co/blog/improved-rowbinary-conversion). RowBinary data is sent to our ingestion proxy (internally called "gatherer") rather than directly to ClickHouse. This is critical because ClickHouse isn't designed for high frequency streaming inserts. Without buffering, you quickly hit `TOO_MANY_PARTS` errors as each insert creates a new part. The gatherer buffers and batches inserts, preventing parts explosion while maintaining low latency. {% image src="rowbinary-cpu-usage.webp" alt="CPU usage reduction after enabling RowBinary format ingestion" caption="CPU usage reduction after enabling RowBinary format ingestion" /%} ### 4. Flexible Flush Strategies We support three configurable flush triggers (first condition met triggers flush): - **Time based**: Flush after time interval - **Record based**: Flush after N records - **Byte based**: Flush after N bytes This allows fine tuning for low latency or high throughput scenarios ### 5. Enterprise Ready Features Beyond the core architecture, we've built features that enterprise customers expect: - **Schema Registry integration**: Full Confluent Schema Registry support with Avro and JSON Schema formats, backward/forward compatible schema evolution and native logical types (UUID, Datetime, Timestamp) - **Automatic metadata injection**: Every message includes Kafka metadata columns (`__topic`, `__partition`, `__offset`, `__timestamp`, `__key`, `__headers`) for routing, debugging and auditing - **Kafka headers support**: Binary and JSON header formats stored as `Map(String,String)` in ClickHouse, available in SQL queries for routing and tracing. See [Using custom Kafka headers](https://www.tinybird.co/blog/using-custom-kafka-headers) for examples - **Quarantine table**: Failed messages (invalid schemas, oversized, encoding errors) are written to a `_quarantine` table with error details for later analysis - **Zero downtime configuration**: Dynamic updates via Redis pub/sub let you create, update or remove datasources without service restarts ## How we scale and monitor billions of events Running a production connector at scale requires intelligent scaling and comprehensive observability. ### Kubernetes Native Deployment with KEDA Autoscaling Our connector deploys as a Kubernetes StatefulSet (not Deployment) for stable pod names and ordered rollouts. Combined with rendezvous hashing, stable pod names reduce consumer group rebalances during restarts (for example, when rolling out a new version). We operate across multiple cloud and provider regions. For autoscaling, [we leverage KEDA](https://www.tinybird.co/blog/scaling-with-keda-and-tinybird) (Kubernetes Event driven Autoscaling) with multiple trigger types: - **CPU based**: Traditional CPU utilization scaling - **Latency based**: Uses consumer latency histogram from Prometheus Latency based scaling is easier to understand than lag based: it's intuitive that we want to scale when latency exceeds 60 seconds, whereas a lag of 10,000 messages is harder to interpret without context. {% image src="keda.png" alt="Auto-scaling architecture with KEDA" caption="Auto-scaling architecture with KEDA" /%} ### Multi Layered Observability Debugging production issues requires comprehensive observability. Our connector provides three parallel metric streams, all generated from the same source data: **1. User Facing Operations Log (ClickHouse)** Every Tinybird region has an internal ClickHouse cluster where we ingest instrumentation from all our services. The Kafka connector logs every operation to [tinybird.kafka_ops_log](https://www.tinybird.co/docs/forward/monitoring/service-datasources#tinybird-kafka-ops-log), a [Service Data Source](https://www.tinybird.co/docs/forward/monitoring/service-datasources) we expose to users so they can monitor their own Kafka ingestion. This is the same pattern we use for API request metrics, ingestion metrics, storage metrics and more, as described in [How we built Tinybird Wrapped](/blog/how-we-built-tinybird-wrapped). Users can query this log to monitor connector health, identify bottlenecks and track performance. For example, to check consumer lag per partition: ```tinybird -- consumer lag trend over time SELECT toStartOfHour(timestamp) as hour, datasource_id, topic, partition, max(lag) as max_lag, avg(lag) as avg_lag FROM tinybird.kafka_ops_log WHERE timestamp > now() - INTERVAL 24 hour AND partition >= 0 AND msg_type = 'info' GROUP BY hour, datasource_id, topic, partition ORDER BY hour DESC, max_lag DESC ``` For comprehensive monitoring queries including throughput tracking, error monitoring and alerting setup, see the [Kafka monitoring guide](https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring). **2. Prometheus Metrics via Tinybird API** Prometheus metrics are extracted from the same `kafka_ops_log` in our Tinybird instance. We expose a [Tinybird API endpoint in Prometheus format](https://www.tinybird.co/docs/forward/work-with-data/publish-data/guides/consume-api-endpoints-in-prometheus-format) that Grafana polls for monitoring dashboards. Internally, we use this Prometheus endpoint for our latency based auto-scaler. This gives us real time metrics without running a separate Prometheus server, just SQL over ClickHouse formatted as Prometheus. Users can also consume this data to build their own monitoring dashboards. ## Getting Started Getting started with the Tinybird Kafka connector is straightforward: **Create connection** ```tinybird-cli tb connection create kafka ``` **Create Datasource** ```tinybird-cli tb datasource create --kafka ``` **Create an API** ```tinybird # first_kafka_api.pipe TOKEN "kafka_read" READ NODE first_kafka_api_node SQL > SELECT toStartOfMinute(__timestamp) AS minute, count() AS events FROM kafka_events GROUP BY minute ORDER BY minute DESC TYPE endpoint ``` **Deploy and use** ```tinybird-cli tb --cloud deploy curl https://api.tinybird.co/v0/pipes/first_kafka_api.json?token=... ``` That's it! You now have a real-time API consuming from Kafka. For a complete end to end tutorial building a real-time e commerce analytics API from Kafka, including enrichment, materialized views and advanced endpoints, see [Build a Real-Time Ecommerce Analytics API from Kafka in 15 Minutes](https://www.tinybird.co/blog/build-real-time-ecommerce-analytics-api-kafka). ## What we learned Building a production Kafka connector requires solving hard problems: distributed coordination, scaling, reliability, observability and operational simplicity. Our approach focused on: - **Controller/worker architecture** for isolation and scalability - **Rendezvous hashing** for intelligent topic assignment - **Circuit breaker pattern** for fault tolerance - **Zero downtime updates** for operational simplicity - **Comprehensive observability** for debugging and monitoring These design decisions allow the connector to handle billions of events daily while maintaining low latency and high reliability. What started as a simple script has evolved into a robust, production grade connector that powers real-time analytics for enterprises across industries. The journey taught us valuable lessons about balancing competing concerns: throughput vs latency, operational simplicity vs configurability and language performance vs developer productivity. Sometimes the "slower" language (Python) is the right choice when it's not the bottleneck and when it brings other advantages like rapid iteration and excellent ecosystem support. By focusing on the true bottlenecks (network I/O and ClickHouse ingestion) and using native libraries for performance critical components, we built a high performance connector without sacrificing developer productivity. Whether you're processing millions or billions of events, the same principles apply: understand your bottlenecks, design for operability and instrument everything. With the right architecture, even complex systems can appear simple to operate. *Want to try it yourself? Get started with the Kafka connector at [tinybird.co](https://tinybird.co/signup).* {% cta title="We're hiring!" text="If you're passionate about building high-performance data infrastructure, check out our open positions." button={href: "https://tinybird.co/careers", target: "_blank", text: "Go to careers"} /%}