---
title: "ClickHouse® Redpanda Integration for Real-Time Streaming"
excerpt: "ClickHouse® Redpanda integration: real-time streaming with minimal latency, no JVM, and simplified high-performance operations."
authors: "Tinybird"
categories: "AI Resources"
createdOn: "2026-08-02 00:00:00"
publishedOn: "2026-08-02 00:00:00"
updatedOn: "2026-08-02 00:00:00"
status: "published"
---

ClickHouse® and Redpanda integration provides **high-performance event streaming** without JVM overhead. Redpanda delivers **Kafka API compatibility** with **simplified operations**—built-in schema registry, HTTP proxy, and WebAssembly transforms eliminating external dependencies. ClickHouse® consumes events via **native Kafka table engine** enabling **sub-second analytics** over streaming data.

This integration matters because Redpanda offers **lower latency** and **simpler operations** than Apache Kafka while maintaining **full API compatibility**. Organizations achieve **real-time analytics** (millisecond event ingestion) with **fewer moving parts** (no ZooKeeper, no separate schema registry) and **predictable performance** (C++ implementation, thread-per-core architecture).

However, Redpanda isn't just "install and stream to ClickHouse®." Success requires understanding **consumer parallelism limits**, **offset management strategies**, **error handling patterns**, **schema evolution governance**, and **exactly-once semantics**—operational complexity both platforms abstract differently than Kafka ecosystems.

This guide examines ClickHouse® Redpanda integration across deployment patterns—**Kafka table engine consumption**, **Redpanda Connect pipelines**, **HTTP Proxy ingestion**, **Data Transforms preprocessing**, and **ClickHouse® Kafka Connect sink**. We'll show when **direct consumption** makes sense versus **intermediate processing**, what **delivery guarantees** you're accepting, and how to architect streaming analytics delivering **performance without duplicate detection nightmares**.

## **Before Redpanda Integration: Do You Actually Need Event Streaming?**

Most teams evaluating ClickHouse® Redpanda integration face one question: should we use **Kafka table engine** for direct consumption, **Redpanda Connect** for pipeline orchestration, or **Kafka Connect sink** for managed delivery?

Before architecting event streaming with ClickHouse®, consider whether you need streaming infrastructure complexity at all.

Many real-time data ingestion scenarios emerge from connected devices across manufacturing, logistics, and consumer ecosystems. Understanding how the [Internet of Things (IoT)](https://www.ibm.com/think/topics/internet-of-things) shapes streaming data requirements helps determine whether complex brokers like Redpanda are necessary or if managed alternatives such as Tinybird are sufficient.

### **Tinybird: Real-Time Analytics Without Event Streaming Operations**

Tinybird is a **fully managed real-time data platform** built on ClickHouse® that **eliminates event streaming complexity** entirely. Instead of deploying Redpanda clusters, configuring Kafka table engines, managing consumer groups, tuning batch sizes, and coordinating offset commits, you **connect streaming sources directly** and **serve analytics as production APIs**.

**Complete Streaming Pipeline Integrated**

Where Redpanda integration requires **cluster deployment** (brokers, partitions, replication), **ClickHouse® configuration** (Kafka table engines, materialized views, consumer settings), **schema registry management** (Avro contracts, compatibility modes), and **error handling coordination** (dead letter queues, retry policies), Tinybird includes managed Kafka connectors consuming from Confluent Cloud, Apache Kafka, Redpanda, and Amazon MSK automatically, enabling [real-time change data capture](https://www.tinybird.co/blog/real-time-change-data-capture) across diverse streaming sources.

Platform handles **consumer group coordination**, **offset management**, **schema evolution**, and **exactly-once semantics** transparently. No broker tuning, no partition balancing, no offset reset procedures.

**SQL Transformations Without External Processing**

Event streaming architectures typically require **preprocessing layers** (Redpanda Data Transforms, Kafka Streams, Flink) for filtering, enrichment, aggregation before ClickHouse® ingestion. Tinybird replaces external processors with **SQL materialized views** updating incrementally as events arrive.

Define **joins, aggregations, deduplication, windowing** using standard SQL. No WebAssembly modules, no Kafka Streams topologies, no Flink jobs. Transformations execute automatically with **millisecond freshness**.

**Sub-100ms Queries Without Consumer Lag Variability**

ClickHouse® consuming from Redpanda depends on **consumer throughput** (messages/second per partition), **batch sizing** (latency vs efficiency trade-offs), **materialized view performance** (INSERT throughput limits), and **merge overhead** (background compaction delays).

Tinybird delivers **p95 latency under 100ms** automatically over billions of rows. No consumer lag monitoring, no batch size tuning, no merge coordination. **Consistent performance** regardless of event rate or backlog.

**Zero Event Streaming Maintenance**

Redpanda operational burden includes: **cluster monitoring** (broker health, partition leadership, rebalancing events), **consumer management** (lag tracking, offset resets, rebalancing coordination), **schema governance** (compatibility enforcement, version migration), and **capacity planning** (partition count, replication factor, retention policies).

Tinybird **eliminates streaming infrastructure operations**. No Redpanda cluster sizing, no consumer group coordination, no schema registry management. Platform handles **automatic scaling** and **high availability** transparently.

**When Tinybird Makes Sense**

Tinybird is ideal when:

* Real-time analytics APIs primary goal ([real-time dashboards](https://www.tinybird.co/blog/real-time-dashboards-are-they-worth-it), embedded analytics, operational monitoring)  
* **Sub-100ms query latency critical** without event streaming variables affecting performance  
* **Operational burden unacceptable**—team lacks bandwidth for Redpanda cluster operations and Kafka table engine tuning  
* **Development velocity matters**—ship features in hours, not weeks configuring consumer pipelines  
* **Avoiding infrastructure complexity** while maintaining ClickHouse® analytical performance

{% cta title="Build real-time APIs without Redpanda complexity" text="Ingest, transform, and serve analytics with guaranteed latency—zero event streaming operations required" button={href: "https://www.tinybird.co/signup", target: "\_blank", text: "Start building free"} /%}

## **Redpanda as Kafka-Compatible Event Streaming**

Redpanda provides **Kafka API-compatible event streaming** with **architectural simplifications** versus Apache Kafka. Built in **C++ using Seastar framework** with **thread-per-core design** and **Raft consensus** (no ZooKeeper dependency).

### **Key Redpanda Advantages**

**Simplified operations:** Single binary including broker, schema registry, HTTP proxy. No separate JVM processes, no ZooKeeper coordination complexity.

Lower [low latency](https://www.cisco.com/site/us/en/learn/topics/cloud-networking/what-is-low-latency.html) performance: C++ implementation with NVMe-optimized storage and zero-copy networking reducing tail latencies versus JVM-based Kafka.

**Built-in features:** **Schema Registry** (Avro/Protobuf/JSON Schema), **HTTP Proxy** (REST API), **Data Transforms** (WebAssembly preprocessing) included without external dependencies.

**Kafka compatibility:** Clients, connectors, and tools expecting Kafka API work with Redpanda **without modification**—including ClickHouse® Kafka table engine.

## **Primary Integration Pattern: Kafka Table Engine**

ClickHouse® **Kafka table engine** provides native streaming consumption from Redpanda. ClickHouse® explicitly documents **Redpanda compatibility** for Kafka integrations.

### **Architecture (Queue → Materialized View → MergeTree)**

**Recommended pattern:**

1. **Kafka table** (queue consuming from topic)  
2. **Materialized view** (draining queue continuously)  
3. **MergeTree table** (persistent storage for queries)

**Critical understanding:** Kafka engine is **one-shot consumption**—each read advances offsets. Cannot "re-query" without resetting offsets. Materialized view **persists data** to queryable MergeTree table.

### **Basic Configuration Example**

\-- Kafka queue table (consumes from Redpanda)  
CREATE TABLE events\_queue (  
    event\_time DateTime,  
    user\_id UInt64,  
    action LowCardinality(String),  
    payload String  
)  
ENGINE \= Kafka  
SETTINGS  
    kafka\_broker\_list \= 'redpanda-1:9092,redpanda-2:9092,redpanda-3:9092',  
    kafka\_topic\_list \= 'events',  
    kafka\_group\_name \= 'ClickHouse®\_events\_consumer',  
    kafka\_format \= 'JSONEachRow';

\-- Target MergeTree table (persistent storage)  
CREATE TABLE events (  
    event\_time DateTime,  
    user\_id UInt64,  
    action LowCardinality(String),  
    payload String  
)  
ENGINE \= MergeTree()  
PARTITION BY toYYYYMM(event\_time)  
ORDER BY (action, user\_id, event\_time);

\-- Materialized view (pipeline from queue to storage)  
CREATE MATERIALIZED VIEW mv\_events TO events  
AS SELECT  
    event\_time,  
    user\_id,  
    action,  
    payload  
FROM events\_queue;

**Key settings:**

* **`kafka_broker_list`:** Redpanda broker endpoints (comma-separated)  
* **`kafka_topic_list`:** Topics to consume (supports multiple)  
* **`kafka_group_name`:** Consumer group for offset coordination

### **Named Collections (Production Best Practice)**

**Avoid hardcoding credentials** in DDL:

\<\!-- Server configuration \--\>  
\<named\_collections\>  
    \<redpanda\_prod\>  
        \<kafka\_broker\_list\>redpanda-1:9092,redpanda-2:9092\</kafka\_broker\_list\>  
        \<kafka\_topic\_list\>events\</kafka\_topic\_list\>  
        \<kafka\_group\_name\>ClickHouse®\_events\</kafka\_group\_name\>  
        \<kafka\_format\>JSONEachRow\</kafka\_format\>  
    \</redpanda\_prod\>  
\</named\_collections\>

\-- Reference named collection  
CREATE TABLE events\_queue (...)  
ENGINE \= Kafka(redpanda\_prod);

**Benefits:** Centralized configuration, credential hiding, easier rotation without DDL changes.

## **Consumer Parallelism and Performance**

### **Partition Count Limits Parallelism**

**Consumer parallelism** limited by **Redpanda topic partition count**. In clusters, typical pattern: **one Kafka table per ClickHouse® node**, each consuming partitions, writing to **local replicated MergeTree tables**.

**More consumers than partitions** leaves consumers idle—**waste of resources**.

### **Critical Settings for Throughput**

CREATE TABLE events\_queue (...)  
ENGINE \= Kafka  
SETTINGS  
    kafka\_broker\_list \= 'redpanda-1:9092',  
    kafka\_topic\_list \= 'events',  
    kafka\_group\_name \= 'ClickHouse®\_consumer',  
    kafka\_format \= 'JSONEachRow',  
    \-- Parallelism  
    kafka\_num\_consumers \= 4,              \-- Match partition count  
    kafka\_thread\_per\_consumer \= 1,        \-- Parallel flush per consumer  
    \-- Batching  
    kafka\_max\_block\_size \= 65536,         \-- Messages per batch  
    kafka\_poll\_max\_batch\_size \= 65536,    \-- Messages per poll  
    kafka\_poll\_timeout\_ms \= 5000,         \-- Poll timeout  
    kafka\_flush\_interval\_ms \= 7500;       \-- Flush interval

**`kafka_num_consumers`:** Number of consumer threads. **Should not exceed** partition count or physical CPU cores.

**`kafka_thread_per_consumer`:** When enabled, each consumer flushes **independently in parallel** improving throughput for high-volume topics. Disabled by default (consumers aggregate into single block).

**`kafka_max_block_size`:** Maximum messages per batch inserted into MergeTree. **Higher values** improve throughput but increase latency and memory usage.

**Tuning strategy:**

1. Ensure **sufficient partitions** in Redpanda topic  
2. Set `kafka_num_consumers` to **min(partitions, cores)**  
3. Enable `kafka_thread_per_consumer` for **high-throughput** workloads  
4. Adjust `kafka_max_block_size` balancing **latency vs efficiency**

## **Format Selection and Performance**

### **JSONEachRow (Easy, Not Free)**

**JSONEachRow** is simplest format (one JSON object per line) but **CPU-intensive** for parsing at scale.

**Recommendation:** Use for **prototyping** and **low-throughput** pipelines (\<10k messages/second).

### **Binary Formats (Production Scale)**

For **high-throughput production** (100k+ messages/second):

**RowBinary:** ClickHouse® native binary format. Most efficient CPU usage, network overhead, compression.

**Avro (AvroConfluent):** Schema Registry integration, better compatibility governance than JSON.

**Protobuf:** Length-delimited binary format, strong typing, compact encoding.

**Performance difference:** Binary formats reduce **parsing CPU** by 3-10x versus JSON enabling higher throughput per consumer.

## **Schema Registry Integration (Avro)**

Redpanda includes **built-in Schema Registry** compatible with Confluent Schema Registry API. ClickHouse® supports **AvroConfluent format** reading Avro topics with schema evolution.

### **Configuration Example**

CREATE TABLE events\_queue (  
    event\_id String,  
    event\_time DateTime64(3),  
    user\_id UInt64,  
    action String  
)  
ENGINE \= Kafka  
SETTINGS  
    kafka\_broker\_list \= 'redpanda-1:9092',  
    kafka\_topic\_list \= 'events\_avro',  
    kafka\_group\_name \= 'ClickHouse®\_avro\_consumer',  
    kafka\_format \= 'AvroConfluent',  
    format\_avro\_schema\_registry\_url \= 'http://redpanda-1:8081';

### **Schema Compatibility Modes**

Redpanda Schema Registry supports **compatibility levels**:

* **BACKWARD:** New schema reads old data (safe consumer upgrades)  
* **FORWARD:** Old schema reads new data (safe producer upgrades)  
* **FULL:** Both directions (strictest, safest for coordinated changes)  
* **BACKWARD\_TRANSITIVE / FORWARD\_TRANSITIVE / FULL\_TRANSITIVE:** Compatibility across all versions, not just adjacent

**Operational benefit:** **Prevents silent breaking changes** from schema evolution. Governance enforced at registry preventing incompatible schema versions from deployment.

## **Error Handling Strategies**

### **Skip Broken Messages (Simple, Lossy)**

CREATE TABLE events\_queue (...)  
ENGINE \= Kafka  
SETTINGS  
    kafka\_skip\_broken\_messages \= 100,  \-- Skip up to 100 broken per block  
    kafka\_max\_block\_size \= 10000;      \-- Context: per 10k messages

**Behavior:** Skip malformed messages **up to threshold per block**. Exceeding threshold raises exception.

**Trade-off:** **Prevents pipeline blockage** from occasional corrupt messages but **loses data silently**. Not suitable for critical pipelines requiring **every event**.

### **Stream Error Mode (Capture for Analysis)**

CREATE TABLE events\_queue (  
    event\_time DateTime,  
    user\_id UInt64,  
    action String  
)  
ENGINE \= Kafka  
SETTINGS  
    kafka\_broker\_list \= 'redpanda-1:9092',  
    kafka\_topic\_list \= 'events',  
    kafka\_group\_name \= 'ClickHouse®\_consumer',  
    kafka\_format \= 'JSONEachRow',  
    kafka\_handle\_error\_mode \= 'stream';

\-- Target for valid events  
CREATE TABLE events (...)  
ENGINE \= MergeTree() ORDER BY event\_time;

CREATE MATERIALIZED VIEW mv\_events TO events  
AS SELECT event\_time, user\_id, action  
FROM events\_queue  
WHERE \_error \= '';

\-- Capture errors for analysis  
CREATE TABLE parse\_errors (  
    ts DateTime DEFAULT now(),  
    raw\_message String,  
    error\_message String  
)  
ENGINE \= MergeTree() ORDER BY ts;

CREATE MATERIALIZED VIEW mv\_errors TO parse\_errors  
AS SELECT  
    \_raw\_message AS raw\_message,  
    \_error AS error\_message  
FROM events\_queue  
WHERE \_error \!= '';

**Virtual columns** available in `stream` mode:

* **`_raw_message`:** Original message bytes (populated on error)  
* **`_error`:** Exception message (populated on error)

**Benefit:** **Audit all failures** without blocking pipeline. Analyze errors offline determining root causes.

### **Dead Letter Queue (System Table)**

CREATE TABLE events\_queue (...)  
ENGINE \= Kafka  
SETTINGS  
    kafka\_handle\_error\_mode \= 'dead\_letter\_queue';

\-- Errors logged to system.dead\_letter\_queue  
SELECT \*  
FROM system.dead\_letter\_queue  
WHERE table \= 'events\_queue'  
ORDER BY timestamp DESC;

**Errors logged** to `system.dead_letter_queue` system table for operational monitoring without creating custom tables.

## **Observability and Monitoring**

### **Consumer Status Tracking**

\-- Consumer group state per node  
SELECT \*  
FROM system.kafka\_consumers  
WHERE table \= 'events\_queue';

**Critical metrics:**

* **Assigned partitions** per consumer (balance check)  
* **Consumer status** (active, rebalancing, error)  
* **Last poll times** (detect stalled consumers)

**Cluster-wide monitoring:** `system.kafka_consumers` is **per-node**. Query all replicas aggregating for complete picture.

### **Lag Monitoring Strategy**

ClickHouse® doesn't expose **consumer lag** directly. Calculate using **Redpanda metrics** (current offset) minus **ClickHouse® committed offsets**.

**External monitoring:** Use Redpanda metrics endpoints or Kafka tooling (`kafka-consumer-groups`) tracking lag by consumer group.

## **Redpanda Connect Alternative Pattern**

**Redpanda Connect** (evolution of Benthos) provides **declarative pipeline orchestration** with built-in connectors, processors, and error handling.

### **When to Use Redpanda Connect**

Prefer Redpanda Connect over Kafka table engine when:

* **Complex routing** required (fan-out to multiple destinations)  
* **Rich transformations** needed before ClickHouse® (enrichment, filtering, aggregation)  
* **Retry logic** and **DLQ coordination** critical  
* **Multiple sinks** (ClickHouse® \+ S3 \+ another database)

### **ClickHouse® Output Configuration**

Redpanda Connect doesn't have **dedicated ClickHouse® connector** but supports via **`sql_raw` and `sql_insert` components**:

input:  
  kafka:  
    addresses: \[redpanda-1:9092\]  
    topics: \[events\]  
    consumer\_group: redpanda\_connect\_ClickHouse®

pipeline:  
  processors:  
    \- bloblang: |  
        root.event\_time \= this.timestamp  
        root.user\_id \= this.user.id  
        root.action \= this.event\_type

output:  
  sql\_insert:  
    driver: ClickHouse®  
    dsn: "tcp://ClickHouse®:9000/default"  
    table: events  
    columns: \[event\_time, user\_id, action\]  
    args\_mapping: |  
      root \= \[this.event\_time, this.user\_id, this.action\]

**Benefits:**

* **Centralized error handling** (retries, DLQ) outside ClickHouse®  
* **Transformation layer** (Bloblang scripting) before ingestion  
* **Multi-destination routing** without duplicating logic

Many Redpanda Connect pipelines also target relational databases for downstream analytics or archiving. Managed data solutions like [PostgreSQL](https://www.ibm.com/think/topics/postgresql) remain a common sink for structured event storage before aggregation in ClickHouse®.

## **Data Transforms (WebAssembly Preprocessing)**

Redpanda **Data Transforms** execute **WebAssembly modules** on broker nodes preprocessing events before consumers read them.

### **Use Cases**

* **Payload reduction** (strip unnecessary fields before ClickHouse®)  
* **Data masking** (PII redaction at source)  
* **Derivation** (calculate fields from raw events)  
* **Routing** (write to different topics based on criteria)  
* **Format conversion** (transform JSON to Avro)

### **Architecture**

Transforms execute in **same core (shard) as partition leader** with minimal overhead. Each record processed **after persistence** writing to **output topics**.

**Example flow:**

1. Producer writes **raw events** to `events_raw` topic  
2. Data Transform filters, enriches, writes to `events_curated` topic  
3. ClickHouse® consumes from `events_curated` (cleaner, smaller payloads)

**Benefit:** **Reduce ClickHouse® ingestion load** and **storage costs** by preprocessing at source rather than storing raw events then transforming.

## **ClickHouse® Kafka Connect Sink Alternative**

**ClickHouse® Kafka Connect Sink** is **official connector** enabling Kafka Connect ecosystem integration with ClickHouse®.

### **When to Use Kafka Connect Sink**

Prefer Kafka Connect Sink when:

* **Kafka Connect infrastructure** already deployed (CDC, SMTs, existing pipelines)  
* **Exactly-once delivery** critical (connector implements ClickHouse®-specific EOS mechanisms)  
* **Operational separation** desired (delivery logic in Connect, not ClickHouse® cluster)  
* **Backpressure handling** needs external coordination

### **Compatibility with Redpanda**

Redpanda is **Kafka API-compatible**—Kafka Connect deployments work **without modification**. Connector coordinates with Redpanda brokers using standard Kafka protocols.

**Configuration example:**

{  
  "name": "ClickHouse®-sink",  
  "config": {  
    "connector.class": "com.ClickHouse®.kafka.connect.ClickHouse®SinkConnector",  
    "tasks.max": "4",  
    "topics": "events",  
    "hostname": "ClickHouse®.example.com",  
    "port": "8123",  
    "database": "default",  
    "table": "events",  
    "ssl": "true",  
    "username": "default",  
    "password": "password",  
    "key.converter": "org.apache.kafka.connect.json.JsonConverter",  
    "value.converter": "org.apache.kafka.connect.json.JsonConverter"  
  }  
}

## **Exactly-Once Semantics (End-to-End Reality)**

### **Redpanda Exactly-Once Support**

Redpanda supports **idempotent producers** and **transactional semantics** compatible with Kafka's exactly-once (EOS) implementation.

**Idempotent producers:** Assign sequence numbers preventing duplicates from retries.

**Transactions:** Enable atomic **consume-transform-produce** workflows with coordinated offset commits.

### **ClickHouse® Delivery Guarantees**

**At-least-once delivery** is **practical default** for ClickHouse® Kafka table engine. Failures, retries, rebalancing can cause **message redelivery**.

**Achieving end-to-end exactly-once** requires:

1. **Redpanda transactional producer** (EOS at broker)  
2. **Idempotent ClickHouse® writes** (ReplacingMergeTree with deduplication key)  
3. **Application-level coordination** (unique event IDs, version columns)

**Critical understanding:** **Broker EOS doesn't guarantee database uniqueness**. Sink must handle duplicates via **deduplication strategies**.

## **Deduplication Strategies in ClickHouse®**

### **ReplacingMergeTree (Last Write Wins)**

CREATE TABLE events (  
    event\_id String,        \-- Deduplication key  
    event\_time DateTime64(3),  
    user\_id UInt64,  
    action String,  
    version UInt64          \-- Version column  
)  
ENGINE \= ReplacingMergeTree(version)  
PARTITION BY toYYYYMM(event\_time)  
ORDER BY (event\_id, event\_time);

**Behavior:** During merges, **keeps row with highest version** per `ORDER BY` key. **Not immediate**—duplicates visible until background merges complete.

**Query pattern for unique results:**

SELECT \*  
FROM events  
FINAL  \-- Forces deduplication at query time (expensive)  
WHERE event\_time \>= today() \- 7;

\-- Or aggregate-based approach  
SELECT  
    event\_id,  
    argMax(action, version) AS latest\_action,  
    max(version) AS final\_version  
FROM events  
WHERE event\_time \>= today() \- 7  
GROUP BY event\_id;

### **Insert Deduplication (Block-Level)**

\-- Enable insert deduplication  
CREATE TABLE events (...)  
ENGINE \= MergeTree()  
SETTINGS replicated\_deduplication\_window \= 100;

\-- ClickHouse® tracks block checksums preventing duplicate inserts

**Limited scope:** Deduplicates **identical INSERT blocks**. Doesn't prevent duplicates from **different message orderings** or **retries with modified batches**.

## **Security Configuration**

### **TLS and SASL Authentication**

CREATE TABLE events\_queue (...)  
ENGINE \= Kafka  
SETTINGS  
    kafka\_broker\_list \= 'redpanda-1:9092',  
    kafka\_topic\_list \= 'events',  
    kafka\_group\_name \= 'ClickHouse®\_consumer',  
    kafka\_format \= 'JSONEachRow',  
    \-- Security  
    kafka\_security\_protocol \= 'sasl\_ssl',  
    kafka\_sasl\_mechanism \= 'SCRAM-SHA-512',  
    kafka\_sasl\_username \= 'ClickHouse®\_user',  
    kafka\_sasl\_password \= 'secure\_password';

**Supported protocols:**

* **`plaintext`:** No encryption (development only)  
* **`ssl`:** TLS encryption, no authentication  
* **`sasl_plaintext`:** SASL authentication, no encryption  
* **`sasl_ssl`:** TLS encryption \+ SASL authentication (production)

**SASL mechanisms:** PLAIN, SCRAM-SHA-256, SCRAM-SHA-512, GSSAPI (Kerberos), OAUTHBEARER.

### **Redpanda HTTP Proxy (REST API)**

Redpanda **HTTP Proxy** (pandaproxy) exposes **REST API** for producing, consuming, managing consumer groups when Kafka protocol unavailable.

**Use cases:**

* **Network restrictions** preventing Kafka protocol  
* **Lightweight clients** without Kafka libraries  
* **Testing and debugging** workflows  
* **HTTP-only environments** (serverless, edge functions)

**Not recommended for:** High-throughput production ingestion (Kafka protocol far more efficient).

## **Offset Management and Replay**

### **Offset Reset Procedures**

**Kafka table engine** advances offsets automatically. **Replaying events** requires **manual offset reset**.

**External offset reset** (using Kafka tooling):

\# Reset consumer group to earliest  
kafka-consumer-groups \--bootstrap-server redpanda-1:9092 \\  
  \--group ClickHouse®\_events\_consumer \\  
  \--topic events \\  
  \--reset-offsets \--to-earliest \\  
  \--execute

**Backfill strategy:**

1. Create **separate consumer group** for backfill (avoid interfering with production)  
2. Reset offsets to **desired starting point**  
3. Let ClickHouse® consume into **temporary table**  
4. Merge into **production table** after validation

**Never reset production consumer group** during active consumption—causes duplicate processing.

## **Tiered Storage and Retention**

Redpanda **Tiered Storage** offloads log segments to **object storage** (S3, GCS, Azure) enabling **long retention** without local disk costs.

### **Impact on Replay and Recovery**

**Long retention enables:**

* **Event replay** from historical checkpoints (days/weeks/months ago)  
* **Disaster recovery** (reprocess from known good state)  
* **Data lake integration** (archive raw events before transformation)

**Configuration consideration:** **Segment closing** triggered by size (`log_segment_size`) or time. **Low write rate topics** may keep segments open preventing tiered upload.

**Recommendation:** Configure **idle timeout** forcing segment archival for low-throughput topics ensuring timely tiered storage.

## **Production Checklist**

Before deploying ClickHouse® Redpanda integration in production:

* **Match partition count:** Redpanda topic partitions ≥ ClickHouse® consumer count for effective parallelism  
* **Configure consumer settings:** Tune `kafka_num_consumers`, `kafka_max_block_size`, `kafka_thread_per_consumer` for throughput vs latency requirements  
* **Choose format wisely:** Binary formats (Avro, Protobuf, RowBinary) for production scale (100k+ msg/sec), JSON for prototyping  
* **Implement error handling:** Choose `stream` mode or `dead_letter_queue` capturing failures without blocking pipeline  
* **Enable Schema Registry:** Enforce compatibility modes (BACKWARD/FORWARD/FULL) preventing silent breaking changes  
* **Design deduplication:** ReplacingMergeTree with version columns or insert deduplication for at-least-once scenarios  
* **Secure connections:** Use `sasl_ssl` with SCRAM-SHA-512 for production, never `plaintext`  
* **Monitor consumer lag:** External tooling tracking Redpanda offsets vs ClickHouse® committed positions  
* **Plan offset resets:** Separate consumer groups for backfills avoiding production interference  
* **Test rebalancing:** Validate behavior during ClickHouse® node additions/removals (partition reassignment)  
* **Configure retention:** Align Redpanda retention with replay requirements accounting for tiered storage segment closing  
* **Validate exactly-once:** Test duplicate handling under failures, retries, rebalancing scenarios matching production patterns

## **Why Tinybird is the Best ClickHouse® Redpanda Alternative**

After examining ClickHouse® Redpanda integration complexity—**consumer parallelism tuning**, **batch size optimization**, **error handling coordination**, **schema registry governance**, **offset management**, **deduplication strategies**—one pattern emerges: most teams don't need event streaming operational complexity. They need **real-time analytics without infrastructure burden**.

The Redpanda conversation often distracts from the real goal. Engineers spend weeks tuning consumer settings, debugging rebalancing events, coordinating schema migrations, implementing deduplication logic, and monitoring consumer lag when what they actually want is fast queries serving [user-facing analytics](https://www.tinybird.co/blog/user-facing-analytics).

Tinybird cuts through this complexity entirely.

### **Zero Event Streaming Operations**

Redpanda integration requires **operational decisions**: partition count (parallelism limits), consumer settings (throughput vs latency), error handling (skip vs capture vs DLQ), schema governance (compatibility enforcement), offset management (reset procedures, backfill coordination).

Tinybird **eliminates streaming infrastructure operations**. Platform handles **all pipeline decisions** automatically. No broker deployment, no consumer tuning, no schema registry management, no offset coordination.

### **Consistent Performance Without Consumer Variability**

Redpanda consumption depends on **batch sizes** (latency variability), **consumer lag** (backlog accumulation), **rebalancing events** (temporary throughput drops), **materialized view INSERT performance** (merge overhead affecting ingestion).

Tinybird delivers **p95 latency under 100ms** automatically. No consumer lag spikes, no batch size trade-offs, no rebalancing coordination. **Predictable performance** regardless of event rate or pipeline state.

### **Complete Platform vs Streaming Assembly**

ClickHouse® Redpanda requires building and maintaining:

* **Redpanda cluster:** Broker deployment, partition planning, replication configuration  
* **ClickHouse® configuration:** Kafka table engines, materialized views, consumer tuning  
* **Schema governance:** Registry deployment, compatibility enforcement, migration coordination  
* **Monitoring:** Consumer lag tracking, error rate alerting, rebalancing detection

Tinybird provides **complete platform**. Kafka connectors, SQL transformations, authenticated APIs, automatic scaling—**integrated solution without streaming assembly**.

## **Frequently Asked Questions (FAQs)**

### **Does ClickHouse® support Redpanda natively?**

Yes. ClickHouse® **Kafka table engine** is compatible with **Redpanda's Kafka API**. ClickHouse® documentation explicitly mentions Redpanda as supported streaming platform. All Kafka table engine features work with Redpanda without modification.

### **How many consumers should I configure per table?**

**Match partition count** but **never exceed physical CPU cores**. If topic has 12 partitions and server has 16 cores, set `kafka_num_consumers = 12`. More consumers than partitions wastes resources (idle consumers). More consumers than cores causes context switching overhead.

### **What's difference between JSONEachRow and Avro for performance?**

**JSONEachRow** simple but **CPU-intensive** (3-10x slower parsing). Use for **prototyping** or **low throughput** (\<10k msg/sec). **Avro** (AvroConfluent) provides **binary efficiency** plus **schema governance** via Schema Registry. Recommended for **production scale** (100k+ msg/sec) and **schema evolution requirements**.

### **How do I handle duplicate messages from Redpanda?**

Redpanda supports **idempotent producers** and **transactions** but ClickHouse® Kafka table engine provides **at-least-once delivery**. Handle duplicates using **ReplacingMergeTree** (merges keep latest version) or **application-level deduplication** (unique event IDs). Query with `FINAL` or **aggregate functions** (`argMax`) for unique results.

### **What's best error handling strategy?**

**`kafka_handle_error_mode = 'stream'`** for critical pipelines requiring **error visibility**. Creates parallel materialized view routing failures (using `_error`, `_raw_message` virtual columns) to audit table. **`kafka_skip_broken_messages`** acceptable for **non-critical** pipelines tolerating occasional data loss. **`dead_letter_queue`** logs errors to `system.dead_letter_queue` for operational monitoring.

### **Can I use Redpanda Schema Registry with ClickHouse®?**

Yes. ClickHouse® **AvroConfluent format** integrates with Redpanda Schema Registry via `format_avro_schema_registry_url` setting. Supports **compatibility modes** (BACKWARD, FORWARD, FULL, TRANSITIVE variants) enforcing schema evolution governance preventing silent breaking changes.

### **How do I reset consumer offsets for replay?**

Use **Kafka tooling** (`kafka-consumer-groups` or Redpanda CLI) resetting offsets for consumer group. Create **separate consumer group** for backfills avoiding production interference. Example: `kafka-consumer-groups --bootstrap-server redpanda:9092 --group ClickHouse®_backfill --topic events --reset-offsets --to-earliest --execute`. **Never reset** production consumer group during active consumption.

### **Should I use Kafka table engine or Kafka Connect sink?**

**Kafka table engine** for **direct consumption** with ClickHouse®-native configuration and monitoring. **Kafka Connect sink** when **existing Connect infrastructure** deployed or **exactly-once delivery** critical (connector implements ClickHouse®-specific EOS). Both work with Redpanda due to **Kafka API compatibility**.