ClickHouse® and Redpanda integration provides high-performance event streaming without JVM overhead. Redpanda delivers Kafka API compatibility with simplified operations—built-in schema registry, HTTP proxy, and WebAssembly transforms eliminating external dependencies. ClickHouse® consumes events via native Kafka table engine enabling sub-second analytics over streaming data.
This integration matters because Redpanda offers lower latency and simpler operations than Apache Kafka while maintaining full API compatibility. Organizations achieve real-time analytics (millisecond event ingestion) with fewer moving parts (no ZooKeeper, no separate schema registry) and predictable performance (C++ implementation, thread-per-core architecture).
However, Redpanda isn't just "install and stream to ClickHouse®." Success requires understanding consumer parallelism limits, offset management strategies, error handling patterns, schema evolution governance, and exactly-once semantics—operational complexity both platforms abstract differently than Kafka ecosystems.
This guide examines ClickHouse® Redpanda integration across deployment patterns—Kafka table engine consumption, Redpanda Connect pipelines, HTTP Proxy ingestion, Data Transforms preprocessing, and ClickHouse® Kafka Connect sink. We'll show when direct consumption makes sense versus intermediate processing, what delivery guarantees you're accepting, and how to architect streaming analytics delivering performance without duplicate detection nightmares.
Before Redpanda Integration: Do You Actually Need Event Streaming?
Most teams evaluating ClickHouse® Redpanda integration face one question: should we use Kafka table engine for direct consumption, Redpanda Connect for pipeline orchestration, or Kafka Connect sink for managed delivery?
Before architecting event streaming with ClickHouse®, consider whether you need streaming infrastructure complexity at all.
Many real-time data ingestion scenarios emerge from connected devices across manufacturing, logistics, and consumer ecosystems. Understanding how the Internet of Things (IoT) shapes streaming data requirements helps determine whether complex brokers like Redpanda are necessary or if managed alternatives such as Tinybird are sufficient.
Tinybird: Real-Time Analytics Without Event Streaming Operations
Tinybird is a fully managed real-time data platform built on ClickHouse® that eliminates event streaming complexity entirely. Instead of deploying Redpanda clusters, configuring Kafka table engines, managing consumer groups, tuning batch sizes, and coordinating offset commits, you connect streaming sources directly and serve analytics as production APIs.
Complete Streaming Pipeline Integrated
Where Redpanda integration requires cluster deployment (brokers, partitions, replication), ClickHouse® configuration (Kafka table engines, materialized views, consumer settings), schema registry management (Avro contracts, compatibility modes), and error handling coordination (dead letter queues, retry policies), Tinybird includes managed Kafka connectors consuming from Confluent Cloud, Apache Kafka, Redpanda, and Amazon MSK automatically, enabling real-time change data capture across diverse streaming sources.
Platform handles consumer group coordination, offset management, schema evolution, and exactly-once semantics transparently. No broker tuning, no partition balancing, no offset reset procedures.
SQL Transformations Without External Processing
Event streaming architectures typically require preprocessing layers (Redpanda Data Transforms, Kafka Streams, Flink) for filtering, enrichment, aggregation before ClickHouse® ingestion. Tinybird replaces external processors with SQL materialized views updating incrementally as events arrive.
Define joins, aggregations, deduplication, windowing using standard SQL. No WebAssembly modules, no Kafka Streams topologies, no Flink jobs. Transformations execute automatically with millisecond freshness.
Sub-100ms Queries Without Consumer Lag Variability
ClickHouse® consuming from Redpanda depends on consumer throughput (messages/second per partition), batch sizing (latency vs efficiency trade-offs), materialized view performance (INSERT throughput limits), and merge overhead (background compaction delays).
Tinybird delivers p95 latency under 100ms automatically over billions of rows. No consumer lag monitoring, no batch size tuning, no merge coordination. Consistent performance regardless of event rate or backlog.
Zero Event Streaming Maintenance
Redpanda operational burden includes: cluster monitoring (broker health, partition leadership, rebalancing events), consumer management (lag tracking, offset resets, rebalancing coordination), schema governance (compatibility enforcement, version migration), and capacity planning (partition count, replication factor, retention policies).
Tinybird eliminates streaming infrastructure operations. No Redpanda cluster sizing, no consumer group coordination, no schema registry management. Platform handles automatic scaling and high availability transparently.
When Tinybird Makes Sense
Tinybird is ideal when:
- Real-time analytics APIs primary goal (real-time dashboards, embedded analytics, operational monitoring)
- Sub-100ms query latency critical without event streaming variables affecting performance
- Operational burden unacceptable—team lacks bandwidth for Redpanda cluster operations and Kafka table engine tuning
- Development velocity matters—ship features in hours, not weeks configuring consumer pipelines
- Avoiding infrastructure complexity while maintaining ClickHouse® analytical performance
Redpanda as Kafka-Compatible Event Streaming
Redpanda provides Kafka API-compatible event streaming with architectural simplifications versus Apache Kafka. Built in C++ using Seastar framework with thread-per-core design and Raft consensus (no ZooKeeper dependency).
Key Redpanda Advantages
Simplified operations: Single binary including broker, schema registry, HTTP proxy. No separate JVM processes, no ZooKeeper coordination complexity.
Lower low latency performance: C++ implementation with NVMe-optimized storage and zero-copy networking reducing tail latencies versus JVM-based Kafka.
Built-in features: Schema Registry (Avro/Protobuf/JSON Schema), HTTP Proxy (REST API), Data Transforms (WebAssembly preprocessing) included without external dependencies.
Kafka compatibility: Clients, connectors, and tools expecting Kafka API work with Redpanda without modification—including ClickHouse® Kafka table engine.
Primary Integration Pattern: Kafka Table Engine
ClickHouse® Kafka table engine provides native streaming consumption from Redpanda. ClickHouse® explicitly documents Redpanda compatibility for Kafka integrations.
Architecture (Queue → Materialized View → MergeTree)
Recommended pattern:
- Kafka table (queue consuming from topic)
- Materialized view (draining queue continuously)
- MergeTree table (persistent storage for queries)
Critical understanding: Kafka engine is one-shot consumption—each read advances offsets. Cannot "re-query" without resetting offsets. Materialized view persists data to queryable MergeTree table.
Basic Configuration Example
-- Kafka queue table (consumes from Redpanda)
CREATE TABLE events_queue (
event_time DateTime,
user_id UInt64,
action LowCardinality(String),
payload String
)
ENGINE = Kafka
SETTINGS
kafka_broker_list = 'redpanda-1:9092,redpanda-2:9092,redpanda-3:9092',
kafka_topic_list = 'events',
kafka_group_name = 'ClickHouse®_events_consumer',
kafka_format = 'JSONEachRow';
-- Target MergeTree table (persistent storage)
CREATE TABLE events (
event_time DateTime,
user_id UInt64,
action LowCardinality(String),
payload String
)
ENGINE = MergeTree()
PARTITION BY toYYYYMM(event_time)
ORDER BY (action, user_id, event_time);
-- Materialized view (pipeline from queue to storage)
CREATE MATERIALIZED VIEW mv_events TO events
AS SELECT
event_time,
user_id,
action,
payload
FROM events_queue;
Key settings:
kafka_broker_list: Redpanda broker endpoints (comma-separated)kafka_topic_list: Topics to consume (supports multiple)kafka_group_name: Consumer group for offset coordination
Named Collections (Production Best Practice)
Avoid hardcoding credentials in DDL:
<!-- Server configuration -->
<named_collections>
<redpanda_prod>
<kafka_broker_list>redpanda-1:9092,redpanda-2:9092</kafka_broker_list>
<kafka_topic_list>events</kafka_topic_list>
<kafka_group_name>ClickHouse®_events</kafka_group_name>
<kafka_format>JSONEachRow</kafka_format>
</redpanda_prod>
</named_collections>
-- Reference named collection
CREATE TABLE events_queue (...)
ENGINE = Kafka(redpanda_prod);
Benefits: Centralized configuration, credential hiding, easier rotation without DDL changes.
Consumer Parallelism and Performance
Partition Count Limits Parallelism
Consumer parallelism limited by Redpanda topic partition count. In clusters, typical pattern: one Kafka table per ClickHouse® node, each consuming partitions, writing to local replicated MergeTree tables.
More consumers than partitions leaves consumers idle—waste of resources.
Critical Settings for Throughput
CREATE TABLE events_queue (...)
ENGINE = Kafka
SETTINGS
kafka_broker_list = 'redpanda-1:9092',
kafka_topic_list = 'events',
kafka_group_name = 'ClickHouse®_consumer',
kafka_format = 'JSONEachRow',
-- Parallelism
kafka_num_consumers = 4, -- Match partition count
kafka_thread_per_consumer = 1, -- Parallel flush per consumer
-- Batching
kafka_max_block_size = 65536, -- Messages per batch
kafka_poll_max_batch_size = 65536, -- Messages per poll
kafka_poll_timeout_ms = 5000, -- Poll timeout
kafka_flush_interval_ms = 7500; -- Flush interval
kafka_num_consumers: Number of consumer threads. Should not exceed partition count or physical CPU cores.
kafka_thread_per_consumer: When enabled, each consumer flushes independently in parallel improving throughput for high-volume topics. Disabled by default (consumers aggregate into single block).
kafka_max_block_size: Maximum messages per batch inserted into MergeTree. Higher values improve throughput but increase latency and memory usage.
Tuning strategy:
- Ensure sufficient partitions in Redpanda topic
- Set
kafka_num_consumersto min(partitions, cores) - Enable
kafka_thread_per_consumerfor high-throughput workloads - Adjust
kafka_max_block_sizebalancing latency vs efficiency
Format Selection and Performance
JSONEachRow (Easy, Not Free)
JSONEachRow is simplest format (one JSON object per line) but CPU-intensive for parsing at scale.
Recommendation: Use for prototyping and low-throughput pipelines (<10k messages/second).
Binary Formats (Production Scale)
For high-throughput production (100k+ messages/second):
RowBinary: ClickHouse® native binary format. Most efficient CPU usage, network overhead, compression.
Avro (AvroConfluent): Schema Registry integration, better compatibility governance than JSON.
Protobuf: Length-delimited binary format, strong typing, compact encoding.
Performance difference: Binary formats reduce parsing CPU by 3-10x versus JSON enabling higher throughput per consumer.
Schema Registry Integration (Avro)
Redpanda includes built-in Schema Registry compatible with Confluent Schema Registry API. ClickHouse® supports AvroConfluent format reading Avro topics with schema evolution.
Configuration Example
CREATE TABLE events_queue (
event_id String,
event_time DateTime64(3),
user_id UInt64,
action String
)
ENGINE = Kafka
SETTINGS
kafka_broker_list = 'redpanda-1:9092',
kafka_topic_list = 'events_avro',
kafka_group_name = 'ClickHouse®_avro_consumer',
kafka_format = 'AvroConfluent',
format_avro_schema_registry_url = 'http://redpanda-1:8081';
Schema Compatibility Modes
Redpanda Schema Registry supports compatibility levels:
- BACKWARD: New schema reads old data (safe consumer upgrades)
- FORWARD: Old schema reads new data (safe producer upgrades)
- FULL: Both directions (strictest, safest for coordinated changes)
- BACKWARD_TRANSITIVE / FORWARD_TRANSITIVE / FULL_TRANSITIVE: Compatibility across all versions, not just adjacent
Operational benefit: Prevents silent breaking changes from schema evolution. Governance enforced at registry preventing incompatible schema versions from deployment.
Error Handling Strategies
Skip Broken Messages (Simple, Lossy)
CREATE TABLE events_queue (...)
ENGINE = Kafka
SETTINGS
kafka_skip_broken_messages = 100, -- Skip up to 100 broken per block
kafka_max_block_size = 10000; -- Context: per 10k messages
Behavior: Skip malformed messages up to threshold per block. Exceeding threshold raises exception.
Trade-off: Prevents pipeline blockage from occasional corrupt messages but loses data silently. Not suitable for critical pipelines requiring every event.
Stream Error Mode (Capture for Analysis)
CREATE TABLE events_queue (
event_time DateTime,
user_id UInt64,
action String
)
ENGINE = Kafka
SETTINGS
kafka_broker_list = 'redpanda-1:9092',
kafka_topic_list = 'events',
kafka_group_name = 'ClickHouse®_consumer',
kafka_format = 'JSONEachRow',
kafka_handle_error_mode = 'stream';
-- Target for valid events
CREATE TABLE events (...)
ENGINE = MergeTree() ORDER BY event_time;
CREATE MATERIALIZED VIEW mv_events TO events
AS SELECT event_time, user_id, action
FROM events_queue
WHERE _error = '';
-- Capture errors for analysis
CREATE TABLE parse_errors (
ts DateTime DEFAULT now(),
raw_message String,
error_message String
)
ENGINE = MergeTree() ORDER BY ts;
CREATE MATERIALIZED VIEW mv_errors TO parse_errors
AS SELECT
_raw_message AS raw_message,
_error AS error_message
FROM events_queue
WHERE _error != '';
Virtual columns available in stream mode:
_raw_message: Original message bytes (populated on error)_error: Exception message (populated on error)
Benefit: Audit all failures without blocking pipeline. Analyze errors offline determining root causes.
Dead Letter Queue (System Table)
CREATE TABLE events_queue (...)
ENGINE = Kafka
SETTINGS
kafka_handle_error_mode = 'dead_letter_queue';
-- Errors logged to system.dead_letter_queue
SELECT *
FROM system.dead_letter_queue
WHERE table = 'events_queue'
ORDER BY timestamp DESC;
Errors logged to system.dead_letter_queue system table for operational monitoring without creating custom tables.
Observability and Monitoring
Consumer Status Tracking
-- Consumer group state per node
SELECT *
FROM system.kafka_consumers
WHERE table = 'events_queue';
Critical metrics:
- Assigned partitions per consumer (balance check)
- Consumer status (active, rebalancing, error)
- Last poll times (detect stalled consumers)
Cluster-wide monitoring: system.kafka_consumers is per-node. Query all replicas aggregating for complete picture.
Lag Monitoring Strategy
ClickHouse® doesn't expose consumer lag directly. Calculate using Redpanda metrics (current offset) minus ClickHouse® committed offsets.
External monitoring: Use Redpanda metrics endpoints or Kafka tooling (kafka-consumer-groups) tracking lag by consumer group.
Redpanda Connect Alternative Pattern
Redpanda Connect (evolution of Benthos) provides declarative pipeline orchestration with built-in connectors, processors, and error handling.
When to Use Redpanda Connect
Prefer Redpanda Connect over Kafka table engine when:
- Complex routing required (fan-out to multiple destinations)
- Rich transformations needed before ClickHouse® (enrichment, filtering, aggregation)
- Retry logic and DLQ coordination critical
- Multiple sinks (ClickHouse® + S3 + another database)
ClickHouse® Output Configuration
Redpanda Connect doesn't have dedicated ClickHouse® connector but supports via sql_raw and sql_insert components:
input:
kafka:
addresses: [redpanda-1:9092]
topics: [events]
consumer_group: redpanda_connect_ClickHouse®
pipeline:
processors:
- bloblang: |
root.event_time = this.timestamp
root.user_id = this.user.id
root.action = this.event_type
output:
sql_insert:
driver: ClickHouse®
dsn: "tcp://ClickHouse®:9000/default"
table: events
columns: [event_time, user_id, action]
args_mapping: |
root = [this.event_time, this.user_id, this.action]
Benefits:
- Centralized error handling (retries, DLQ) outside ClickHouse®
- Transformation layer (Bloblang scripting) before ingestion
- Multi-destination routing without duplicating logic
Many Redpanda Connect pipelines also target relational databases for downstream analytics or archiving. Managed data solutions like PostgreSQL remain a common sink for structured event storage before aggregation in ClickHouse®.
Data Transforms (WebAssembly Preprocessing)
Redpanda Data Transforms execute WebAssembly modules on broker nodes preprocessing events before consumers read them.
Use Cases
- Payload reduction (strip unnecessary fields before ClickHouse®)
- Data masking (PII redaction at source)
- Derivation (calculate fields from raw events)
- Routing (write to different topics based on criteria)
- Format conversion (transform JSON to Avro)
Architecture
Transforms execute in same core (shard) as partition leader with minimal overhead. Each record processed after persistence writing to output topics.
Example flow:
- Producer writes raw events to
events_rawtopic - Data Transform filters, enriches, writes to
events_curatedtopic - ClickHouse® consumes from
events_curated(cleaner, smaller payloads)
Benefit: Reduce ClickHouse® ingestion load and storage costs by preprocessing at source rather than storing raw events then transforming.
ClickHouse® Kafka Connect Sink Alternative
ClickHouse® Kafka Connect Sink is official connector enabling Kafka Connect ecosystem integration with ClickHouse®.
When to Use Kafka Connect Sink
Prefer Kafka Connect Sink when:
- Kafka Connect infrastructure already deployed (CDC, SMTs, existing pipelines)
- Exactly-once delivery critical (connector implements ClickHouse®-specific EOS mechanisms)
- Operational separation desired (delivery logic in Connect, not ClickHouse® cluster)
- Backpressure handling needs external coordination
Compatibility with Redpanda
Redpanda is Kafka API-compatible—Kafka Connect deployments work without modification. Connector coordinates with Redpanda brokers using standard Kafka protocols.
Configuration example:
{
"name": "ClickHouse®-sink",
"config": {
"connector.class": "com.ClickHouse®.kafka.connect.ClickHouse®SinkConnector",
"tasks.max": "4",
"topics": "events",
"hostname": "ClickHouse®.example.com",
"port": "8123",
"database": "default",
"table": "events",
"ssl": "true",
"username": "default",
"password": "password",
"key.converter": "org.apache.kafka.connect.json.JsonConverter",
"value.converter": "org.apache.kafka.connect.json.JsonConverter"
}
}
Exactly-Once Semantics (End-to-End Reality)
Redpanda Exactly-Once Support
Redpanda supports idempotent producers and transactional semantics compatible with Kafka's exactly-once (EOS) implementation.
Idempotent producers: Assign sequence numbers preventing duplicates from retries.
Transactions: Enable atomic consume-transform-produce workflows with coordinated offset commits.
ClickHouse® Delivery Guarantees
At-least-once delivery is practical default for ClickHouse® Kafka table engine. Failures, retries, rebalancing can cause message redelivery.
Achieving end-to-end exactly-once requires:
- Redpanda transactional producer (EOS at broker)
- Idempotent ClickHouse® writes (ReplacingMergeTree with deduplication key)
- Application-level coordination (unique event IDs, version columns)
Critical understanding: Broker EOS doesn't guarantee database uniqueness. Sink must handle duplicates via deduplication strategies.
Deduplication Strategies in ClickHouse®
ReplacingMergeTree (Last Write Wins)
CREATE TABLE events (
event_id String, -- Deduplication key
event_time DateTime64(3),
user_id UInt64,
action String,
version UInt64 -- Version column
)
ENGINE = ReplacingMergeTree(version)
PARTITION BY toYYYYMM(event_time)
ORDER BY (event_id, event_time);
Behavior: During merges, keeps row with highest version per ORDER BY key. Not immediate—duplicates visible until background merges complete.
Query pattern for unique results:
SELECT *
FROM events
FINAL -- Forces deduplication at query time (expensive)
WHERE event_time >= today() - 7;
-- Or aggregate-based approach
SELECT
event_id,
argMax(action, version) AS latest_action,
max(version) AS final_version
FROM events
WHERE event_time >= today() - 7
GROUP BY event_id;
Insert Deduplication (Block-Level)
-- Enable insert deduplication
CREATE TABLE events (...)
ENGINE = MergeTree()
SETTINGS replicated_deduplication_window = 100;
-- ClickHouse® tracks block checksums preventing duplicate inserts
Limited scope: Deduplicates identical INSERT blocks. Doesn't prevent duplicates from different message orderings or retries with modified batches.
Security Configuration
TLS and SASL Authentication
CREATE TABLE events_queue (...)
ENGINE = Kafka
SETTINGS
kafka_broker_list = 'redpanda-1:9092',
kafka_topic_list = 'events',
kafka_group_name = 'ClickHouse®_consumer',
kafka_format = 'JSONEachRow',
-- Security
kafka_security_protocol = 'sasl_ssl',
kafka_sasl_mechanism = 'SCRAM-SHA-512',
kafka_sasl_username = 'ClickHouse®_user',
kafka_sasl_password = 'secure_password';
Supported protocols:
plaintext: No encryption (development only)ssl: TLS encryption, no authenticationsasl_plaintext: SASL authentication, no encryptionsasl_ssl: TLS encryption + SASL authentication (production)
SASL mechanisms: PLAIN, SCRAM-SHA-256, SCRAM-SHA-512, GSSAPI (Kerberos), OAUTHBEARER.
Redpanda HTTP Proxy (REST API)
Redpanda HTTP Proxy (pandaproxy) exposes REST API for producing, consuming, managing consumer groups when Kafka protocol unavailable.
Use cases:
- Network restrictions preventing Kafka protocol
- Lightweight clients without Kafka libraries
- Testing and debugging workflows
- HTTP-only environments (serverless, edge functions)
Not recommended for: High-throughput production ingestion (Kafka protocol far more efficient).
Offset Management and Replay
Offset Reset Procedures
Kafka table engine advances offsets automatically. Replaying events requires manual offset reset.
External offset reset (using Kafka tooling):
# Reset consumer group to earliest
kafka-consumer-groups --bootstrap-server redpanda-1:9092 \
--group ClickHouse®_events_consumer \
--topic events \
--reset-offsets --to-earliest \
--execute
Backfill strategy:
- Create separate consumer group for backfill (avoid interfering with production)
- Reset offsets to desired starting point
- Let ClickHouse® consume into temporary table
- Merge into production table after validation
Never reset production consumer group during active consumption—causes duplicate processing.
Tiered Storage and Retention
Redpanda Tiered Storage offloads log segments to object storage (S3, GCS, Azure) enabling long retention without local disk costs.
Impact on Replay and Recovery
Long retention enables:
- Event replay from historical checkpoints (days/weeks/months ago)
- Disaster recovery (reprocess from known good state)
- Data lake integration (archive raw events before transformation)
Configuration consideration: Segment closing triggered by size (log_segment_size) or time. Low write rate topics may keep segments open preventing tiered upload.
Recommendation: Configure idle timeout forcing segment archival for low-throughput topics ensuring timely tiered storage.
Production Checklist
Before deploying ClickHouse® Redpanda integration in production:
- Match partition count: Redpanda topic partitions ≥ ClickHouse® consumer count for effective parallelism
- Configure consumer settings: Tune
kafka_num_consumers,kafka_max_block_size,kafka_thread_per_consumerfor throughput vs latency requirements - Choose format wisely: Binary formats (Avro, Protobuf, RowBinary) for production scale (100k+ msg/sec), JSON for prototyping
- Implement error handling: Choose
streammode ordead_letter_queuecapturing failures without blocking pipeline - Enable Schema Registry: Enforce compatibility modes (BACKWARD/FORWARD/FULL) preventing silent breaking changes
- Design deduplication: ReplacingMergeTree with version columns or insert deduplication for at-least-once scenarios
- Secure connections: Use
sasl_sslwith SCRAM-SHA-512 for production, neverplaintext - Monitor consumer lag: External tooling tracking Redpanda offsets vs ClickHouse® committed positions
- Plan offset resets: Separate consumer groups for backfills avoiding production interference
- Test rebalancing: Validate behavior during ClickHouse® node additions/removals (partition reassignment)
- Configure retention: Align Redpanda retention with replay requirements accounting for tiered storage segment closing
- Validate exactly-once: Test duplicate handling under failures, retries, rebalancing scenarios matching production patterns
Why Tinybird is the Best ClickHouse® Redpanda Alternative
After examining ClickHouse® Redpanda integration complexity—consumer parallelism tuning, batch size optimization, error handling coordination, schema registry governance, offset management, deduplication strategies—one pattern emerges: most teams don't need event streaming operational complexity. They need real-time analytics without infrastructure burden.
The Redpanda conversation often distracts from the real goal. Engineers spend weeks tuning consumer settings, debugging rebalancing events, coordinating schema migrations, implementing deduplication logic, and monitoring consumer lag when what they actually want is fast queries serving user-facing analytics.
Tinybird cuts through this complexity entirely.
Zero Event Streaming Operations
Redpanda integration requires operational decisions: partition count (parallelism limits), consumer settings (throughput vs latency), error handling (skip vs capture vs DLQ), schema governance (compatibility enforcement), offset management (reset procedures, backfill coordination).
Tinybird eliminates streaming infrastructure operations. Platform handles all pipeline decisions automatically. No broker deployment, no consumer tuning, no schema registry management, no offset coordination.
Consistent Performance Without Consumer Variability
Redpanda consumption depends on batch sizes (latency variability), consumer lag (backlog accumulation), rebalancing events (temporary throughput drops), materialized view INSERT performance (merge overhead affecting ingestion).
Tinybird delivers p95 latency under 100ms automatically. No consumer lag spikes, no batch size trade-offs, no rebalancing coordination. Predictable performance regardless of event rate or pipeline state.
Complete Platform vs Streaming Assembly
ClickHouse® Redpanda requires building and maintaining:
- Redpanda cluster: Broker deployment, partition planning, replication configuration
- ClickHouse® configuration: Kafka table engines, materialized views, consumer tuning
- Schema governance: Registry deployment, compatibility enforcement, migration coordination
- Monitoring: Consumer lag tracking, error rate alerting, rebalancing detection
Tinybird provides complete platform. Kafka connectors, SQL transformations, authenticated APIs, automatic scaling—integrated solution without streaming assembly.
Frequently Asked Questions (FAQs)
Does ClickHouse® support Redpanda natively?
Yes. ClickHouse® Kafka table engine is compatible with Redpanda's Kafka API. ClickHouse® documentation explicitly mentions Redpanda as supported streaming platform. All Kafka table engine features work with Redpanda without modification.
How many consumers should I configure per table?
Match partition count but never exceed physical CPU cores. If topic has 12 partitions and server has 16 cores, set kafka_num_consumers = 12. More consumers than partitions wastes resources (idle consumers). More consumers than cores causes context switching overhead.
What's difference between JSONEachRow and Avro for performance?
JSONEachRow simple but CPU-intensive (3-10x slower parsing). Use for prototyping or low throughput (<10k msg/sec). Avro (AvroConfluent) provides binary efficiency plus schema governance via Schema Registry. Recommended for production scale (100k+ msg/sec) and schema evolution requirements.
How do I handle duplicate messages from Redpanda?
Redpanda supports idempotent producers and transactions but ClickHouse® Kafka table engine provides at-least-once delivery. Handle duplicates using ReplacingMergeTree (merges keep latest version) or application-level deduplication (unique event IDs). Query with FINAL or aggregate functions (argMax) for unique results.
What's best error handling strategy?
kafka_handle_error_mode = 'stream' for critical pipelines requiring error visibility. Creates parallel materialized view routing failures (using _error, _raw_message virtual columns) to audit table. kafka_skip_broken_messages acceptable for non-critical pipelines tolerating occasional data loss. dead_letter_queue logs errors to system.dead_letter_queue for operational monitoring.
Can I use Redpanda Schema Registry with ClickHouse®?
Yes. ClickHouse® AvroConfluent format integrates with Redpanda Schema Registry via format_avro_schema_registry_url setting. Supports compatibility modes (BACKWARD, FORWARD, FULL, TRANSITIVE variants) enforcing schema evolution governance preventing silent breaking changes.
How do I reset consumer offsets for replay?
Use Kafka tooling (kafka-consumer-groups or Redpanda CLI) resetting offsets for consumer group. Create separate consumer group for backfills avoiding production interference. Example: kafka-consumer-groups --bootstrap-server redpanda:9092 --group ClickHouse®_backfill --topic events --reset-offsets --to-earliest --execute. Never reset production consumer group during active consumption.
Should I use Kafka table engine or Kafka Connect sink?
Kafka table engine for direct consumption with ClickHouse®-native configuration and monitoring. Kafka Connect sink when existing Connect infrastructure deployed or exactly-once delivery critical (connector implements ClickHouse®-specific EOS). Both work with Redpanda due to Kafka API compatibility.