Kafka partitioning strategies¶

This guide covers best practices for Kafka topic partitioning and how it affects Tinybird's Kafka connector performance.

Understanding Kafka partitions¶

Kafka topics are divided into partitions, which allow:

• Parallel processing - Multiple consumers can process different partitions simultaneously
• Scalability - Distribute load across partitions
• Ordering - Messages with the same key go to the same partition, maintaining order

Tinybird's Kafka connector automatically assigns partitions to consumer instances and scales consumers based on lag and load.

Kafka partitions vs ClickHouse® table partitions¶

Kafka partitions and ClickHouse® table partitions are different concepts:

• Kafka partitions: Logical divisions of a topic for parallel processing and ordering. Each Kafka partition can be consumed independently.
• ClickHouse® table partitions: Physical divisions of data in ClickHouse® tables, typically organized by time (for example, monthly partitions).

How they relate:

• Each Kafka partition can write data to multiple ClickHouse® parts within a ClickHouse® partition
• Multiple Kafka partitions can write to the same ClickHouse® partition
• ClickHouse® typically partitions by time (for example, ENGINE_PARTITION_KEY "toYYYYMM(timestamp)")

TOO_MANY_PARTS error:

If you have too many Kafka partitions writing to the same ClickHouse® partition, you may encounter the TOO_MANY_PARTS error. This happens when ClickHouse® creates too many parts in a single partition, which can impact query performance.

Best practice: For ClickHouse®, partitioning by time (for example, monthly or daily) is usually fine for time-series data. This is independent of your Kafka partition key strategy. Kafka partition keys should focus on distributing load evenly across Kafka partitions, while ClickHouse® partitioning organizes data by time for efficient querying.

Partition design best practices¶

Number of partitions¶

Guidelines:

• Start with 3-6 partitions for most use cases
• Scale up if you see uneven lag distribution
• More partitions = more parallelism, but also more overhead
• Too many partitions can cause frequent rebalancing and ClickHouse® TOO_MANY_PARTS errors

Factors to consider:

• Expected message throughput
• Consumer capacity
• ClickHouse® partition strategy

Kafka partition key strategy¶

The partition key determines which Kafka partition a message goes to. This is about Kafka partitions, not ClickHouse partitions.

Goal: Even distribution across Kafka partitions

• Use keys that distribute messages evenly across Kafka partitions
• Avoid keys that create "hot partitions" (one Kafka partition receiving most messages)
• Consider hash-based keys for even distribution

Example of good partition key:

# Even distribution using hash - distributes messages across Kafka partitions
partition_key = hash(user_id) % num_partitions

Example of problematic partition key:

# Creates hot partition - all messages go to one Kafka partition
partition_key = "all-users"

Note: Partition keys are for distributing load across Kafka partitions. For ClickHouse® table partitioning, you typically partition by time (for example, ENGINE_PARTITION_KEY "toYYYYMM(timestamp)"), which is usually fine for time-series data. ClickHouse® partitioning is independent of Kafka partition keys.

Monitoring partition performance¶

Monitor partition lag¶

Partition lag indicates how far behind your consumer is:

SELECT
    datasource_id,
    topic,
    partition,
    max(lag) as current_lag,
    avg(lag) as avg_lag
FROM tinybird.kafka_ops_log
WHERE timestamp > now() - INTERVAL 1 hour
  AND partition >= 0
GROUP BY datasource_id, topic, partition
ORDER BY current_lag DESC

Detect hot partitions¶

Check if messages are evenly distributed across partitions:

SELECT
    __partition,
    count(*) as message_count,
    count(*) * 100.0 / sum(count(*)) OVER () as percentage
FROM your_datasource
WHERE timestamp > now() - INTERVAL 1 hour
GROUP BY __partition
ORDER BY message_count DESC

Uneven distribution (for example, one partition has >50% of messages) indicates a hot partition.

Monitor partition assignment¶

Track how partitions are assigned to consumers:

SELECT
    datasource_id,
    topic,
    partition,
    count(*) as assignment_count
FROM tinybird.kafka_ops_log
WHERE timestamp > now() - INTERVAL 1 hour
  AND partition >= 0
GROUP BY datasource_id, topic, partition
ORDER BY assignment_count DESC

How it works in practice¶

1. Producer sends messages to Kafka topics with optional partition keys
2. Kafka assigns messages to partitions based on the key (or round-robin if no key)
3. Tinybird connector consumes from all partitions in parallel
4. Data is ingested into ClickHouse®, where it's organized into ClickHouse® partitions (typically by time)
5. Multiple Kafka partitions can write to the same ClickHouse® partition
6. Each Kafka partition may create multiple ClickHouse® parts within a partition

Example flow:

Kafka Topic: events (6 partitions)
  ├─ Partition 0 → ClickHouse® partition 202401 (multiple parts)
  ├─ Partition 1 → ClickHouse® partition 202401 (multiple parts)
  ├─ Partition 2 → ClickHouse® partition 202402 (multiple parts)
  └─ ...

Rebalancing:

When partitions are added, removed, or consumer instances change, Kafka rebalances partition assignments. This causes a brief pause in processing. Monitor for rebalancing by tracking partition assignment changes in kafka_ops_log.

Troubleshooting¶

Issue: Uneven partition lag¶

Symptoms: Some partitions have high lag while others are idle

Solutions:

1. Review partition key strategy
2. Check message distribution across partitions
3. Consider increasing partitions if needed
4. Adjust partition key to distribute more evenly

Issue: Hot partitions¶

Symptoms: One partition receives most messages, causing high lag

Solutions:

1. Change partition key strategy
2. Use hash-based keys for even distribution
3. Split high-volume keys across multiple partitions

Issue: TOO_MANY_PARTS error¶

Symptoms: ClickHouse® error indicating too many parts in a partition

Solutions:

1. Reduce number of Kafka partitions writing to the same ClickHouse® partition
2. Adjust ClickHouse® merge settings if needed
3. Consider increasing ClickHouse® partition granularity (for example, daily instead of monthly)
4. Note: Kafka partition key strategy doesn't directly affect this - ClickHouse® partitioning by time is usually fine

Best practices¶

1. Start with 3-6 Kafka partitions and scale as needed
2. Use Kafka partition keys that distribute messages evenly across Kafka partitions
3. For ClickHouse®, partition by time (for example, monthly) - this is usually fine for time-series data
4. Monitor partition lag regularly
5. Avoid hot Kafka partitions by using hash-based keys
6. Watch for TOO_MANY_PARTS errors and adjust ClickHouse® partition granularity if needed
7. Remember: Kafka partition keys (for load distribution) and ClickHouse® partitioning (for data organization) serve different purposes

Related documentation¶

• Performance optimization guide - Overall performance tuning
• Monitor Kafka connectors - Monitoring partition performance
• Troubleshooting guide - Resolve partition issues