Kafka connector guides

This section contains detailed guides for setting up, configuring, and optimizing your Kafka connector. Each guide focuses on a specific aspect of working with Kafka in Tinybird.

Setup guides

Vendor-specific setup

  • Confluent Cloud setup - Connect Tinybird to your Confluent Cloud cluster with step-by-step instructions for authentication, SSL configuration, and testing.

  • AWS MSK setup - Integrate Tinybird with Amazon MSK, including IAM authentication, security group configuration, and connection testing.

  • Redpanda setup - Connect to your Redpanda cluster, configure authentication, and set up your first Data Source.

  • Local development - Set up a local Kafka environment using Docker Compose for development and testing.

Configuration and management

  • CI/CD and version control - Manage Kafka connector configurations in CI/CD pipelines, handle environment-specific settings, and implement best practices for version control.

  • Schema management - Understand schema evolution, handle Avro and JSON Schema formats, and implement schema versioning strategies.

Performance and optimization

  • Partitioning strategies - Learn best practices for Kafka topic partitioning, understand partition assignment, and optimize for performance.

  • Message size handling - Handle large Kafka messages, implement compression strategies, and work within message size limits.

  • Performance optimization - Optimize throughput, reduce consumer lag, scale consumers effectively, and tune your Kafka connector for maximum performance.

Updated