Kafka connector limits and quotas

This page describes the limits and quotas that apply to the Kafka connector in Tinybird. For general workspace limits, see Limits.

How limits are applied

Kafka connector limits are applied at the organization level, meaning all workspaces within your organization share the same limits. This ensures consistent resource allocation across your organization.

Topic limits

The number of Kafka topics you can connect to depends on your plan:

PlanTopic limitNotes
Free5 topicsHard limit
Developer5 topicsHard limit
EnterpriseUnlimited topicsContact support for high topic counts

Requesting a topic limit increase

Enterprise plan users can request an increase to the topic limit by contacting support@tinybird.co. Include the following information in your request:

  • Your organization name
  • Current number of topics in use
  • Expected number of topics needed
  • Use case description

Consumer group limits

There are no hard limits on the number of consumer groups you can use. However, each combination of KAFKA_TOPIC and KAFKA_GROUP_ID can only be used in one Data Source.

Best practices:

  • Use unique consumer group IDs for each Data Source
  • Use environment-specific group IDs (for example, prod-group, staging-group)
  • Document which consumer groups are in use

For managing consumer groups across environments, see the CI/CD and version control guide.

Connection limits

You can create up to 3 Kafka connections per workspace. Each connection can be used by multiple Data Sources.

If you need more connections, contact support@tinybird.co to discuss your use case.

Message size limits

Tinybird has a default message size limit of 10 MB per message. Messages exceeding this limit are automatically sent to the Quarantine Data Source.

Strategies for handling large messages:

  • Use Kafka compression to reduce message size
  • Split large messages into smaller chunks
  • Store large payloads in object storage (S3, GCS) and reference them in Kafka messages

For detailed guidance, see the message size handling guide.

Throughput limits

The Kafka connector has the following throughput limits:

  • Minimum flush time: 4 seconds
  • Throughput (uncompressed): 20 MB/s per connection

If you're regularly hitting these limits, contact support@tinybird.co for support. Enterprise customers may have higher limits based on their infrastructure.

Deployment failures due to limits

If a deployment fails due to limits, you see an error message indicating which limit was exceeded. Common scenarios:

  1. Topic limit exceeded: You've reached the maximum number of topics for your plan
  2. Connection limit exceeded: You've created the maximum number of connections for your workspace
  3. Message size limit: A message exceeds the 10 MB limit (sent to quarantine, not a deployment failure)

To resolve:

  1. Review your current usage against the limits
  2. Remove unused topics or connections if needed
  3. Contact support to request limit increases (Enterprise plans)

Best practices for managing limits

  1. Monitor your usage: Regularly check how many topics and connections you're using
  2. Clean up unused resources: Remove Data Sources and connections that are no longer needed
  3. Plan for growth: Consider your future needs when designing your Kafka integration
  4. Use environment-specific configurations: Separate development, staging, and production to better manage resources

Requesting limit increases

Enterprise plan users can request limit increases by contacting support@tinybird.co. Include:

  • Your organization name
  • Which limit you need increased (topics, connections, throughput)
  • Current usage and expected growth
  • Use case description

Free and Developer plan users can upgrade to Enterprise to access higher limits. See Pricing for plan details.

Updated