Confluent Cloud setup guide

This guide walks you through setting up Tinybird's Kafka connector with Confluent Cloud, including authentication, network configuration, and Schema Registry integration.

Prerequisites

  • A Confluent Cloud account with an active cluster
  • API keys for your Confluent Cloud cluster
  • Access to your Confluent Cloud console
  • A Tinybird workspace

Step 1: Get your Confluent Cloud connection details

Bootstrap servers

  1. Log in to the Confluent Cloud console
  2. Navigate to your cluster
  3. Go to Cluster settings > Bootstrap server
  4. Copy the bootstrap server address (for example, pkc-xxxxx.us-east-1.aws.confluent.cloud:9092)

API keys

  1. In Confluent Cloud, go to API Keys
  2. Select Add key and select Global access or Resource-specific
  3. Copy the API Key and API Secret
  4. Save these securely - you need them for the Tinybird connection

Step 2: Create the Kafka connection

You can create the Kafka connection using the CLI wizard or by manually creating a connection file.

Option 1: Use the CLI wizard (recommended)

Run the following command to create a connection interactively:

tb connection create kafka

The wizard prompts you for:

  1. Connection name
  2. Bootstrap server
  3. Kafka key (API Key)
  4. Kafka secret (API Secret)

After the wizard completes, you can edit the generated .connection file to add additional settings like KAFKA_SCHEMA_REGISTRY_URL if needed.

Option 2: Manually create a connection file

Create a .connection file in your Tinybird project:

connections/confluent_cloud.connection
TYPE kafka
KAFKA_BOOTSTRAP_SERVERS <YOUR_BOOTSTRAP_SERVER>
KAFKA_SECURITY_PROTOCOL SASL_SSL
KAFKA_SASL_MECHANISM PLAIN
KAFKA_KEY {{ tb_secret("CONFLUENT_API_KEY") }}
KAFKA_SECRET {{ tb_secret("CONFLUENT_API_SECRET") }}

Set the secrets:

tb [--cloud] secret set CONFLUENT_API_KEY <YOUR_API_KEY>
tb [--cloud] secret set CONFLUENT_API_SECRET <YOUR_API_SECRET>

Step 3: Configure Schema Registry (optional)

If you're using Avro or JSON Schema with Schema Registry:

  1. In Confluent Cloud, go to Schema Registry
  2. Get the Schema Registry endpoint URL
  3. Create API keys for Schema Registry (separate from cluster API keys)
  4. Add to your connection file:
connections/confluent_cloud_with_registry.connection
TYPE kafka
KAFKA_BOOTSTRAP_SERVERS <YOUR_BOOTSTRAP_SERVER>
KAFKA_SECURITY_PROTOCOL SASL_SSL
KAFKA_SASL_MECHANISM PLAIN
KAFKA_KEY {{ tb_secret("CONFLUENT_API_KEY") }}
KAFKA_SECRET {{ tb_secret("CONFLUENT_API_SECRET") }}
KAFKA_SCHEMA_REGISTRY_URL https://<REGISTRY_API_KEY>:<REGISTRY_API_SECRET>@<REGISTRY_ENDPOINT>

Set the Schema Registry credentials:

tb [--cloud] secret set CONFLUENT_REGISTRY_API_KEY <YOUR_REGISTRY_API_KEY>
tb [--cloud] secret set CONFLUENT_REGISTRY_API_SECRET <YOUR_REGISTRY_API_SECRET>

Then use in the connection:

KAFKA_SCHEMA_REGISTRY_URL https://{{ tb_secret("CONFLUENT_REGISTRY_API_KEY") }}:{{ tb_secret("CONFLUENT_REGISTRY_API_SECRET") }}@<REGISTRY_ENDPOINT>

Step 4: Network configuration

Standard setup

For most Confluent Cloud clusters, no additional network configuration is needed. Tinybird connects directly to Confluent Cloud's public endpoints.

Private Link (Enterprise only)

If you're using Confluent Cloud Private Link:

  1. Ensure your Confluent Cloud cluster has Private Link turned on
  2. Contact Tinybird support to set up Private Link connectivity
  3. Use the Private Link endpoint as your bootstrap server

For Private Link setup, contact support@tinybird.co with:

  • Your Confluent Cloud cluster details
  • Private Link endpoint information
  • Your Tinybird organization name

Step 5: Create the Kafka Data Source

Now that your connection is configured, create a Kafka Data Source. See Create a Kafka data source in the main Kafka connector guide for detailed instructions on:

  • Using tb datasource create --kafka for a guided setup
  • Manually creating .datasource files
  • Defining schemas with JSONPath expressions
  • Configuring Kafka-specific settings

Step 6: Test the connection

Test your connection and preview data:

tb connection data confluent_cloud

This command prompts you to select a topic and consumer group ID, then returns preview data. This verifies that Tinybird can connect to your Confluent Cloud cluster, authenticate, and consume messages.

Common Confluent Cloud issues

Issue: Authentication failed

Symptoms:

  • "Authentication failed" errors in kafka_ops_log
  • Connection check fails

Solutions:

  1. Verify API keys are correct and active in Confluent Cloud
  2. Check API key has the correct permissions (read access to topics)
  3. Ensure you're using PLAIN as the SASL mechanism
  4. Verify the API key hasn't been deleted or rotated

Issue: Schema Registry connection failed

Symptoms:

  • "Schema Registry connection failed" errors
  • Avro messages not being ingested

Solutions:

  1. Verify Schema Registry endpoint URL is correct
  2. Check Schema Registry API keys are separate from cluster API keys
  3. Ensure Schema Registry API keys have read permissions
  4. Verify the schema exists in Schema Registry for your topic

Issue: Network connectivity

Symptoms:

  • Connection timeout errors
  • Broker unreachable

Solutions:

  1. Verify bootstrap server address is correct
  2. Check if your Confluent Cloud cluster allows public access
  3. For Private Link, ensure the connection is properly configured
  4. Verify firewall rules allow outbound connections

Best practices

  1. Use separate API keys for different environments (dev, staging, prod)
  2. Rotate API keys regularly for security
  3. Use resource-specific API keys instead of global access when possible
  4. Monitor API key usage in Confluent Cloud console
  5. Set up alerts for authentication failures
Updated