---
title: Confluent Cloud setup guide
meta:
  description: Complete guide to setting up Tinybird's Kafka connector with Confluent Cloud, including authentication, Schema Registry, and Private Link configuration.
---

# Confluent Cloud setup guide

This guide walks you through setting up Tinybird's Kafka connector with Confluent Cloud, including authentication, network configuration, and Schema Registry integration.

## Prerequisites

- A Confluent Cloud account with an active cluster
- API keys for your Confluent Cloud cluster
- Access to your Confluent Cloud console
- A Tinybird workspace

## Step 1: Get your Confluent Cloud connection details

### Bootstrap servers

1. Log in to the [Confluent Cloud console](https://confluent.cloud)
2. Navigate to your cluster
3. Go to **Cluster settings** > **Bootstrap server**
4. Copy the bootstrap server address (for example, `pkc-xxxxx.us-east-1.aws.confluent.cloud:9092`)

### API keys

1. In Confluent Cloud, go to **API Keys**
2. Select **Add key** and select **Global access** or **Resource-specific**
3. Copy the **API Key** and **API Secret**
4. Save these securely - you need them for the Tinybird connection

## Step 2: Create the Kafka connection

You can create the Kafka connection using the CLI wizard or by manually creating a connection file.

### Option 1: Use the CLI wizard (recommended)

Run the following command to create a connection interactively:

```bash
tb connection create kafka
```

The wizard prompts you for:
1. Connection name
2. Bootstrap server
3. Kafka key (API Key)
4. Kafka secret (API Secret)

After the wizard completes, you can edit the generated `.connection` file to add additional settings like `KAFKA_SCHEMA_REGISTRY_URL` if needed.

### Option 2: Manually create a connection file

Create a `.connection` file in your Tinybird project:

```tb {% title="connections/confluent_cloud.connection" %}
TYPE kafka
KAFKA_BOOTSTRAP_SERVERS <YOUR_BOOTSTRAP_SERVER>
KAFKA_SECURITY_PROTOCOL SASL_SSL
KAFKA_SASL_MECHANISM PLAIN
KAFKA_KEY {{ tb_secret("CONFLUENT_API_KEY") }}
KAFKA_SECRET {{ tb_secret("CONFLUENT_API_SECRET") }}
```

Set the secrets:

```bash
tb [--cloud] secret set CONFLUENT_API_KEY <YOUR_API_KEY>
tb [--cloud] secret set CONFLUENT_API_SECRET <YOUR_API_SECRET>
```

## Step 3: Configure Schema Registry (optional)

If you're using Avro or JSON Schema with Schema Registry:

1. In Confluent Cloud, go to **Schema Registry**
2. Get the Schema Registry endpoint URL
3. Create API keys for Schema Registry (separate from cluster API keys)
4. Add to your connection file:

```tb {% title="connections/confluent_cloud_with_registry.connection" %}
TYPE kafka
KAFKA_BOOTSTRAP_SERVERS <YOUR_BOOTSTRAP_SERVER>
KAFKA_SECURITY_PROTOCOL SASL_SSL
KAFKA_SASL_MECHANISM PLAIN
KAFKA_KEY {{ tb_secret("CONFLUENT_API_KEY") }}
KAFKA_SECRET {{ tb_secret("CONFLUENT_API_SECRET") }}
KAFKA_SCHEMA_REGISTRY_URL https://<REGISTRY_API_KEY>:<REGISTRY_API_SECRET>@<REGISTRY_ENDPOINT>
```

Set the Schema Registry credentials:

```bash
tb [--cloud] secret set CONFLUENT_REGISTRY_API_KEY <YOUR_REGISTRY_API_KEY>
tb [--cloud] secret set CONFLUENT_REGISTRY_API_SECRET <YOUR_REGISTRY_API_SECRET>
```

Then use in the connection:

```tb
KAFKA_SCHEMA_REGISTRY_URL https://{{ tb_secret("CONFLUENT_REGISTRY_API_KEY") }}:{{ tb_secret("CONFLUENT_REGISTRY_API_SECRET") }}@<REGISTRY_ENDPOINT>
```

## Step 4: Network configuration

### Standard setup

For most Confluent Cloud clusters, no additional network configuration is needed. Tinybird connects directly to Confluent Cloud's public endpoints.

### Private Link (Enterprise only)

If you're using Confluent Cloud Private Link:

1. Ensure your Confluent Cloud cluster has Private Link turned on
2. Contact Tinybird support to set up Private Link connectivity
3. Use the Private Link endpoint as your bootstrap server

For Private Link setup, contact <support@tinybird.co> with:
- Your Confluent Cloud cluster details
- Private Link endpoint information
- Your Tinybird organization name

## Step 5: Create the Kafka Data Source

Now that your connection is configured, create a Kafka Data Source. See [Create a Kafka data source](/forward/get-data-in/connectors/kafka#create-a-kafka-data-source) in the main Kafka connector guide for detailed instructions on:

- Using `tb datasource create --kafka` for a guided setup
- Manually creating `.datasource` files
- Defining schemas with JSONPath expressions
- Configuring Kafka-specific settings

## Step 6: Test the connection

Test your connection and preview data:

```bash
tb connection data confluent_cloud
```

This command prompts you to select a topic and consumer group ID, then returns preview data. This verifies that Tinybird can connect to your Confluent Cloud cluster, authenticate, and consume messages.

## Common Confluent Cloud issues

### Issue: Authentication failed

**Symptoms:**
- "Authentication failed" errors in `kafka_ops_log`
- Connection check fails

**Solutions:**
1. Verify API keys are correct and active in Confluent Cloud
2. Check API key has the correct permissions (read access to topics)
3. Ensure you're using `PLAIN` as the SASL mechanism
4. Verify the API key hasn't been deleted or rotated

### Issue: Schema Registry connection failed

**Symptoms:**
- "Schema Registry connection failed" errors
- Avro messages not being ingested

**Solutions:**
1. Verify Schema Registry endpoint URL is correct
2. Check Schema Registry API keys are separate from cluster API keys
3. Ensure Schema Registry API keys have read permissions
4. Verify the schema exists in Schema Registry for your topic

### Issue: Network connectivity

**Symptoms:**
- Connection timeout errors
- Broker unreachable

**Solutions:**
1. Verify bootstrap server address is correct
2. Check if your Confluent Cloud cluster allows public access
3. For Private Link, ensure the connection is properly configured
4. Verify firewall rules allow outbound connections

## Best practices

1. **Use separate API keys for different environments** (dev, staging, prod)
2. **Rotate API keys regularly** for security
3. **Use resource-specific API keys** instead of global access when possible
4. **Monitor API key usage** in Confluent Cloud console
5. **Set up alerts** for authentication failures

## Related documentation

- [Kafka connector documentation](/forward/get-data-in/connectors/kafka) - Main setup and configuration guide
- [Monitor Kafka connectors](/forward/monitoring/kafka-clickhouse-monitoring) - Set up monitoring and alerts
- [Troubleshooting guide](../troubleshooting) - Common issues and solutions