---
title: Redpanda setup guide
meta:
  description: Guide to setting up Tinybird's Kafka connector with Redpanda, including authentication and network configuration.
---

# Redpanda setup guide

This guide walks you through setting up Tinybird's Kafka connector with Redpanda, a Kafka-compatible streaming data platform.

## Prerequisites

- A Redpanda cluster (self-hosted or Redpanda Cloud)
- Redpanda credentials (if authentication is turned on)
- Network access to your Redpanda cluster
- A Tinybird workspace

## Redpanda compatibility

Redpanda is fully compatible with the Apache Kafka protocol, so you can use Tinybird's Kafka connector with Redpanda without any special configuration. The connector works with Redpanda the same way it works with Apache Kafka.

## Step 1: Get your Redpanda connection details

### Bootstrap servers

1. For **Redpanda Cloud**: Get the bootstrap server from your Redpanda Cloud console
2. For **self-hosted Redpanda**: Use your Redpanda broker addresses

The bootstrap server format is: `<host>:<port>`

Common ports:
- `9092` for PLAINTEXT
- `9093` for SASL_SSL
- `9094` for TLS

### Authentication

Redpanda supports multiple authentication methods:
- **SASL/PLAIN** - Username and password
- **SASL/SCRAM-SHA-256** or **SCRAM-SHA-512** - SCRAM authentication
- **TLS/mTLS** - Certificate-based authentication
- **PLAINTEXT** - No authentication (local development only)

## Step 2: Create the Kafka connection

You can create the Kafka connection using the CLI wizard or by manually creating a connection file.

### Option 1: Use the CLI wizard (recommended)

Run the following command to create a connection interactively:

```bash
tb connection create kafka
```

The wizard prompts you for:
1. Connection name
2. Bootstrap server
3. Kafka key (username for SASL/PLAIN or SCRAM)
4. Kafka secret (password for SASL/PLAIN or SCRAM)

After the wizard completes, you can edit the generated `.connection` file to add additional settings like `KAFKA_SASL_MECHANISM` if needed (defaults to PLAIN).

### Option 2: Manually create a connection file

#### SASL/PLAIN authentication

```tb {% title="connections/redpanda.connection" %}
TYPE kafka
KAFKA_BOOTSTRAP_SERVERS <REDPANDA_BOOTSTRAP_SERVER>
KAFKA_SECURITY_PROTOCOL SASL_SSL
KAFKA_SASL_MECHANISM PLAIN
KAFKA_KEY {{ tb_secret("REDPANDA_USERNAME") }}
KAFKA_SECRET {{ tb_secret("REDPANDA_PASSWORD") }}
```

Set the secrets:

```bash
tb [--cloud] secret set REDPANDA_USERNAME <YOUR_USERNAME>
tb [--cloud] secret set REDPANDA_PASSWORD <YOUR_PASSWORD>
```

### SASL/SCRAM authentication

```tb {% title="connections/redpanda_scram.connection" %}
TYPE kafka
KAFKA_BOOTSTRAP_SERVERS <REDPANDA_BOOTSTRAP_SERVER>
KAFKA_SECURITY_PROTOCOL SASL_SSL
KAFKA_SASL_MECHANISM SCRAM-SHA-256
KAFKA_KEY {{ tb_secret("REDPANDA_USERNAME") }}
KAFKA_SECRET {{ tb_secret("REDPANDA_PASSWORD") }}
```

### PLAINTEXT (local development only)

```tb {% title="connections/redpanda_local.connection" %}
TYPE kafka
KAFKA_BOOTSTRAP_SERVERS localhost:9092
KAFKA_SECURITY_PROTOCOL PLAINTEXT
```

{% callout type="warning" %}
PLAINTEXT connections should only be used for local development. Production Redpanda clusters should use SASL_SSL.
{% /callout %}

## Step 3: SSL/TLS certificate (if required)

If your Redpanda cluster uses self-signed certificates or a private CA, provide the CA certificate:

```bash
tb [--cloud] secret set --multiline REDPANDA_SSL_CA_PEM
```

Add to your connection file:

```tb
KAFKA_SSL_CA_PEM >
  {{ tb_secret("REDPANDA_SSL_CA_PEM") }}
```

## Step 4: Create the Kafka Data Source

Now that your connection is configured, create a Kafka Data Source. See [Create a Kafka data source](/forward/get-data-in/connectors/kafka#create-a-kafka-data-source) in the main Kafka connector guide for detailed instructions on:

- Using `tb datasource create --kafka` for a guided setup
- Manually creating `.datasource` files
- Defining schemas with JSONPath expressions
- Configuring Kafka-specific settings

## Step 5: Test the connection

Test your connection and preview data:

```bash
tb connection data redpanda
```

This command prompts you to select a topic and consumer group ID, then returns preview data. This verifies that Tinybird can connect to your Redpanda cluster, authenticate, and consume messages.

## Redpanda Cloud specific setup

If you're using Redpanda Cloud:

1. **Get connection details** from the Redpanda Cloud console
2. **Use SASL/PLAIN** authentication with your Redpanda Cloud credentials
3. **Use the provided bootstrap server** address
4. **No SSL certificate needed** - Redpanda Cloud uses public CA certificates

## Common Redpanda issues

### Issue: Authentication failed

**Solutions:**
1. Verify username and password are correct
2. Check SASL mechanism matches your Redpanda configuration
3. Ensure the user has read permissions on the topic
4. Verify security protocol matches (SASL_SSL vs PLAINTEXT)

### Issue: Connection timeout

**Solutions:**
1. Verify bootstrap server address is correct
2. Check network connectivity to Redpanda cluster
3. Verify firewall rules allow outbound connections
4. For self-hosted Redpanda, ensure the cluster is accessible

### Issue: SSL certificate validation

**Solutions:**
1. Provide CA certificate if using self-signed certificates
2. Verify certificate is in PEM format
3. Check certificate hasn't expired

## Redpanda vs Kafka differences

While Redpanda is Kafka-compatible, there are some differences to be aware of:

1. **Performance**: Redpanda is optimized for lower latency and higher throughput
2. **Resource usage**: Redpanda typically uses less CPU and memory
3. **Configuration**: Some Kafka-specific configurations may not apply
4. **Schema Registry**: Redpanda doesn't include Schema Registry by default (use Confluent Schema Registry separately if needed)

## Best practices

1. **Use SASL_SSL** for production Redpanda clusters
2. **Monitor consumer lag** using `kafka_ops_log`
3. **Use unique consumer group IDs** for each Data Source
4. **Set up alerts** for high lag or errors
5. **Test connections** before deploying to production

## Related documentation

- [Kafka connector documentation](/forward/get-data-in/connectors/kafka) - Main setup and configuration guide
- [Monitor Kafka connectors](/forward/monitoring/kafka-clickhouse-monitoring) - Set up monitoring and alerts
- [Troubleshooting guide](../troubleshooting) - Common issues and solutions