---
title: Kafka connector CI/CD and version control
meta:
  description: Manage Kafka connector configurations in CI/CD pipelines. Learn how to use secrets for environment-specific settings, deploy with GitHub Actions and GitLab CI, and implement version control best practices.
---

# CI/CD and version control

This guide covers managing Kafka connector configurations across different environments (local, staging, production) using secrets.

## Managing secrets across environments

### Local development

Use default values in `tb_secret()` for local development:

```tb
KAFKA_BOOTSTRAP_SERVERS {{ tb_secret("KAFKA_BOOTSTRAP_SERVERS", "kafka:29092") }}
KAFKA_SECURITY_PROTOCOL {{ tb_secret("KAFKA_SECURITY_PROTOCOL", "PLAINTEXT") }}
KAFKA_KEY {{ tb_secret("KAFKA_KEY", "key") }}
KAFKA_SECRET {{ tb_secret("KAFKA_SECRET", "secret") }}
```

- **Local**: Uses the default values (for example, `kafka:29092` for local Docker Kafka)
- **Cloud**: Uses the secret values set in each Tinybird workspace

### Staging and production

Set secrets in each workspace using the `--token` flag:

```bash
# Staging workspace
tb --cloud --host <STAGING_HOST> --token <STAGING_TOKEN> secret set KAFKA_BOOTSTRAP_SERVERS "staging-kafka:9092"
tb --cloud --host <STAGING_HOST> --token <STAGING_TOKEN> secret set KAFKA_KEY "staging-key"
tb --cloud --host <STAGING_HOST> --token <STAGING_TOKEN> secret set KAFKA_SECRET "staging-secret"

# Production workspace
tb --cloud --host <PROD_HOST> --token <PROD_TOKEN> secret set KAFKA_BOOTSTRAP_SERVERS "prod-kafka:9092"
tb --cloud --host <PROD_HOST> --token <PROD_TOKEN> secret set KAFKA_KEY "prod-key"
tb --cloud --host <PROD_HOST> --token <PROD_TOKEN> secret set KAFKA_SECRET "prod-secret"
```

The same Connection and Data Source files work across all environments - secrets handle the differences.

## CI/CD integration

### GitHub Actions example

```yaml
name: Deploy to Tinybird

on:
  push:
    branches: [main]

env:
  TINYBIRD_HOST: ${{ secrets.TINYBIRD_HOST }}
  TINYBIRD_TOKEN: ${{ secrets.TINYBIRD_TOKEN }}

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Setup Tinybird CLI
        run: |
          curl https://tinybird.co | sh
      
      - name: Test connection
        run: |
          tb --cloud --host ${{ env.TINYBIRD_HOST }} --token ${{ env.TINYBIRD_TOKEN }} connection data <connection_name>
      
      - name: Deploy
        run: |
          tb --cloud --host ${{ env.TINYBIRD_HOST }} --token ${{ env.TINYBIRD_TOKEN }} deploy
```

{% callout type="info" %}
**Secrets setup**: Set secrets in your Tinybird workspace before running the pipeline. Secrets are configured once per workspace, not on every deployment. See [Managing secrets across environments](#managing-secrets-across-environments) for instructions.
{% /callout %}

### GitLab CI example

```yaml
deploy:
  image: ubuntu:latest
  before_script:
    - apt update && apt install -y curl
    - curl https://tinybird.co | sh
    - export PATH="$HOME/.local/bin:$PATH"
  script:
    - tb --cloud --host $TINYBIRD_HOST --token $TINYBIRD_TOKEN connection data <connection_name>
    - tb --cloud --host $TINYBIRD_HOST --token $TINYBIRD_TOKEN deploy
  only:
    - main
```

## Consumer group ID management

Always use different consumer group IDs for each environment to avoid conflicts:

```tb
KAFKA_GROUP_ID {{ tb_secret("KAFKA_GROUP_ID", "dev-events-group") }}
```

Set different group IDs in each workspace:
- Local: Uses default `"dev-events-group"`
- Staging: Set `tb --cloud --host <STAGING_HOST> --token <STAGING_TOKEN> secret set KAFKA_GROUP_ID "staging-events-group"`
- Production: Set `tb --cloud --host <PROD_HOST> --token <PROD_TOKEN> secret set KAFKA_GROUP_ID "prod-events-group"`

## Version control best practices

### What to commit

**Commit:**
- Connection file structure (with `tb_secret()` references, not actual secret values)
- Data Source schemas
- Pipe definitions

**Don't commit:**
- Secret values
- API keys
- Passwords
- Production credentials

## Related documentation

- [Kafka connector documentation](/forward/get-data-in/connectors/kafka) - Main setup and configuration guide
- [Test and deploy](/forward/test-and-deploy) - General deployment guide
- [Troubleshooting guide](../troubleshooting) - Resolving deployment issues