Working with version control¶

With Tinybird, you can take your existing version control knowledge from software engineering, and apply it immediately to your real-time data products, using Git.

How the Git integration works¶

Tinybird's Git integration creates a bi-directional link between your Tinybird Workspace and a remote Git repository. This means you can work on Tinybird resources locally, push to Git and then sync to Tinybird, or work in the Tinybird UI and push your changes to Git. In this way, Git becomes the ultimate source of truth for your project.

You can then follow standard development patterns, such as working in feature branches, using pull requests, and running CI/CD pipelines to validate changes and deploy to production. You can push a specific Git commit to production as a semantically-versioned Release.

When you connect your Workspace to a Git repository, Tinybird will create a CI/CD pipeline for you. The CI/CD pipelines need to be executed outside of Tinybird in a runner such as GitHub Action or GitLab Runner.

You can connect your Workspace to Git using either the UI or the CLI.

Project structure¶

A Tinybird project is represented by a collection of text files (called Datafiles) that are organised in folders.

You can initialize a new Tinybird project with automatic scaffolding using the tb init CLI command. This creates the following files and folders:

- /datasources
- /datasources/fixtures
- /endpoints
- /pipes
- /tests
- /scripts
- /scripts/exec_test.sh
- /deploy
.tinyenv

If you have an existing project created in the UI, use the tb pull --auto --force command to download all resources from your Workspace, creating the same structure as above.

The purpose of these files and folder is as follows:

• datasources: Where you put your .datasource files.
• datasources/fixtures: Place as many CSV or NDJSON files that will be pushed when using the default ./scripts/append_fixtures.sh script. They need to share name with a .datasource file.
• endpoints: You can use this folder to create a logical separation between non-Endpoint Pipes and Endpoint Pipes, though it is not necessary. By default, all .pipe files will be placed in the pipes/ directory when pulled from Tinybird.
• pipes: Where you put your .pipe files.
• tests: Where you put data quality and fixture tests.
• scripts: Useful scripts for common operations like data migrations, fixtures tests, etc.
• deploy: Custom deployment shell scripts.
• .tinyenv: Global variables for the project, for instance the VERSION.

Connect your Workspace to Git from the UI¶

To connect your Workspace to Git, you will need a Tinybird Workspace and a Git repository. You can either connect a pre-existing repository, or create a new one as part of this process.

1. Start the connection¶

Open the Tinybird UI and go to your Workspace dashboard. Click the "Connect to Git" button in the top left corner.

You will see a modal with information about the Git integration. Click the "Connect your Repository" button.

2. Choose your Git provider¶

Select the Git provider you want to use. Choose a Git provider and click the "Connect" button.

You will now be directed to the Git provider's authorisation page and prompted to allow Tinybird to access your Git account. Click the "Authorize Tinybird" button.

When successfully authorized, you will be redirected back to the Tinybird UI to complete the setup.

3. Select a repository¶

You will now be able to select your Git repository from the dropdown menu. You can also choose to customize the folder within the repository where you will place your Tinybird resources. We recommend keeping your Tinybird resources in a /tinybird folder.

When you have selected your repository, click the "Connect" button.

Your Workspace and Git repository are now connected.

4. Initialize the repository¶

You can now choose how to push your existing Tinybird resources to your Git repository. The UI can help you do this, giving you the options to push directly to the main branch, or creating a pull request. If you'd rather do this manually, you can skip this step and use the CLI later.

When you push from the UI, this will automatically commit the default CI/CD workflow scripts to your repository.

Connect your Workspace to Git from the CLI¶

To connect your Workspace to Git, you will need a Tinybird Workspace and a Git repository. You can either connect a pre-existing repository, or create a new one as part of this process.

To initialize your Workspace with Git, run the tb init --git command. It performs the following actions:

• Checks there are no differences between your local files and Tinybird Workspace.
• Saves a reference to the current Git repository commit in the Workspace. This commit reference is used later on to diff Workspace resources and resources in a branch, to ease deployment.
• Lets you opt-in to create CI/CD pipelines for GitHub or GitLab. If you opt-in, you'll have to create either a secret or environment variable (depending on your Git provider) called TB_ADMIN_TOKEN with your Workspace admin Auth Token.

$ tb init --git

    ** - /datasources already exists, skipping
    ** - /datasources/fixtures already exists, skipping
    ** - /endpoints already exists, skipping
    ** - /pipes already exists, skipping
    ** - /tests already exists, skipping
    ** - /scripts already exists, skipping
    ** - /deploy already exists, skipping
    ** - '.tinyenv' already exists, skipping
    ** - '.tinyb' already in .gitignore, skipping

    ** Initializing releases based on git for Workspace 'workspace'
    ** Checking diffs between remote Workspace and local. Hint: use 'tb diff' to check if your data project and Workspace synced
    Pulling datasources  [####################################]  100%
    Pulling pipes  [####################################]  100%
    Pulling tokens  [####################################]  100%
    ** No diffs detected for 'workspace'

    Do you want to generate CI/CD config files? [Y/n]: y
    ** List of available providers:
    [1] GitHub
    [2] GitLab
    [0] Cancel

    Use provider [1]: 1
    ** File .github/workflows/tinybird_ci.yml generated for CI/CD
    ** File .github/workflows/tinybird_cd.yml generated for CI/CD
    ** Warning: Set TB_ADMIN_TOKEN in GitHub secrets. Use the Workspace admin token. Hint: use `tb token copy b9b0f7cc-1f7c-4765-b73f-d77b2de74469` to copy clipboard
    ** GitHub CI/CD config files generated. Read this guide to learn how to run CI/CD pipelines: https://www.tinybird.co/docs/version-control/continuous-integration.html
    ** Workspace 'workspace' release initialized to commit '019a9718b67a914bda8c2ac59a9368adf5d95b0b.'
    Now start working with git, pushing changes to pull requests and let the CI/CD work for you. More details in this guide: https://www.tinybird.co/docs/version-control/working-with-version-control.html.

Once complete, push the Tinybird CI/CD actions to your Git provider. These actions are configurable .yml actions based on your development pipeline, and the templates Tinybird provides offer an excellent basis upon which to validate changes and deploy to Tinybird safely from Git. Those templates are being actively developed so we recommend you to use them directly or base yours on them, so upgrading to newer versions is as straightforward as possible.

    $ echo ".tinyb" >> .gitignore
    $ git add .
    $ git commit -m "Add Tinybird CI/CD actions"
    $ git push

You must save your Workspace admin Auth Token as a secret in your repository. For example, with GitHub, go to your repository settings, and under Secrets and Variables / Actions, add the Auth Token value in a secret called TB_ADMIN_TOKEN.

Protecting Production¶

Once you decide to use this version-controlled workflow, the Git repository becomes your single source of truth. You'll want to keep your Releases protected so users can't modify resources and break the Git workflow. Releases are protected by default: no users (including admins) can change any resources directly from the UI in a Release without the CI/CD processes to avoid divergence. The Live Release only permits data operations, and Preview/Rollback Releases only permit data operations in changed resources.

If you want to also prevent users from making changes to a Release from the CLI or API, open the Tinybird UI and navigate to the Workspace settings menu, then the Members tab, and assign the Viewer role. Users with a Viewer role aren't able to create, edit, or delete resources or run data operations. They are allowed to create a new Branch and change resources there.

Development workflow¶

This section explains how to safely develop new features using Branches.

Develop using the UI¶

Once your Workspace is connected to Git, you can take advantage of the Git-based workflow from the UI. This includes creating Git branches, committing changes, and creating Pull Requests to your repository.

Let's create a new branch from the UI. Select the Create Branch button on the Overview page or from the dropdown at the top:

Create a new branch called add_new_endpoint with partial data so we can validate:

Once the branch is created, the UI updates to show the resources of the Branch instead of the Live Release. Now, we can start iterating and prototyping our new Endpoint.

In this case, we've created a new Endpoint that will return the unique visitors per day. On a Branch, you can create new resources and edit existing ones. Any modified resources are marked with a blue square, so you can easily identify them.

Once you are happy with the prototype, create a commit to your Git branch by selecting the Commit changes button at the bottom left of the screen.

You are now ready to create a Pull Request to your Git repository by clicking the Create Pull Request button at the bottom left of the screen.

As part of creating the Pull Request, you need to increment the version of your project. In this case, as we are not changing any existing resource, we can just increase the patch version, and create the Pull Request.

You can read more about the meaning of the version number in the deployment strategies docs.

Once the Pull Request is created, the CI/CD pipeline is triggered. You can read more about the CI/CD pipeline in the CI/CD docs.

Develop using the CLI¶

When wanting to prototype a new endpoint or a change in its logic, we recommend you use the UI. It is the easiest and fastest way to iterate and validate your changes, and you can see the results of your changes in real time.

But when needing to make changes like data migrations or changes in column types, you need to use the CLI and modify the datafiles. Make sure you're familiar with the Tinybird CLI docs.

For changes like these, use the Git workflow: Create a new branch, make the changes in the datafiles, and create a Pull Request to validate the changes. For further guidance, read the CI/CD docs and deployment strategies docs.

This image visualizes each process step when setting up and working with Git via the Tinybird CLI:

Exploration workflow¶

The Playground¶

Once you have protected your Live Release from user modifications, you will not be able to create Pipes directly in the Live Release.

To explore the data, you need to either create a new Branch, or use the Playground.

Once you've prototyped your new Pipe, download the Pipe from the UI and commit the change to a new Git branch to follow the CI worfklow.

By default, Playground content is private to your Workspace view. However, you have the option to share your Playground with other Workspace members.

Troubleshooting¶

This section covers some of the problems you might face when working with version control.

Connect to more than one Workspace¶

You can have one Git repo connected to 2 Tinybird Workspaces - a common scenario is using this for staging and production Workspaces. Just use a different ADMIN_TOKEN in the GitHub Actions.

Initialization¶

When your Git repository and Tinybird Workspace have the same resources, tb init --git allows you to easily start up a CI/CD pipeline. However, when these are not in sync, you might experience one of these three scenarios:

Problem: There are resources in the Workspace that are not present in your Git repository.

Remove them from your Workspace (they are probably unnecessary) or run tb pull --match "<resource_name>" --auto to download the resource(s) locally and push them to the Git repository before continuing.

You can also bypass remote resources by running the init command, like this: tb init --git --ignore-remote

Problem: There are resources in your Git repository that are not present in your Workspace.

Either remove them from the project or run tb push them to the Workspace and re-run the init command.

Problem: There are differences between the resources in the Git repository and the ones in your Workspace.

In this instance, you must decide which version is the source of truth. If it is the resources in the Workspace, run tb pull --auto --force to overwrite your local files. If it is your local files, run tb push --force file by file so the Workspace is synchronized with your project in Git.

Git and Workspace no longer synced¶

When you deploy, the Git commit ID of that deployment is stored in the Release. This means you can use the git diff command to compare your Branch against the main one, and know which resources have been modified and need to be deployed.

If you introduce a manual change in your Workspace or Git outside of the CI/CD workflow, they will no longer be synced. To get both components to once again be in sync with each other, use the command tb init --override-commit <GIT_COMMIT_ID>. This command will override the Git commit ID from the Live Release.

The wrong Git repo is connected¶

First, delete the Preview Release and any Branches in your Tinybird Workspace. Then, In the Tinybird UI, select the name of the connected Git repo (right by the GitHub/GitLab icon, top left in the Tinybird UI). In the modal, select "Disconnect" and contact us at support@tinybird.co. We'll re-set everything behind the scenes and you'll be able to re-connect in the UI using the "Connect to Git" button.

Status is stuck on "Syncing"¶

Check your CI/CD processes (for example, GitHub Actions). When they pass successfully, your Tinybird version control status changes to "Synced".

Internal Server Error when connecting to Git¶

This is related to a GitHub API rate limit that affects Tinybird. Wait a few minutes and try again, or as a workaround connect the Workspace to Git using the CLI, not the UI (follow the docs here). If the problem continues, let us know (support@tinybird.co).

Common use cases¶

Version control allows you to incrementally change or iterate your data project. It's ideal managing changes like adding a column, changing data types, redefining whole views, and lots more.

If you're new to using version control or want to be sure how to do it on Tinybird, there's an entire repository of use cases available: tinybirdco/use-case-examples.

Next steps¶

• Read about Datafiles, the text files that describe your Tinybird resources (Data Sources, Pipes).
• Understand CI/CD processes on Tinybird.