Versioning your pipes

Intermediate

At Tinybird you can analyze and transform your data through Pipes in three ways:

  • Using the UI. This is the preferred option when you want to do some data exploration and solution discovery, measuring performance and getting results in an interactive way

  • Using the CLI. This is the preferred option when you are collaborating on a data project or working in production or development environments connected to a source control repository.

  • Using the API. Both the UI and the CLI use the Pipes API behind the scenes, although you will rarely use it directly unless you have a use case where you need to automate the publication of certain API endpoints.

In this section we’ll focus on how to version your pipes using the CLI.

Versions and prefixes

All the Data Sources and Pipes in your account have the following naming convention:

{prefix}__{datasource|pipe}__{version}

Use prefixes to:

  • Maintain different environments for your Data Sources and Pipes (staging, development, production)

  • Work on feature branches or test different ways to improve any of your Pipes.

Use version numbers when:

  • You change the schema of a Data Source

  • You change the output of an endpoint

  • You change the parameters of an endpoint

  • In general, any change on your Data Sources or Pipes that is not backwards compatible.

Guide preparation

Let’s start by pushing the ecommerce_data_project (we’ll use the --prefix pro to indicate it’s used in production):

How to use prefixes

In a regular software development workflow, when fixing a bug or changing an existing API, you’ll do something like this:

  • Someone assigns you an issue with the work to be done

  • You create a branch from your master branch in your Git repository

  • You change the source code as needed (and optionally write some tests), do some commits and create a pull request

  • Someone reviews the code, does some acceptance tests and finally the commits are merged to the master branch

  • The code is deployed to production.

You use prefixes to follow a similar development workflow with your data project. Let’s see a very simple example, imagine you are asked to change the default filter for the top_products.pipe in the ecommerce_data_project.

This is how the top_products endpoint looks in the repository master branch:

Now you create a branch to work on the changes:

Do the changes and commit to the repository, in this case we are changing the interval to 30 days:

Push the changes to your Tinybird account so that someone can do acceptance tests:

Note that the endpoint and all it’s dependencies will be pushed as change_interval__{datasource|pipe}. That way you have an isolated version of the Pipes and Data Sources to validate the feature you are working on.

For new Data Sources created you’ll need to append some data, unless you have some samples in the datasources/fixtures folder of your data project.

Once the feature is reviewed and approved, you’ll merge the changes to the master branch and since this change is backwards compatible with your previous version of the endpoint, you can force push it like this:

Finally drop everything related to the change_interval prefix in your account to have a clean workspace:

How to use versions

Now let’s imagine you are asked to change the output of the top_products endpoint, so instead of returning an array with the top 10 products grouped by day, the team that is integrating the API decides it’s better to have the arrays unnested so you’ll have a row for each date and product.

The way to go in this case would be as follows:

First, you work as usual on a branch and follow the How to use prefixes section.

In this case, you don’t want to force push the endpoint because it’ll break the application, dashboard or any other tool that integrates it, instead you want to increase the version number as follows:

Once the endpoint is validated, you can take a look at what is going to be pushed to your account with this command:

You can see it’ll use the last available version of the Data Source top_product_per_day (in this case with no version) but it’ll increase the version suffix of the top_products endpoint to __v1.

Finally push it:

Once you do this, you’ll have two versions of the top_products endpoint: top_products__v0 and top_products__v1, so now the team in charge of integrating the API can decide when they want to switch to the new version.