Pipes

What is a Pipe?

A Pipe contains one or more SQL queries that result in either an API endpoint or a Materialised View.

What should I use Pipes for?

Use Pipes to build features over your data.

Write SQL that joins, aggregates or otherwise transforms your data and publish the result.

You have two options to publish the result of a Pipe: API Endpoints, or Materialised Views.

Publishing as an API Endpoint gives you an HTTP GET API with the result of the Pipe in JSON, NDJSON, CSV or Parquet formats. You can use this API directly in your applications to access your data.

Materialised Views are published as additional Data Sources in your Tinybird Workspace. This means that you can have a Pipe publish a Materialised View, and then build other Pipes that read from those Materialised Views.

Creating Pipes

You can create as many Pipes in your Workspace as needed.

Creating Pipes in the UI

This section describes how to create a Pipe in the UI.

To add a Pipe, click the Plus (+) icon in the left side navigation bar next to the Pipes section (see Mark 1 below).

../_images/concepts-pipes-creating-a-pipe-ui-1.png

At the top of your new Pipe, you can change the name & description. Click on the name or description to start entering text (see Mark 1 below).

../_images/concepts-pipes-creating-a-pipe-ui-2.png

Creating Pipes in the CLI

This section describes how to create a Pipe in the CLI.

There are two ways to create Pipes in the CLI that behave differently. Both methods are valid & will fit different use cases.

tb pipe new

You can use tb pipe new to directly create a new Pipe in Tinybird. You must provide a name for the Pipe & a single SQL statement.

The Pipe will be created with a single Node containing the provided SQL.

The SQL statement must be wrapped in quotes or the command will fail.

tb pipe new my_pipe_name "SELECT 1"

If you list your Pipes, you’ll see that your Pipe exists in Tinybird.

tb pipe ls

** Pipes:
-----------------------------------------------------------------------------
| prefix | version | name                     | published date      | nodes |
-----------------------------------------------------------------------------
|        |         | my_pipe_name             | 2022-11-30 21:44:55 |     1 |
-----------------------------------------------------------------------------

tb pipe generate

You can also use tb pipe generate to generate a .pipe file. You must provide a name for the Pipe & a single SQL statement.

This command will generate the necessary syntax for a single-Node Pipe inside the file. You can open the file in any text editor to change the name, description, query and add more Nodes.

Defining your Pipes in files allows you to version control your Tinybird resources with git.

The SQL statement must be wrapped in quotes or the command will fail.

tb pipe generate my_pipe_name "SELECT 1"

Generating the .pipe file does not create the Pipe in Tinybird. When you are finished editing the file, you must push the Pipe to Tinybird.

tb push my_pipe_name.pipe

If you list your Pipes, you’ll see that your Pipe exists in Tinybird.

tb pipe ls

** Pipes:
-----------------------------------------------------------------------------
| prefix | version | name                     | published date      | nodes |
-----------------------------------------------------------------------------
|        |         | my_pipe_name             | 2022-11-30 21:44:55 |     1 |
-----------------------------------------------------------------------------

Format descriptions using Markdown

It is possible to use Markdown syntax in Pipe description fields so you can add richer formatting.

Here’s an example using headings, bold and external links:

## This is my first Pipe!

You can use **Markdown** in descriptions.

Add [links](https://tinybird.co) to other bits of info!

This will be rendered in the UI like this:

../_images/concepts-pipes-markdown-in-pipes-desc-1.png