Pipes

What is a Pipe?

A Pipe contains one or more SQL queries (Nodes) that result in either an API endpoint or a Materialized View.

What should I use Pipes for?

Use Pipes to build features over your data.

Write SQL that joins, aggregates or otherwise transforms your data and publish the result.

You have two options to publish the result of a Pipe: API Endpoints, or Materialized Views.

A Pipe can only have a single output at one time. This means that you cannot create a Materialized View and an API Endpoint from the same Pipe, at the same time.

Publishing as an API Endpoint gives you an HTTP GET API with the result of the Pipe in JSON, NDJSON, CSV or Parquet formats. You can use this API directly in your applications to access your data.

Materialized Views are published as additional Data Sources in your Tinybird Workspace. This means that you can have a Pipe publish a Materialized View, and then build other Pipes that read from those Materialized Views.

Creating Pipes

You can create as many Pipes in your Workspace as needed.

Creating Pipes in the UI

This section describes how to create a Pipe in the UI.

To add a Pipe, click the Plus (+) icon in the left side navigation bar next to the Pipes section (see Mark 1 below).

../_images/concepts-pipes-creating-a-pipe-ui-1.png

At the top of your new Pipe, you can change the name & description. Click on the name or description to start entering text (see Mark 1 below).

../_images/concepts-pipes-creating-a-pipe-ui-2.png

Creating Pipes in the CLI

This section describes how to create a Pipe in the CLI.

There are two ways to create Pipes in the CLI that behave differently. Both methods are valid & will fit different use cases.

tb pipe new

You can use tb pipe new to directly create a new Pipe in Tinybird. You must provide a name for the Pipe & a single SQL statement.

The Pipe will be created with a single Node containing the provided SQL.

The SQL statement must be wrapped in quotes or the command will fail.

tb pipe new my_pipe_name "SELECT 1"

If you list your Pipes, you’ll see that your Pipe exists in Tinybird.

tb pipe ls

** Pipes:
--------------------------------------------------------------------
| version | name                     | published date      | nodes |
--------------------------------------------------------------------
|         | my_pipe_name             | 2022-11-30 21:44:55 |     1 |
--------------------------------------------------------------------

tb pipe generate

You can also use tb pipe generate to generate a .pipe file. You must provide a name for the Pipe & a single SQL statement.

This command will generate the necessary syntax for a single-Node Pipe inside the file. You can open the file in any text editor to change the name, description, query and add more Nodes.

Defining your Pipes in files allows you to version control your Tinybird resources with git.

The SQL statement must be wrapped in quotes or the command will fail.

tb pipe generate my_pipe_name "SELECT 1"

Generating the .pipe file does not create the Pipe in Tinybird. When you are finished editing the file, you must push the Pipe to Tinybird.

tb push my_pipe_name.pipe

If you list your Pipes, you’ll see that your Pipe exists in Tinybird.

tb pipe ls

** Pipes:
--------------------------------------------------------------------
| version | name                     | published date      | nodes |
--------------------------------------------------------------------
|         | my_pipe_name             | 2022-11-30 21:44:55 |     1 |
--------------------------------------------------------------------

Format descriptions using Markdown

It is possible to use Markdown syntax in Pipe description fields so you can add richer formatting.

Here’s an example using headings, bold and external links:

## This is my first Pipe!

You can use **Markdown** in descriptions.

Add [links](https://tinybird.co) to other bits of info!

This will be rendered in the UI like this:

../_images/concepts-pipes-markdown-in-pipes-desc-1.png

Nodes

What is a Node?

A Node is a container for a single SQL SELECT statement. Nodes live within Pipes, and you can have many Nodes inside the same Pipe.

A query in a Node can read data from a Data Source, other Nodes inside the same Pipe, or from API Endpoint Nodes in other Pipes.

What should I use Nodes for?

Nodes allow you to break your query logic down into multiple, smaller queries. You can then chain Nodes together to build the logic incrementally.

Each Node can be developed & tested individually. This makes it much easier to build complex query logic in Tinybird as you avoid creating large monolithic queries with many sub-queries.