Pipes

What is a Pipe?

A Pipe is a collection of one or more SQL queries (each query is called a Node).

What should I use Pipes for?

Use Pipes to build features over your data.

Write SQL that joins, aggregates, or otherwise transforms your data and publish the result.

You have three options to publish the result of a Pipe: API Endpoints, Materialized Views, and Copy Pipes.

A Pipe can only have a single output at one time. This means that you cannot create a Materialized View and an API Endpoint from the same Pipe, at the same time.

Creating Pipes

You can create as many Pipes in your Workspace as needed.

Creating Pipes in the UI

To add a Pipe, click the Plus (+) icon in the left side navigation bar next to the Pipes section (see Mark 1 below).

image

At the top of your new Pipe, you can change the name & description. Click on the name or description to start entering text (see Mark 1 below).

image

Creating Pipes in the CLI

tb pipe generate

You can use tb pipe generate to generate a .pipe file. You must provide a name for the Pipe & a single SQL statement.

This command will generate the necessary syntax for a single-Node Pipe inside the file. You can open the file in any text editor to change the name, description, query and add more Nodes.

Defining your Pipes in files allows you to version control your Tinybird resources with git.

The SQL statement must be wrapped in quotes or the command will fail.

tb pipe generate my_pipe_name "SELECT 1"

Generating the .pipe file does not create the Pipe in Tinybird. When you are finished editing the file, you must push the Pipe to Tinybird.

tb push my_pipe_name.pipe

If you list your Pipes, you'll see that your Pipe exists in Tinybird.

tb pipe ls

** Pipes:
--------------------------------------------------------------------
| version | name                     | published date      | nodes |
--------------------------------------------------------------------
|         | my_pipe_name             | 2022-11-30 21:44:55 |     1 |
--------------------------------------------------------------------

Format descriptions using Markdown

It is possible to use Markdown syntax in Pipe description fields so you can add richer formatting.

Here's an example using headings, bold and external links:

## This is my first Pipe!

You can use **Markdown** in descriptions.

Add [links](https://www.tinybird.co) to other bits of info!

This will be rendered in the UI like this:

image

Nodes

What is a Node?

A Node is a container for a single SQL SELECT statement. Nodes live within Pipes, and you can have many Nodes inside the same Pipe.

A query in a Node can read data from a Data Source, other Nodes inside the same Pipe, or from API Endpoint Nodes in other Pipes.

What should I use Nodes for?

Nodes allow you to break your query logic down into multiple, smaller queries. You can then chain Nodes together to build the logic incrementally.

Each Node can be developed & tested individually. This makes it much easier to build complex query logic in Tinybird as you avoid creating large monolithic queries with many sub-queries.