Data sources

When you send data to Tinybird, it's stored in a data source. You then write SQL queries to publish API endpoints that will serve that data.

For example, if your event data lives in a Kafka topic, you can create a data source that connects directly to Kafka and writes the events to Tinybird. Similarly, you can send events or data from a file.

There are also intermediate data sources that are the result of materialization or a copy pipe.

Data sources are defined in .datasource files.

sample.datasource
SCHEMA >
    `timestamp` DateTime `json:$.timestamp`,
    `session_id` String `json:$.session_id`,
    `action` LowCardinality(String) `json:$.action`,
    `version` LowCardinality(String) `json:$.version`,
    `payload` String `json:$.payload`

See all syntax options in the Reference.

Create Data Sources

To create a new data source, you can manually define the .datasource file or run tb datasource create command:

tb datasource create

Once you run the command, you’ll be asked which type of data source you want to create.

  • Blank. Generates a .datasource file with a couple of example columns you can edit.
  • Local file. Creates a .datasource based on the schema of a file you have locally.
  • Remote URL. Creates a .datasource based on the schema of a file from a remote URL.
  • Kafka. Creates a datasource designed to work with a Kafka connection. If you don’t have one yet, you’ll need to create it first, since the schema is built from the topic you select.
  • Amazon S3. For working with an S3 connection. You’ll need an existing connection, as the schema is built from the file in the bucket you choose.
  • GCS. Same idea as S3, but for Google Cloud Storage. Make sure you have a connection set up first.

You can run tb datasource create -h anytime to see this list in the command help.

Delete Data Sources

To delete a data source in Tinybird, you need to remove its corresponding .datasource file and deploy your changes using the --allow-destructive-operations flag to confirm the removal:

tb deploy --allow-destructive-operations

This operation will permanently remove the data source and all its data from your Tinybird workspace. Make sure to review dependencies such as pipes or materialized views that might rely on the data source before deleting it.

Quarantine data sources

Every data source you create in your Workspace has a quarantine data source associated that stores data that doesn't fit the schema. If you send rows that don't fit the data source schema, they're automatically sent to the quarantine table so that the ingest process doesn't fail.

See the Quarantine page for more details.

Updated