---
title: Datasource files
meta:
  description: 'Datasource files describe your Data Sources. Define the schema, engine, and other settings.'
---

# Datasource files (.datasource)

Datasource files describe your Data Sources. You can use .datasource files to define the schema, engine, and other settings of your Data Sources. See [Data Sources](/classic/get-data-in/data-sources).

## Available instructions

The following instructions are available for .datasource files.

{% table %}
   * Declaration
   * Required
   * Description
   ---
   * `SCHEMA <indented_schema_definition>`
   * Yes
   * Defines a block for a Data Source schema. The block must be indented.
   ---
   * `DESCRIPTION <markdown_string>`
   * No
   * Description of the Data Source.
   ---
   * `TOKEN <token_name> APPEND`
   * No
   * Grants append access to a Data Source to the token named <token_name>. If the token isn't specified or <token_name> doesn't exist, Tinybird creates it automatically.
   ---
   * `TAGS <tag_names>`
   * No
   * Comma-separated list of tags. Tags are used to [organize your data project](/classic/work-with-data/organize-your-work/organizing-resources).
   ---
   * `ENGINE <engine_type>`
   * No
   * Sets the engine for Data Source. Default value is `MergeTree`.
   ---
   * `ENGINE_SORTING_KEY <sql>`
   * No
   * Sets the `ORDER BY` expression for the Data Source. If unset, it defaults to DateTime, numeric, or String columns, in that order.
   ---
   * `ENGINE_PARTITION_KEY <sql>`
   * No
   * Sets the `PARTITION` expression for the Data Source. **Use monthly (`toYYYYMM(date_column)`) or yearly (`toYear(date_column)`) partitions only.** Overly granular partition keys severely degrade write performance. See [MergeTree engine](/sql-reference/engines/mergetree#engine-settings) for details.
   ---
   * `ENGINE_TTL <sql>`
   * No
   * Sets the `TTL` expression for the Data Source.
   ---
   * `ENGINE_VER <column_name>`
   * No
   * Column with the version of the object state. Required when using `ENGINE ReplacingMergeTree`.
   ---
   * `ENGINE_SIGN <column_name>`
   * No
   * Column to compute the state. Required when using `ENGINE CollapsingMergeTree` or `ENGINE VersionedCollapsingMergeTree`. 
   ---
   * `ENGINE_VERSION <column_name>`
   * No
   * Column with the version of the object state. Required when `ENGINE VersionedCollapsingMergeTree`. 
   ---
   * `ENGINE_SETTINGS <settings>`
   * No
   * Comma-separated list of key-value pairs that describe engine settings for the Data Source. 
   ---
   * `INDEXES <index definitions>`
   * No
   * Defines one or more indexes for the Data Source. See [Data Skipping Indexes](/sql-reference/engines/mergetree#data-skipping-indexes) for more information.
   ---
   * `SHARED_WITH <workspace_name>`
   * No
   * Shares the Data Source with one or more Workspaces. Use in combination with `--user_token` with admin rights in the origin Workspace.
{% /table %}

The following example shows a typical .datasource file:

```tinybird {% title="tinybird/datasources/example.datasource" %}
# A comment
TOKEN tracker APPEND

DESCRIPTION >
    Analytics events **landing data source**

TAGS stock, recommendations

SCHEMA >
    `timestamp` DateTime `json:$.timestamp`,
    `session_id` String `json:$.session_id`,
    `action` LowCardinality(String) `json:$.action`,
    `version` LowCardinality(String) `json:$.version`,
    `payload` String `json:$.payload`

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(timestamp)"
ENGINE_SORTING_KEY "timestamp"
ENGINE_TTL "timestamp + toIntervalDay(60)"
ENGINE_SETTINGS "index_granularity=8192"

INDEXES >
    INDEX idx1 action TYPE bloom_filter GRANULARITY 3

SHARED_WITH >
    analytics_production
    analytics_staging
```

### SCHEMA

A `SCHEMA` declaration is a newline, comma-separated list of columns definitions. For example:

```tinybird {% title="Example SCHEMA declaration" %}
SCHEMA >
    `timestamp` DateTime `json:$.timestamp`,
    `session_id` String `json:$.session_id`,
    `action` LowCardinality(String) `json:$.action`,
    `version` LowCardinality(String) `json:$.version`,
    `payload` String `json:$.payload`
```

Each column in a `SCHEMA` declaration is in the format `<column_name> <data_type> <json_path> <default_value>`, where:

- `<column_name>` is the name of the column in the Data Source.
- `<data_type>` is one of the supported [Data Types](/classic/get-data-in/data-sources#supported-data-types).
- `<json_path>` is optional and only required for NDJSON Data Sources.
- `<default_value>` sets a default value to the column when it's null. A common use case is to set a default date to a column, like `updated_at DateTime DEFAULT now()`.

To change or update JSONPaths or other default values in the schema, push a new version of the schema using [`tb push --force`](/classic/cli/command-ref#tb-push) or use the [alter endpoint on the Data Sources API](/api-reference/datasource-api#post--v0-datasources-(.+)-alter).

### JSONPath expressions

`SCHEMA` definitions need JSONPath expressions when working with Parquet or NDJSON data.

It supports base fields `json:$.field`, arrays `json:$.an_array[:]`, nested fields `json:$.nested.nested_field`, and storing the whole object `json:$`.

For example, given this JSON object:

```json
{
  "field": "test",
  "nested": { "nested_field": "bla" },
  "an_array": [1, 2, 3],
  "a_nested_array": { "nested_array": [1, 2, 3] }
}
```

The schema would be something like this:

```tinybird {% title="jsonpaths.datasource" %}
SCHEMA >
    field String `json:$.field`,
    nested_nested_field String `json:$.nested.nested_field`,
    an_array Array(Int16) `json:$.an_array[:]`,
    a_nested_array_nested_array Array(Int16) `json:$.a_nested_array.nested_array[:]`,
    whole_message String `json:$`
```

Use brackets for attributes with dots that are not actual nested attributes, like this:

```json
{
    "attributes": {
        "otel.attributes": {
            "cli_command": "datasource ls",
        }
    }
}
```

```tinybird {% title="jsonpaths.datasource" %}
SCHEMA >
    cli_command String `json:$.attributes.['otel.attributes'].cli_command`
```

{% callout %}
Tinybird's JSONPath syntax support has some limitations: It support nested objects at multiple levels, but it __supports nested arrays only at the first level__, as in the example before. To ingest and transform more complex JSON objects, store the whole JSON as a String "<column_name> String `json:$`", and use [JSONExtract functions](/sql-reference/functions/json-functions#jsonextract-functions) to parse at query time or in materializations.
{% /callout %}

### ENGINE settings

`ENGINE` declares the engine used for the Data Source. The default value is `MergeTree`.

See [Engines](/sql-reference/engines) for more information.

## Connectors

Connector settings are part of the .datasource content. You can use include files to reuse connection settings and credentials.

When working with connectors, understand how tokens interact with `.datasource` files. If a token doesn't exist or isn't explicitly specified in the `.datasource` file, Tinybird creates it automatically. This ensures that connectors can establish a working connection by default.

However, once a token is created and associated with a connector, it's crucial to handle it with care. Avoid deleting the token or modifying its scopes, as this can break the connection and disrupt the import process. The token is a critical component for maintaining a stable connection and ensuring that data is imported correctly.

### Kafka, Confluent, RedPanda

The Kafka, Confluent, and RedPanda connectors use the following settings:

{% table %}
   * Instruction
   * Required
   * Description
   ---
   * `KAFKA_CONNECTION_NAME`
   * Yes
   * The name of the configured Kafka connection in Tinybird.
   ---
   * `KAFKA_BOOTSTRAP_SERVERS`
   * Yes
   * Comma-separated list of one or more Kafka brokers, including Port numbers.
   ---
   * `KAFKA_KEY`
   * Yes
   * Key used to authenticate with Kafka. Sometimes called Key, Client Key, or Username, depending on the Kafka distribution.
   ---
   * `KAFKA_SECRET`
   * Yes
   * Secret used to authenticate with Kafka. Sometimes called Secret, Secret Key, or Password, depending on the Kafka distribution.
   ---
   * `KAFKA_TOPIC`
   * Yes
   * Name of the Kafka topic to consume from.
   ---
   * `KAFKA_GROUP_ID`
   * Yes
   * Consumer Group ID to use when consuming from Kafka.
   ---
   * `KAFKA_AUTO_OFFSET_RESET`
   * No
   * Offset to use when no previous offset can be found, for example when creating a new consumer. Supported values are `latest`, `earliest`. Default: `latest`.
   ---
   * `KAFKA_STORE_HEADERS`
   * No
   * Store Kafka headers as field `__headers` for later processing. Default value is `'False'`.
   ---
   * `KAFKA_STORE_BINARY_HEADERS`
   * No
   * Stores all Kafka headers as binary data in field `__headers` as a binary map of type `Map(String, String)`. To access the header `'key'` run: `__headers['key']`. Default value is `'True'`. This field only applies if `KAFKA_STORE_HEADERS` is set to `True`.
   ---
   * `KAFKA_STORE_RAW_VALUE`
   * No
   * Stores the raw message in its entirety as an additional column. Supported values are `'True'`, `'False'`. Default: `'False'`.
   ---
   * `KAFKA_SCHEMA_REGISTRY_URL`
   * No
   * URL of the Kafka schema registry.
   ---
   * `KAFKA_TARGET_PARTITIONS`
   * No
   * Target partitions to place the messages.
   ---
   * `KAFKA_KEY_FORMAT`
   * No
   * Format of the message value. Valid values are `avro`, `json_with_schema`, and `json_without_schema`. Using `avro` or `json_with_schema` requires `KAFKA_SCHEMA_REGISTRY_URL` to be set.
   ---
   * `KAFKA_VALUE_FORMAT`
   * No
   * Format of the message value. Valid values are `avro`, `json_with_schema`, and `json_without_schema`. Using `avro` or `json_with_schema` requires `KAFKA_SCHEMA_REGISTRY_URL` to be set.
   ---
   * `KAFKA_KEY_AVRO_DESERIALIZATION`
   * No
   * If the key of the message is serialized in avro format, allow decoding it using the Avro definition stored in `KAFKA_SCHEMA_REGISTRY_URL`. The key is converted to JSON and stored in the `__key` column. Defaults to `'False'`. A deprecated option is to use the `KAFKA_KEY_FORMAT` parameter with the `avro` value.
   ---
   * `KAFKA_SSL_CA_PEM`
   * No
   * Path to CA certificate in PEM format for SSL connections. 
   ---
   * `KAFKA_SASL_MECHANISM`
   * No
   * SASL mechanism to use for authentication. Supported values are `'PLAIN'`, `'SCRAM-SHA-256'`, `'SCRAM-SHA-512'`. Default values is `'PLAIN'`.
{% /table %}

The following example defines a Data Source with a new Kafka, Confluent, or RedPanda connection in a .datasource file:

```tinybird {% title="Data Source with a new Kafka/Confluent/RedPanda connection" %}
SCHEMA >
  `value` String,
  `topic` LowCardinality(String),
  `partition` Int16,
  `offset` Int64,
  `timestamp` DateTime,
  `key` String

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(timestamp)"
ENGINE_SORTING_KEY "timestamp"

KAFKA_CONNECTION_NAME my_connection_name
KAFKA_BOOTSTRAP_SERVERS my_server:9092
KAFKA_KEY my_username
KAFKA_SECRET my_password
KAFKA_TOPIC my_topic
KAFKA_GROUP_ID my_group_id
```

The following example defines a Data Source that uses an existing Kafka, Confluent, or RedPanda connection:

```tinybird {% title="Data Source with an existing Kafka/Confluent/RedPanda connection" %}
SCHEMA >
  `value` String,
  `topic` LowCardinality(String),
  `partition` Int16,
  `offset` Int64,
  `timestamp` DateTime,
  `key` String

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(timestamp)"
ENGINE_SORTING_KEY "timestamp"

KAFKA_CONNECTION_NAME my_connection_name
KAFKA_TOPIC my_topic
KAFKA_GROUP_ID my_group_id
```

Refer to the [Kafka Connector](/classic/get-data-in/connectors/kafka), [Amazon MSK](/classic/get-data-in/connectors/msk), [Confluent Connector](/classic/get-data-in/connectors/confluent), or [RedPanda Connector](/classic/get-data-in/connectors/redpanda) documentation for more details.

### S3

The S3 connector uses the following settings:

{% table %}
   * Instruction
   * Required
   * Description
   ---
   * `IMPORT_SERVICE`
   * Yes
   * Name of the import service to use. Use `s3_iamrole` for S3 connections.
   ---
   * `IMPORT_SCHEDULE`
   * Yes
   * Use `@auto` to ingest new files automatically, or `@once` to only execute manually.
   ---
   * `IMPORT_CONNECTION_NAME`
   * Yes
   * Name given to the connection inside Tinybird. For example, `'my_connection'`.
   ---
   * `IMPORT_BUCKET_URI`
   * Yes
   * Full bucket path, including the `s3://` protocol, bucket name, object path, and an optional pattern to match against object keys. For example, `s3://my-bucket/my-path` discovers all files in the bucket `my-bucket` under the prefix `/my-path`. You can use patterns in the path to filter objects, for example, ending the path with `*.csv` matches all objects that end with the `.csv` suffix.
   ---
   * `IMPORT_FROM_TIMESTAMP`
   * No
   * Sets the date and time from which to start ingesting files on an S3 bucket. The format is `YYYY-MM-DDTHH:MM:SSZ`.
{% /table %}

See [S3 Connector](/classic/get-data-in/connectors/s3) for more details.

#### S3 example

The following example shows an S3 Data Source described in a .datasource file:

```tinybird {% title="tinybird/datasources/s3.datasource - Data Source with an S3 connection" %}
DESCRIPTION >
    Analytics events landing data source

SCHEMA >
    `timestamp` DateTime `json:$.timestamp`,
    `session_id` String `json:$.session_id`,
    `action` LowCardinality(String) `json:$.action`,
    `version` LowCardinality(String) `json:$.version`,
    `payload` String `json:$.payload`

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(timestamp)"
ENGINE_SORTING_KEY "timestamp"
ENGINE_TTL "timestamp + toIntervalDay(60)"

IMPORT_SERVICE s3_iamrole
IMPORT_CONNECTION_NAME connection_name
IMPORT_BUCKET_URI s3://my-bucket/*.csv
IMPORT_SCHEDULE @auto
```