Sink Pipes API

The Sink Pipes API allows you to create, delete, schedule, and trigger Sink Pipes.

POST /v0/pipes/{pipe_id}/nodes/{node_id}/sink

Set the pipe as a Sink pipe, optionally scheduled. Required token permission: Admin.

Restrictions

  • You can set only one schedule per Sink pipe.
  • You can’t set a Sink pipe if the pipe is already materializing. You must unlink the Materialization first.
  • You can’t set a Sink pipe if the pipe is already an endpoint. You must unpublish the endpoint first.

Example

curl \
  -X POST "https://api.tinybird.co/v0/pipes/:pipe/nodes/:node/sink" \
  -H "Authorization: Bearer <ADMIN token>" \
  -d "connection=my_connection_name" \
  -d "path=s3://bucket-name/prefix" \
  -d "region=eu-west-3" \  # Only a required parameter for IAM role
  -d "
=exported_file_template" \
  -d “format=csv” \
  -d “compression=gz” \
  -d "schedule_cron=0 */1 * * *"

Request parameters

KeyTypeDescription
connectionStringName of the connection to holding the credentials to run the sink
pathStringObject store prefix into which the sink will write data
file_templateStringFile template string. See file template for more details
formatStringOptional. Format of the exported files. Default: CSV
compressionStringOptional. Compression of the output files. Default: None
schedule_cronStringOptional. The sink’s execution schedule, in crontab format.

Successful response example

{
    "id": "t_529f46626c324674b3a84cd820ac2649",
    "name": "p_teste",
    "description": null,
    "endpoint": null,
    "created_at": "2024-01-18 12:57:36.503834",
    "updated_at": "2024-01-18 13:01:21.435012",
    "parent": null,
    "type": "sink",
    "last_commit": {
        "content_sha": "",
        "path": "",
        "status": "changed"
    },
    "sink_node": "t_6e8afdb8c691459b80e16541433f951b",
    "schedule": {
        "timezone": "Etc/UTC",
        "cron": "0 */1 * * *",
        "status": "running"
    },
    "nodes": [
        {
            "id": "t_6e8afdb8c691459b80e16541433f951b",
            "name": "p_teste_0",
            "sql": "SELECT * FROM teste",
            "description": null,
            "materialized": null,
            "cluster": null,
            "tags": {},
            "created_at": "2024-01-18 12:57:36.503843",
            "updated_at": "2024-01-18 12:57:36.503843",
            "version": 0,
            "project": null,
            "result": null,
            "ignore_sql_errors": false,
            "node_type": "sink",
            "dependencies": [
                "teste"
            ],
            "params": []
        }
    ]
}

Response codes

CodeDescription
200OK
404Pipe, Node, or data connector not found, bucket doesn’t exist
403Limit reached, Query includes forbidden keywords, Pipe is already a Sink Pipe, cannot assume role
401Invalid credentials (from connection)
400Invalid or missing parameters, bad ARN role, invalid region name

DELETE /v0/pipes/{pipe_id}/nodes/{node_id}/sink

Removes the Sink from the pipe. This does not delete the pipe nor the node, only the sink configuration and any associated settings.

Example

curl \
  -X DELETE "https://api.tinybird.co/v0/pipes/$1/nodes/$2/sink" \
  -H "Authorization: Bearer <ADMIN token>"

Successful response example
{
    "id": "t_529f46626c324674b3a84cd820ac2649",
    "name": "p_teste",
    "description": null,
    "endpoint": null,
    "created_at": "2024-01-18 12:57:36.503834",
    "updated_at": "2024-01-19 09:27:12.069650",
    "parent": null,
    "type": "default",
    "last_commit": {
        "content_sha": "",
        "path": "",
        "status": "changed"
    },
    "nodes": [
        {
            "id": "t_6e8afdb8c691459b80e16541433f951b",
            "name": "p_teste_0",
            "sql": "SELECT * FROM teste",
            "description": null,
            "materialized": null,
            "cluster": null,
            "tags": {},
            "created_at": "2024-01-18 12:57:36.503843",
            "updated_at": "2024-01-19 09:27:12.069649",
            "version": 0,
            "project": null,
            "result": null,
            "ignore_sql_errors": false,
            "node_type": "standard",
            "dependencies": [
                "teste"
            ],
            "params": []
        }
    ],
    "url": "https://api.split.tinybird.co/v0/pipes/p_teste.json"
}

Response codes

CodeDescription
200OK
404Pipe, Node, or data connector not found
403Limit reached, Query includes forbidden keywords, Pipe is already a Sink Pipe
400Invalid or missing parameters, Pipe is not a Sink Pipe

POST /v0/pipes/{pipe_id}/sink

Triggers the sink pipe, creating a sink job. Allows overriding some of the sink settings for this particular execution

Example

curl \
  -X POST "https://api.tinybird.co/v0/pipes/p_teste/sink" \
  -H "Authorization: Bearer <ADMIN token>" \
  -d “format=csv” \
  -d “compression=gz” \
  -d “file_template=export_file” \
  -d <key>=<val>

Request parameters

KeyTypeDescription
connectionStringName of the connection to holding the credentials to run the sink
pathStringObject store prefix into which the sink will write data
file_templateStringFile template string. See file template for more details
formatStringOptional. Format of the exported files. Default: CSV
compressionStringOptional. Compression of the output files. Default: None
<key>StringOptional. Additional variables to be injected into the file template. See file template for more details

Successful response example

{
    "id": "t_6e8afdb8c691459b80e16541433f951b",
    "name": "p_teste_0",
    "sql": "SELECT * FROM teste",
    "description": null,
    "materialized": null,
    "cluster": null,
    "tags": {},
    "created_at": "2024-01-18 12:57:36.503843",
    "updated_at": "2024-01-19 09:27:12.069649",
    "version": 0,
    "project": null,
    "result": null,
    "ignore_sql_errors": false,
    "node_type": "sink",
    "dependencies": [
        "teste"
    ],
    "params": [],
    "job": {
        "id": "685e7395-3b08-492b-9fe8-2944859d6a06",
        "kind": "sink",
        "status": "waiting",
        "created_at": "2024-01-19 15:58:46.688525",
        "updated_at": "2024-01-19 15:58:46.688532",
        "is_cancellable": true,
        "job_url": "https://api.split.tinybird.co/v0/jobs/685e7395-3b08-492b-9fe8-2944859d6a06",
        "pipe": {
            "id": "t_529f46626c324674b3a84cd820ac2649",
            "name": "p_teste"
        }
    }
}

Response codes

CodeDescription
200OK
404Pipe, Node, or data connector not found
403Limit reached, Query includes forbidden keywords, Pipe is already a Sink Pipe
400Invalid or missing parameters, Pipe is not a Sink Pipe

GET /v0/integrations/s3/policies/trust-policy

Retrieves the trust policy to be attached to the IAM role that will be used for the connection. External IDs are different for each Workspace, but shared between Branches of the same Workspace to avoid having to change the trust policy for each Branch.

Example

curl \
  -X GET "https://$TB_HOST/v0/integrations/s3/policies/trust-policy" \
  -H "Authorization: Bearer <ADMIN token>"

Successful response example
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Principal": {
                "AWS": "arn:aws:iam::123456789:root"
            },
            "Condition": {
                "StringEquals": {
                    "sts:ExternalId": "c6ee2795-aae3-4a55-a7a1-92d92fab0e41"
                }
            }
        }
    ]
}

Response codes

CodeDescription
200OK
404S3 integration not supported in your region

GET /v0/integrations/s3/policies/access-policy

Retrieves the trust policy to be attached to the IAM Role that will be used for the connection. External IDs are different for each workspace, but shared between branches of the same workspace to avoid having to change the trust policy for each branch.

Example

curl \
  -X GET "https://$TB_HOST/v0/integrations/s3/policies/access-policy?bucket=test-bucket" \
  -H "Authorization: Bearer <ADMIN token>"

Request parameters

KeyTypeDescription
bucketOptional[String]Bucket to use for rendering the template. If not provided the ‘<bucket>’ placeholder is used

Successful response example

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetBucketLocation",
                "s3:ListBucket"
            ],
            "Resource": "arn:aws:s3:::<bucket>"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": "arn:aws:s3:::<bucket>/*"
        }
    ]
}

Response codes

CodeDescription
200OK

GET /v0/datasources-bigquery-credentials

Retrieves the workspace’s GCP service account to be authorized to write to the destination bucket.

Example

curl \
  -X POST "${TINYBIRD_HOST}/v0/connectors" \
  -H "Authorization: Bearer <ADMIN TOKEN>" \
  -d "service=gcs_service_account" \
  -d "name=<name>"

Request parameters

None

Successful response example

{
    "account": "cdk-E-d83f6d01-b5c1-40-43439d@development-353413.iam.gserviceaccount.com"
}

Response codes

CodeDescription
200OK
503Feature not enabled in your region