Sink Pipes API
The Sink Pipes API allows you to create, delete, schedule, and trigger Sink Pipes.
POST /v0/pipes/{pipe_id}/nodes/{node_id}/sink
Set the pipe as a Sink pipe, optionally scheduled. Required token permission: Admin.
Restrictions
- You can set only one schedule per Sink pipe.
- You can’t set a Sink pipe if the pipe is already materializing. You must unlink the Materialization first.
- You can’t set a Sink pipe if the pipe is already an endpoint. You must unpublish the endpoint first.
Example
curl \
-X POST "https://api.tinybird.co/v0/pipes/:pipe/nodes/:node/sink" \
-H "Authorization: Bearer <ADMIN token>" \
-d "connection=my_connection_name" \
-d "path=s3://bucket-name/prefix" \
-d "region=eu-west-3" \ # Only a required parameter for IAM role
-d "
=exported_file_template" \
-d “format=csv” \
-d “compression=gz” \
-d "schedule_cron=0 */1 * * *"
Request parameters
Key | Type | Description |
---|
connection | String | Name of the connection to holding the credentials to run the sink |
path | String | Object store prefix into which the sink will write data |
file_template | String | File template string. See file template for more details |
format | String | Optional. Format of the exported files. Default: CSV |
compression | String | Optional. Compression of the output files. Default: None |
schedule_cron | String | Optional. The sink’s execution schedule, in crontab format. |
Successful response example
{
"id": "t_529f46626c324674b3a84cd820ac2649",
"name": "p_teste",
"description": null,
"endpoint": null,
"created_at": "2024-01-18 12:57:36.503834",
"updated_at": "2024-01-18 13:01:21.435012",
"parent": null,
"type": "sink",
"last_commit": {
"content_sha": "",
"path": "",
"status": "changed"
},
"sink_node": "t_6e8afdb8c691459b80e16541433f951b",
"schedule": {
"timezone": "Etc/UTC",
"cron": "0 */1 * * *",
"status": "running"
},
"nodes": [
{
"id": "t_6e8afdb8c691459b80e16541433f951b",
"name": "p_teste_0",
"sql": "SELECT * FROM teste",
"description": null,
"materialized": null,
"cluster": null,
"tags": {},
"created_at": "2024-01-18 12:57:36.503843",
"updated_at": "2024-01-18 12:57:36.503843",
"version": 0,
"project": null,
"result": null,
"ignore_sql_errors": false,
"node_type": "sink",
"dependencies": [
"teste"
],
"params": []
}
]
}
Response codes
Code | Description |
---|
200 | OK |
404 | Pipe, Node, or data connector not found, bucket doesn’t exist |
403 | Limit reached, Query includes forbidden keywords, Pipe is already a Sink Pipe, cannot assume role |
401 | Invalid credentials (from connection) |
400 | Invalid or missing parameters, bad ARN role, invalid region name |
DELETE /v0/pipes/{pipe_id}/nodes/{node_id}/sink
Removes the Sink from the pipe. This does not delete the pipe nor the node, only the sink configuration and any associated settings.
Example
curl \
-X DELETE "https://api.tinybird.co/v0/pipes/$1/nodes/$2/sink" \
-H "Authorization: Bearer <ADMIN token>"
Successful response example
{
"id": "t_529f46626c324674b3a84cd820ac2649",
"name": "p_teste",
"description": null,
"endpoint": null,
"created_at": "2024-01-18 12:57:36.503834",
"updated_at": "2024-01-19 09:27:12.069650",
"parent": null,
"type": "default",
"last_commit": {
"content_sha": "",
"path": "",
"status": "changed"
},
"nodes": [
{
"id": "t_6e8afdb8c691459b80e16541433f951b",
"name": "p_teste_0",
"sql": "SELECT * FROM teste",
"description": null,
"materialized": null,
"cluster": null,
"tags": {},
"created_at": "2024-01-18 12:57:36.503843",
"updated_at": "2024-01-19 09:27:12.069649",
"version": 0,
"project": null,
"result": null,
"ignore_sql_errors": false,
"node_type": "standard",
"dependencies": [
"teste"
],
"params": []
}
],
"url": "https://api.split.tinybird.co/v0/pipes/p_teste.json"
}
Response codes
Code | Description |
---|
200 | OK |
404 | Pipe, Node, or data connector not found |
403 | Limit reached, Query includes forbidden keywords, Pipe is already a Sink Pipe |
400 | Invalid or missing parameters, Pipe is not a Sink Pipe |
POST /v0/pipes/{pipe_id}/sink
Triggers the sink pipe, creating a sink job. Allows overriding some of the sink settings for this particular execution
Example
curl \
-X POST "https://api.tinybird.co/v0/pipes/p_teste/sink" \
-H "Authorization: Bearer <ADMIN token>" \
-d “format=csv” \
-d “compression=gz” \
-d “file_template=export_file” \
-d <key>=<val>
Request parameters
Key | Type | Description |
---|
connection | String | Name of the connection to holding the credentials to run the sink |
path | String | Object store prefix into which the sink will write data |
file_template | String | File template string. See file template for more details |
format | String | Optional. Format of the exported files. Default: CSV |
compression | String | Optional. Compression of the output files. Default: None |
<key> | String | Optional. Additional variables to be injected into the file template. See file template for more details |
Successful response example
{
"id": "t_6e8afdb8c691459b80e16541433f951b",
"name": "p_teste_0",
"sql": "SELECT * FROM teste",
"description": null,
"materialized": null,
"cluster": null,
"tags": {},
"created_at": "2024-01-18 12:57:36.503843",
"updated_at": "2024-01-19 09:27:12.069649",
"version": 0,
"project": null,
"result": null,
"ignore_sql_errors": false,
"node_type": "sink",
"dependencies": [
"teste"
],
"params": [],
"job": {
"id": "685e7395-3b08-492b-9fe8-2944859d6a06",
"kind": "sink",
"status": "waiting",
"created_at": "2024-01-19 15:58:46.688525",
"updated_at": "2024-01-19 15:58:46.688532",
"is_cancellable": true,
"job_url": "https://api.split.tinybird.co/v0/jobs/685e7395-3b08-492b-9fe8-2944859d6a06",
"pipe": {
"id": "t_529f46626c324674b3a84cd820ac2649",
"name": "p_teste"
}
}
}
Response codes
Code | Description |
---|
200 | OK |
404 | Pipe, Node, or data connector not found |
403 | Limit reached, Query includes forbidden keywords, Pipe is already a Sink Pipe |
400 | Invalid or missing parameters, Pipe is not a Sink Pipe |
GET /v0/integrations/s3/policies/trust-policy
Retrieves the trust policy to be attached to the IAM role that will be used for the connection. External IDs are different for each Workspace, but shared between Branches of the same Workspace to avoid having to change the trust policy for each Branch.
Example
curl \
-X GET "https://$TB_HOST/v0/integrations/s3/policies/trust-policy" \
-H "Authorization: Bearer <ADMIN token>"
Successful response example
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Principal": {
"AWS": "arn:aws:iam::123456789:root"
},
"Condition": {
"StringEquals": {
"sts:ExternalId": "c6ee2795-aae3-4a55-a7a1-92d92fab0e41"
}
}
}
]
}
Response codes
Code | Description |
---|
200 | OK |
404 | S3 integration not supported in your region |
GET /v0/integrations/s3/policies/access-policy
Retrieves the trust policy to be attached to the IAM Role that will be used for the connection. External IDs are different for each workspace, but shared between branches of the same workspace to avoid having to change the trust policy for each branch.
Example
curl \
-X GET "https://$TB_HOST/v0/integrations/s3/policies/access-policy?bucket=test-bucket" \
-H "Authorization: Bearer <ADMIN token>"
Request parameters
Key | Type | Description |
---|
bucket | Optional[String] | Bucket to use for rendering the template. If not provided the ‘<bucket>’ placeholder is used |
Successful response example
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetBucketLocation",
"s3:ListBucket"
],
"Resource": "arn:aws:s3:::<bucket>"
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:PutObjectAcl"
],
"Resource": "arn:aws:s3:::<bucket>/*"
}
]
}
Response codes
GET /v0/datasources-bigquery-credentials
Retrieves the workspace’s GCP service account to be authorized to write to the destination bucket.
Example
curl \
-X POST "${TINYBIRD_HOST}/v0/connectors" \
-H "Authorization: Bearer <ADMIN TOKEN>" \
-d "service=gcs_service_account" \
-d "name=<name>"
Request parameters
None
Successful response example
{
"account": "cdk-E-d83f6d01-b5c1-40-43439d@development-353413.iam.gserviceaccount.com"
}
Response codes
Code | Description |
---|
200 | OK |
503 | Feature not enabled in your region |