# Tinybird Documentation (FWD)
Generated on: 2026-08-10T12:52:45.630Z
URL: https://www.tinybird.co/docs/forward/support
Last update: 2026-08-07T12:15:37.000Z
Content:
---
title: "Support \| Tinybird"
theme-color: "#171612"
description: "Tinybird support plans, scope, channels, hours, and how to request help."
inkeep:version: "forward"
---
# Support (https://www.tinybird.co/docs/forward/support#support)
Tinybird provides support through different channels depending on your plan. Free workspaces can request support only through the Community Slack. Developer includes email support. SaaS and Enterprise include Standard or Premier support with priority email, live chat, and a dedicated customer Slack channel.
See [Pricing plans](/docs/forward/pricing) for plan details.
## Support scope (https://www.tinybird.co/docs/forward/support#support-scope)
We divide support into three categories. Each covers different needs and engagement levels.
### Technical support (https://www.tinybird.co/docs/forward/support#technical-support)
Technical support resolves issues you encounter while using Tinybird:
- Production incident investigation
- Troubleshooting errors and blocked workflows
- Potential product bug triage
- Operational and authentication issues
- Questions about existing features
- Supported usage and configuration guidance
### Customer Success (https://www.tinybird.co/docs/forward/support#customer-success)
Customer Success focuses on long-term outcomes, not urgent incidents:
- Adoption guidance
- Success planning
- High-level architecture reviews
- Usage optimization
- Performance recommendations
### Professional Services (https://www.tinybird.co/docs/forward/support#professional-services)
Professional Services is a separate paid engagement, not included with Standard or Premier support:
- Hands-on implementation
- Migrations
- Detailed architecture design
- Deep optimization
- Pipeline building
- Training
## Support plans (https://www.tinybird.co/docs/forward/support#support-plans)
We offer two support plans for SaaS and Enterprise: Standard and Premier.
Standard is included with SaaS and Enterprise. Premier is available for SaaS S-24+ and Enterprise. Your customer agreement defines your support plan, pricing, and terms.
### Standard (https://www.tinybird.co/docs/forward/support#standard)
Standard includes:
- Priority email
- Live chat
- Dedicated customer Slack channel
### Premier (https://www.tinybird.co/docs/forward/support#premier)
Premier includes everything in Standard plus:
- Designated Customer Success engineer
- Prioritized SLA
- On-call requests
- Monthly partnership meeting
- Onboarding support
Premier pricing depends on your plan, deployment, and customer agreement. Support charges don't depend on ticket volume or platform activity. On-call allowances vary by deployment and agreement. For details, use the [pricing calculator](https://www.tinybird.co/pricing/calculator) or [contact sales](https://www.tinybird.co/contact-us).
## Support channels (https://www.tinybird.co/docs/forward/support#support-channels)
| Plan | Channels |
| --- | --- |
| Free | [Community Slack](/docs/community) only |
| Developer | Community Slack and email support at [support@tinybird.co](mailto:support@tinybird.co) |
| SaaS and Enterprise (Standard or Premier) | Priority email, live chat, Community Slack, and a dedicated customer Slack channel |
## Support hours and response commitments (https://www.tinybird.co/docs/forward/support#support-hours-and-response-commitments)
Our email support at [support@tinybird.co](mailto:support@tinybird.co) is available from 9:00 AM to 6:30 PM CET on a Business Day. Other agreed support channels follow those hours unless your customer agreement says otherwise.
Response commitments depend on your plan, incident severity, and customer agreement. We don't provide default 24/7 support. Out-of-hours support is only available when specifically agreed and for Critical or High severity incidents.
Your customer agreement is the source of truth for hours, response commitments, and severity definitions. See the [Terms and Conditions](https://www.tinybird.co/terms-and-conditions).
## How to contact support (https://www.tinybird.co/docs/forward/support#how-to-contact-support)
To help us investigate quickly, include the following when you contact support:
- Organization or Workspace ID. See [Workspace ID](/docs/forward/core-concepts/workspaces#workspace-id).
- Affected resources (Data Sources, Pipes, Endpoints, Connections)
- Timestamps and timezone
- Request or query IDs
- Exact error messages
- Expected and actual behavior
- Recent changes (deploys, schema changes, configuration updates)
- Steps to reproduce the issue
Never send API tokens or other secrets to support. We never ask for them.
## Integrated troubleshooting (https://www.tinybird.co/docs/forward/support#integrated-troubleshooting)
Tinybird tries to give you direct feedback and notifications if it spots anything going wrong. Use Tinybird's [Service Data Sources](/docs/forward/monitoring/service-datasources) to get more details on what's going on in your data and queries.
## Recover deleted items (https://www.tinybird.co/docs/forward/support#recover-deleted-items)
Tinybird creates and backs up daily snapshots and retains them for 7 days.
## Recover data from quarantine (https://www.tinybird.co/docs/forward/support#recover-data-from-quarantine)
The quickest way to recover rows from quarantine is to fix the cause of the errors and then reingest the data. See [Recover data from quarantine](/docs/forward/guides/quarantine#recovering-rows-from-quarantine).
You can also use the [Service Data Sources](/docs/forward/monitoring/service-datasources), like `datasources_ops_log`.
## Useful links (https://www.tinybird.co/docs/forward/support#useful-links)
- [Community Slack](/docs/community)
- email support: [support@tinybird.co](mailto:support@tinybird.co)
- [Service status](https://status.tinybird.co/)
- [Terms and Conditions](https://www.tinybird.co/terms-and-conditions)
- [Pricing calculator](https://www.tinybird.co/pricing/calculator)
- [Contact sales](https://www.tinybird.co/contact-us)
- [Service Data Sources](/docs/forward/monitoring/service-datasources)
---
URL: https://www.tinybird.co/docs/forward/quickstarts
Last update: 2026-07-29T12:18:04.000Z
Content:
---
title: "Quickstarts \| Tinybird"
theme-color: "#171612"
description: "Start building on managed ClickHouse with Tinybird."
inkeep:version: "forward"
---
# Quickstarts (https://www.tinybird.co/docs/forward/quickstarts#quickstarts)
Tinybird combines managed ClickHouse® with a developer experience and tooling fine-tuned for agentic coding and modern development workflows. Build analytics features by ingesting data, transforming it with SQL, and exposing the results to applications, tools, and agents.
## What Tinybird adds to ClickHouse (https://www.tinybird.co/docs/forward/quickstarts#what-tinybird-adds-to-clickhouse)
- Ingest streaming and batch data through the Events API and managed [Connections](/docs/forward/core-concepts/connections) for Kafka, Amazon S3, and Google Cloud Storage.
- Define Data Sources, Connections, Pipes, API Endpoints, and Materialized Views as code with TypeScript, Python, or datafiles.
- Develop with [Tinybird Local](/docs/forward/core-concepts/tinybird-local) and isolated [Cloud Branches](/docs/forward/development-workflow/using-cloud-branches), then deploy through [CI/CD](/docs/forward/development-workflow/cicd).
- Query data through Endpoints, the [Query API](/docs/api-reference/query-api), the [ClickHouse® interface](/docs/forward/query-data/clickhouse-interface), or [MCP](/docs/forward/query-data/mcp).
## Choose how to build (https://www.tinybird.co/docs/forward/quickstarts#choose-how-to-build)
- TypeScript SDK
Define Data Sources and Endpoints with typed resources.
- Python SDK
Define resources in Python and deploy with the Tinybird CLI.
- Tinybird CLI
Use datafiles, the Tinybird CLI, and cloud branches.
## Not sure where to start? (https://www.tinybird.co/docs/forward/quickstarts#not-sure-where-to-start)
- Prefer SQL-first workflows and datafiles: start with [Tinybird CLI](/docs/forward/quickstarts/cli).
- Want typed resources and a TypeScript-native workflow: start with [TypeScript SDK](/docs/forward/quickstarts/typescript-sdk).
- Want Python-native resource definitions and runtime usage: start with [Python SDK](/docs/forward/quickstarts/python-sdk).
## Work with Tinybird using AI agents (https://www.tinybird.co/docs/forward/quickstarts#work-with-tinybird-using-ai-agents)
[Tinybird Agent Skills](/docs/forward/development-workflow/agent-skills) give compatible coding agents instructions for creating and reviewing Tinybird project files, SQL, tests, and deployments. Install them in your coding environment:
Install Tinybird Agent Skills
```
npx skills add tinybirdco/tinybird-agent-skills
```
The [Tinybird MCP server](/docs/forward/query-data/mcp) gives an agent Token-scoped access to a live Workspace. The agent can inspect and query data and call published Endpoints according to the Token's permissions. Use Agent Skills and MCP together when an agent needs both project guidance and access to live Workspace data.
## Frequently asked questions about Tinybird (https://www.tinybird.co/docs/forward/quickstarts#frequently-asked-questions-about-tinybird)
### Why use Tinybird instead of managing ClickHouse yourself? (https://www.tinybird.co/docs/forward/quickstarts#why-use-tinybird-instead-of-managing-clickhouse-yourself)
Self-hosting ClickHouse requires you to operate clusters, sharding, replicas, security, monitoring, updates, backups, and deployment pipelines. Tinybird manages the ClickHouse infrastructure and adds ingestion, API publishing, authentication, and development tooling so you can focus on building analytics features. See [Migrate from ClickHouse self-hosted to Tinybird](/docs/forward/guides/migrate-from-clickhouse).
### When should you choose Tinybird instead of ClickHouse Cloud? (https://www.tinybird.co/docs/forward/quickstarts#when-should-you-choose-tinybird-instead-of-clickhouse-cloud)
Choose Tinybird when you need an app development workflow around ClickHouse®: managed ingestion, resources as code, authenticated API Endpoints, Tinybird Local, Cloud Branches, CI/CD, and MCP. You can still connect existing SQL clients and BI tools through the [ClickHouse® interface](/docs/forward/query-data/clickhouse-interface).
### Does Tinybird support data branching and schema iteration? (https://www.tinybird.co/docs/forward/quickstarts#does-tinybird-support-data-branching-and-schema-iteration)
Yes. [Cloud Branches](/docs/forward/core-concepts/branches) create isolated environments where you can change schemas, queries, APIs, and connector ingestion without affecting production. A Branch can attach the latest production partition without copying the underlying data, so you can validate changes before deploying them to your main Workspace.
### How do you define and deploy Tinybird resources as code? (https://www.tinybird.co/docs/forward/quickstarts#how-do-you-define-and-deploy-tinybird-resources-as-code)
Define Tinybird resources with the [TypeScript SDK](/docs/forward/quickstarts/typescript-sdk), [Python SDK](/docs/forward/quickstarts/python-sdk), or [datafiles](/docs/forward/dev-reference/datafiles). Commit them to Git, test with Tinybird Local or Cloud Branches, and deploy with the Tinybird CLI or [CI/CD](/docs/forward/development-workflow/cicd).
### Which ingestion sources and managed Connections does Tinybird support? (https://www.tinybird.co/docs/forward/quickstarts#which-ingestion-sources-and-managed-connections-does-tinybird-support)
Use the [Events API](/docs/forward/ingest-data/events-api) for HTTP streaming, managed Connections for [Kafka](/docs/forward/ingest-data/connectors/kafka), [Amazon S3](/docs/forward/ingest-data/connectors/s3), and [Google Cloud Storage](/docs/forward/ingest-data/connectors/gcs), or upload local files. Connections store credentials and integration settings and deploy with the rest of your Tinybird project.
### How can apps and AI agents query Tinybird? (https://www.tinybird.co/docs/forward/quickstarts#how-can-apps-and-ai-agents-query-tinybird)
Apps can call published [Endpoints](/docs/forward/core-concepts/api-endpoints), run SQL through the [Query API](/docs/api-reference/query-api), or use the ClickHouse® interface. AI agents can use the [Tinybird MCP server](/docs/forward/query-data/mcp) to query permitted data and call published Endpoints. [Tokens](/docs/forward/core-concepts/tokens) control which resources each client or agent can access.
### Which Tinybird pricing plans are available? (https://www.tinybird.co/docs/forward/quickstarts#which-tinybird-pricing-plans-are-available)
Tinybird offers Free, Developer, and SaaS plans on shared infrastructure, plus an Enterprise plan with a dedicated ClickHouse® cluster. See [Pricing plans](/docs/forward/pricing) for plan features and the current [Tinybird pricing](https://www.tinybird.co/pricing) for prices.
---
URL: https://www.tinybird.co/docs/forward/query-data
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Query data \| Tinybird"
theme-color: "#171612"
description: "Learn the ways to query data in Tinybird."
inkeep:version: "forward"
---
# Query data (https://www.tinybird.co/docs/forward/query-data#query-data)
Tinybird gives you several ways to query data, depending on where the query runs and who consumes the result.
Use [API Endpoints](/docs/forward/query-data/api-endpoints) when an app, dashboard, or agent needs a stable HTTP API. Use the Query API or ClickHouse® interface when a tool needs to run ad hoc SQL. Use Playgrounds, Explorations, and Time Series when you are investigating data inside Tinybird Cloud.
## Query from outside Tinybird (https://www.tinybird.co/docs/forward/query-data#query-from-outside-tinybird)
**[API Endpoints](/docs/forward/query-data/api-endpoints)** publish the result of a Pipe as a REST API. They are the production path for serving query results to applications, user interfaces, dashboards, and agents. Use Tinybird's [templating language](/docs/forward/core-concepts/templating-language) to add typed query parameters, filters, sorting, and limits.
**[Query API](/docs/forward/query-data/sql-api)** runs SQL directly against Pipes and Data Sources in your Workspace through the `/v0/sql` endpoint. Use it for internal tools, automation, notebooks, scripts, and ad hoc queries that do not need a published Endpoint contract.
**[ClickHouse® interface](/docs/forward/query-data/clickhouse-interface)** lets ClickHouse-compatible clients, BI tools, and SQL IDEs query Tinybird Data Sources over HTTPS.
**[MCP](/docs/forward/query-data/mcp)** lets AI agents query your Workspace and use published Endpoints as tools through the [Model Context Protocol](https://modelcontextprotocol.io/).
## Query inside Tinybird (https://www.tinybird.co/docs/forward/query-data#query-inside-tinybird)
**[Playgrounds](/docs/forward/query-data/playgrounds)** let you write and test SQL interactively before moving finished logic into Pipes.
**[Explorations](/docs/forward/query-data/explorations)** is a conversational UI in Tinybird Cloud for asking questions about your data in natural language.
**[Time Series](/docs/forward/query-data/time-series)** visualizes time-based metrics from Data Sources, Service Data Sources, and Endpoints.
## Next steps (https://www.tinybird.co/docs/forward/query-data#next-steps)
- Learn how to [query Endpoints](/docs/forward/query-data/api-endpoints).
- Learn about the [Query API](/docs/forward/query-data/sql-api).
- Learn about the [templating language](/docs/forward/core-concepts/templating-language).
- Learn about the [ClickHouse interface](/docs/forward/query-data/clickhouse-interface).
- Learn about [Explorations](/docs/forward/query-data/explorations), [Playgrounds](/docs/forward/query-data/playgrounds), and [Time Series](/docs/forward/query-data/time-series).
---
URL: https://www.tinybird.co/docs/forward/pricing
Last update: 2026-08-07T12:15:37.000Z
Content:
---
title: "Pricing Plans \| Tinybird"
theme-color: "#171612"
description: "Choose the Tinybird plan that fits your needs. Available plans are Free, Developer, SaaS, and Enterprise."
inkeep:version: "forward"
---
# Pricing plans (https://www.tinybird.co/docs/forward/pricing#pricing-plans)
Plans describe what your organization can do in Tinybird, the compute resources at your disposal, and how usage is billed. All plan features, limits, and resources are applied at the organization level and shared across all workspaces within your organization.
To see current prices, see [Pricing](https://www.tinybird.co/pricing).
## Available plans (https://www.tinybird.co/docs/forward/pricing#available-plans)
| Plan | Infrastructure | What you get | When to upgrade |
| --- | --- | --- | --- |
| [Free](/docs/forward/pricing/free) | Shared | - All Tinybird features, with limitations.
- Use Tinybird without time limits.
- You can upgrade at any time. | - You need more resources.
- You're ready for production.
- You want Tinybird support. |
| [Developer](/docs/forward/pricing/developer) | Shared infrastructure without a commitment | - More resources and fewer limits.
- Select compute needs of up to 8 vCPUs.
- vCPU overages for usage exceeding your plan size. | - You need more than 8 vCPUs on shared infrastructure.
- You need credit-based billing with a commitment.
- You need a dedicated cluster. |
| [SaaS](/docs/forward/pricing/saas) | Shared infrastructure with a commitment | - Shared infrastructure, 4+ vCPUs.
- Credit-based billing with a 12-month commitment.
- QPS and active minutes overages. | - You need a dedicated cluster.
- You need private networking. |
| [Enterprise](/docs/forward/pricing/enterprise) | Dedicated | - Your own ClickHouse® cluster.
- No resource sharing.
- No predefined limits; scale with replica SKUs. | - You need to adjust cluster size or add private networking. |
You can upgrade from Free and Developer plans at any time. SaaS and Enterprise plans require [contacting sales](https://www.tinybird.co/contact-us).
## Features included in all plans (https://www.tinybird.co/docs/forward/pricing#features-included-in-all-plans)
- Unlimited seats
- Unlimited source connections
- Unlimited tables, pipes, and endpoints
- Multi-cloud, multi-region support
- RBAC and Workspace administrator controls. See [Organizations](/docs/forward/core-concepts/organizations).
- SOC2 Type II and HIPAA compliance. See [Compliance](/docs/forward/compliance).
## Deployment options (https://www.tinybird.co/docs/forward/pricing#deployment-options)
Plans run on one of two infrastructure types:
- **Shared infrastructure for Free, Developer, and SaaS plans**: You share resources with other customers. Each plan page documents billing and plan-specific limits.
- **Dedicated infrastructure for Enterprise plans**: Your own isolated ClickHouse® cluster. There are no predefined resource limits. Billing depends on replica SKUs and usage. See [Enterprise](/docs/forward/pricing/enterprise).
## On-demand CPUs (https://www.tinybird.co/docs/forward/pricing#on-demand-cpus)
On-demand CPUs cover temporary compute capacity for jobs that need dedicated compute instances. On-demand compute is available on both shared and dedicated infrastructure.
This currently applies to:
- Deployment populate jobs.
- Copy Pipe jobs run with `--on-demand-compute`.
Deployment populates use on-demand compute automatically for large data migrations when one of these thresholds is met:
- Data size exceeds 50 GB.
- Row count exceeds 100 million rows.
Copy Pipe jobs use on-demand compute only when you explicitly run them with `--on-demand-compute`.
On-demand compute uses 64-core instances by default. You can request smaller instances for less resource-intensive workloads. Contact Tinybird support for configuration options.
Billing is based on the actual compute time used, measured in credits per core per minute. Pricing varies by region:
| Region | Price (Credits per core per minute) |
| --- | --- |
| `aws-us-east-1` | 0.0029 |
| `aws-us-west-2` | 0.0029 |
| `aws-eu-west-1` | 0.0032 |
| `aws-eu-central-1` | 0.0035 |
| `aws-ap-east-1` | 0.004 |
| `aws-ap-southeast-2` | 0.0036 |
| `gcp-us-east4` | 0.0027 |
| `gcp-europe-west3` | 0.0031 |
| `gcp-europe-west2` | 0.0031 |
| `gcp-northamerica-northeast2` | 0.0027 |
Example calculations:
- A 2-hour deployment populate in AWS `us-east-1`: 0.0029 credits × 120 minutes × 64 cores = **22.3 credits**.
- A 30-minute Copy Pipe job in AWS `us-east-1`: 0.0029 credits × 30 minutes × 64 cores = **5.57 credits**.
The credits are automatically deducted from your monthly invoice.
## Next steps (https://www.tinybird.co/docs/forward/pricing#next-steps)
- [Free plan](/docs/forward/pricing/free): What's included and how to upgrade.
- [Developer plan](/docs/forward/pricing/developer): Plan sizes, billing, and resize options.
- [SaaS plan](/docs/forward/pricing/saas): Shared infrastructure with 4+ vCPUs and a commitment.
- [Enterprise plan](/docs/forward/pricing/enterprise): Dedicated cluster, replica SKUs, and Standard or Premier support.
- [Limits](/docs/forward/pricing/limits): Platform and operational limits.
---
URL: https://www.tinybird.co/docs/forward/monitoring
Last update: 2026-05-29T14:27:44.000Z
Content:
---
title: "Monitoring \| Tinybird"
theme-color: "#171612"
description: "Learn how to monitor your Tinybird data platform."
inkeep:version: "forward"
---
# Monitoring (https://www.tinybird.co/docs/forward/monitoring#monitoring)
Tinybird is built around the idea of data that changes or grows continuously. Use the built-in Tinybird tools to monitor your data ingestion and API Endpoint processes.
- [Endpoint latency](/docs/forward/monitoring/latency)
- [Endpoint performance](/docs/forward/monitoring/analyze-endpoints-performance)
- [Health checks](/docs/forward/monitoring/health-checks)
- [Kafka logs](/docs/forward/monitoring/kafka-clickhouse-monitoring)
- [Logs](/docs/forward/monitoring/query-log)
- [Organization consumption](/docs/forward/monitoring/organization-consumption)
- [Service Data Sources](/docs/forward/monitoring/service-datasources)
- [System tables](/docs/forward/monitoring/query-log)
- [Workspace jobs](/docs/forward/monitoring/jobs)
---
URL: https://www.tinybird.co/docs/forward/migrations
Last update: 2026-05-29T14:27:44.000Z
Content:
---
title: "Migrations \| Tinybird"
theme-color: "#171612"
description: "Guides for migrating to or within Tinybird."
inkeep:version: "forward"
---
# Migrations (https://www.tinybird.co/docs/forward/migrations#migrations)
Use these guides to migrate your data and workspaces to Tinybird or between Tinybird infrastructure types.
## Available migration guides (https://www.tinybird.co/docs/forward/migrations#available-migration-guides)
- [Migrate from Tinybird Classic](/docs/forward/guides/migrate-from-classic): Migrate your existing Classic workspace to Tinybird Forward.
- [Migrate from Shared to Dedicated Infrastructure](/docs/forward/guides/migrate-from-shared-to-dedicated): Move your data from shared to dedicated infrastructure without disrupting live services.
---
URL: https://www.tinybird.co/docs/forward/ingest-data
Last update: 2026-06-18T10:43:45.000Z
Content:
---
title: "Get data into Tinybird \| Tinybird"
theme-color: "#171612"
description: "Learn how to get data into Tinybird."
inkeep:version: "forward"
---
# Get data into Tinybird (https://www.tinybird.co/docs/forward/ingest-data#get-data-into-tinybird)
Choose the ingestion method that matches how your data is produced and how often it changes:
- **Events**: Send high-frequency JSON or NDJSON events with the [Events API](/docs/forward/ingest-data/events-api).
- **Files**: Append or replace data from local files, remote URLs, CSV, NDJSON, or Parquet with [file ingestion](/docs/forward/ingest-data/local-file).
- **Kafka**: Stream from Kafka-compatible platforms with the [Kafka connector](/docs/forward/ingest-data/connectors/kafka).
- **S3 and GCS**: Ingest files from object storage with the [S3 connector](/docs/forward/ingest-data/connectors/s3) or [GCS connector](/docs/forward/ingest-data/connectors/gcs).
- **DynamoDB**: Mirror an Amazon DynamoDB table into Tinybird with an initial backfill plus Change Data Capture using the [DynamoDB connector](/docs/forward/ingest-data/connectors/dynamodb).
- **External databases and URLs**: Read from external systems with [Table functions](/docs/forward/ingest-data/table-functions) and schedule the sync with Copy Pipes.
- **Ingestion protection**: Learn how Tinybird handles retries, backpressure, and overload in [Ingestion protection](/docs/forward/ingest-data/ingestion-protection).
---
URL: https://www.tinybird.co/docs/forward/guides
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Guides \| Tinybird"
theme-color: "#171612"
description: "Practical guides for building, migrating, optimizing, and monitoring Tinybird projects."
inkeep:version: "forward"
---
# Guides (https://www.tinybird.co/docs/forward/guides#guides)
Use these guides for practical workflows that span Tinybird concepts, resources, and deployment options.
- [Add Tinybird to your Open Source Project](/docs/forward/guides/integrate-with-open-source)
- [Connect Grafana to Tinybird](/docs/forward/guides/connect-grafana)
- [Connect Hex to Tinybird](/docs/forward/guides/connect-hex)
- [Connect Metabase to Tinybird](/docs/forward/guides/connect-metabase)
- [Connect Superset to Tinybird](/docs/forward/guides/connect-superset)
- [Connect DataGrip to Tinybird](/docs/forward/guides/connect-datagrip)
- [Connect DBeaver to Tinybird](/docs/forward/guides/connect-dbeaver)
- [Connect ClickHouse® JS Client to Tinybird](/docs/forward/guides/connect-clickhouse-js)
- [Connect ClickHouse® Go Client to Tinybird](/docs/forward/guides/connect-clickhouse-go)
- [Connect ClickHouse® Python Client to Tinybird](/docs/forward/guides/connect-clickhouse-python)
- [Connect Microsoft Power BI to Tinybird](/docs/forward/guides/connect-powerbi)
- [Connect Redash to Tinybird](/docs/forward/guides/connect-redash)
- [Connect Chartbrew to Tinybird](/docs/forward/guides/connect-chartbrew)
- [Connect Deepnote to Tinybird](/docs/forward/guides/connect-deepnote)
- [Connect Draxlr to Tinybird](/docs/forward/guides/connect-draxlr)
- [Connect Fabi.ai to Tinybird](/docs/forward/guides/connect-fabi)
- [Connect Holistics to Tinybird](/docs/forward/guides/connect-holistics)
- [Connect Luzmo to Tinybird](/docs/forward/guides/connect-luzmo)
- [Connect Mitzu to Tinybird](/docs/forward/guides/connect-mitzu)
- [Connect Tableau to Tinybird](/docs/forward/guides/connect-tableau)
- [Consume API endpoints in Prometheus format](/docs/forward/guides/consume-api-endpoints-in-prometheus-format)
- [Consume APIs in a Notebook](/docs/forward/guides/consume-apis-in-a-notebook)
- [Consume APIs in a Next.js frontend with JWTs](/docs/forward/guides/consume-apis-nextjs)
- [Multi-tenant real-time APIs with Clerk and Tinybird](/docs/forward/guides/multitenant-real-time-apis-with-clerk-and-tinybird)
- [Reliable scheduling with Trigger.dev](/docs/forward/guides/reliable-scheduling-with-trigger)
- [Share API endpoints documentation](/docs/forward/guides/share-endpoint-documentation)
- [Advanced template functions for dynamic API endpoint](/docs/forward/guides/advanced-dynamic-endpoints-functions)
- [Cluster management](/docs/forward/guides/cluster-management)
- [Add a self-managed region manually](/docs/forward/guides/add-self-managed-region-manually)
- [Use the CLI to add a self-managed region](/docs/forward/guides/add-self-managed-region-cli)
- [Test your project](/docs/forward/guides/test-your-project)
- [Evolve Data Sources](/docs/forward/guides/evolve-data-source)
- [Confluent Cloud setup](/docs/forward/guides/confluent-cloud-setup)
- [AWS MSK setup](/docs/forward/guides/aws-msk-setup)
- [Redpanda setup](/docs/forward/guides/redpanda-setup)
- [Migrate from Classic](/docs/forward/guides/migrate-from-classic)
- [Migrate from PostgreSQL](/docs/forward/guides/migrate-from-postgres)
- [Migrate from ClickHouse® self-hosted](/docs/forward/guides/migrate-from-clickhouse)
- [Migrate from Shared to Dedicated Infrastructure](/docs/forward/guides/migrate-from-shared-to-dedicated)
- [Lambda architecture](/docs/forward/guides/lambda-architecture)
- [Deduplication strategies](/docs/forward/guides/deduplication-strategies)
- [Copy and export data](/docs/forward/guides/copy-and-export-data)
- [Monitoring examples](/docs/forward/guides/monitoring-examples)
- [Quarantine Data Sources](/docs/forward/guides/quarantine)
- [Replace and delete data](/docs/forward/guides/replace-and-delete-data)
- [GDPR-compliant data deletion](/docs/forward/guides/gdpr-compliant-data-deletion)
- [Send Auth0 Log Streams to Tinybird](/docs/forward/guides/ingest-auth0-logs)
- [Ingest AWS ELB logs](/docs/forward/guides/ingest-aws-elb-logs)
- [Stream from AWS Kinesis](/docs/forward/guides/ingest-from-aws-kinesis)
- [Ingest from BigQuery using Google Cloud Storage](/docs/forward/guides/ingest-from-bigquery-using-google-cloud-storage)
- [Send Clerk webhooks to Tinybird](/docs/forward/guides/ingest-from-clerk)
- [Ingest CSV files](/docs/forward/guides/ingest-from-csv-files)
- [Send Dub webhooks to Tinybird](/docs/forward/guides/ingest-from-dub)
- [Working with DynamoDB Single-Table Design](/docs/forward/guides/ingest-from-dynamodb-single-table-design)
- [Send GitHub Events to Tinybird](/docs/forward/guides/ingest-from-github)
- [Send GitLab Events to Tinybird](/docs/forward/guides/ingest-from-gitlab)
- [Ingest from Google Pub/Sub](/docs/forward/guides/ingest-from-google-pubsub)
- [Send Knock Events to Tinybird](/docs/forward/guides/ingest-from-knock)
- [Send Mailgun Events to Tinybird](/docs/forward/guides/ingest-from-mailgun)
- [Ingest data from MongoDB](/docs/forward/guides/ingest-from-mongodb)
- [Ingest from OpenTelemetry](/docs/forward/guides/ingest-from-opentelemetry)
- [Send Orb events to Tinybird](/docs/forward/guides/ingest-from-orb)
- [Send PagerDuty events to Tinybird](/docs/forward/guides/ingest-from-pagerduty)
- [Send Resend webhooks to Tinybird](/docs/forward/guides/ingest-from-resend)
- [Stream from RudderStack](/docs/forward/guides/ingest-from-rudderstack)
- [Send Sentry Webhooks to Tinybird](/docs/forward/guides/ingest-from-sentry)
- [Ingest from Snowflake using AWS S3](/docs/forward/guides/ingest-from-snowflake-using-aws-s3)
- [Ingest from Snowflake using Azure Blob Storage](/docs/forward/guides/ingest-from-snowflake-using-azure-blob-storage)
- [Ingest from Snowflake using incremental updates](/docs/forward/guides/ingest-from-snowflake-using-incremental-updates)
- [Send Stripe Events to Tinybird](/docs/forward/guides/ingest-from-stripe)
- [Ingest data using Vector.dev](/docs/forward/guides/ingest-from-vector)
- [Send Vercel Webhooks to Tinybird](/docs/forward/guides/ingest-from-vercel)
- [Send LiteLLM Events to Tinybird](/docs/forward/guides/ingest-litellm)
- [Instrument LLM calls with the Vercel AI SDK and Tinybird](/docs/forward/guides/ingest-vercel-ai-sdk)
- [Send Vercel log drains to Tinybird](/docs/forward/guides/ingest-vercel-logdrains)
- [Ingest with Estuary](/docs/forward/guides/ingest-with-estuary)
- [Postgres CDC with Redpanda Connect](/docs/forward/guides/postgres-cdc-with-redpanda-connect)
---
URL: https://www.tinybird.co/docs/forward/development-workflow
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Development workflow \| Tinybird"
theme-color: "#171612"
description: "Develop in a branch, test your changes, and deploy to your Cloud Workspace."
inkeep:version: "forward"
---
# Development workflow (https://www.tinybird.co/docs/forward/development-workflow#development-workflow)
Tinybird follows a branch-based development workflow. You edit project files in your local repo that is connected to Git, test changes in Tinybird Local or a Cloud Branch, and deploy a new version to your Cloud Workspace.
Use this workflow when you change Data Source schemas, iterate on Pipe logic, add Endpoints, or validate connector ingestion before it reaches production.
## Choose an environment (https://www.tinybird.co/docs/forward/development-workflow#choose-an-environment)
| Environment | Use it for | Example |
| --- | --- | --- |
| [Tinybird Local](/docs/forward/development-workflow/local-development) | Fast local iteration and CI checks with fixture data. | Change a Pipe query, run `tb build`, and call the local Endpoint until it returns the expected result. |
| [Cloud Branches](/docs/forward/development-workflow/using-cloud-branches) | Testing with production-like data without changing production. | Check out a Git branch, run `tb build`, change a Data Source schema, and query the branch before deploying. |
| [Staging deployments](/docs/forward/development-workflow/manual-deployment#staging-deployments) | Validating the deployment that can become live. | Run `tb --cloud deployment create --wait`, write compatible test events to staging, then promote. |
Local and branch environments are for development. Deployments are for moving a validated project version toward production.
1
### Develop in a branch (https://www.tinybird.co/docs/forward/development-workflow#develop-in-a-branch)
Choose the development loop that fits the data you need:
- [Local development](/docs/forward/development-workflow/local-development): Run Tinybird Local with fixture data and local APIs.
- [Cloud Branches](/docs/forward/development-workflow/using-cloud-branches): Use a Cloud environment for production-like data, connector validation, preview apps, and pull request review.
2
### Test your changes (https://www.tinybird.co/docs/forward/development-workflow#test-your-changes)
Test Tinybird changes as part of your app workflow, the same way you test the rest of your app. Test locally with fixtures, or test against a Cloud Branch when you need production-like data. You can also validate your data project independently with fixture files, data quality tests, and deployment checks. See [Test your project](/docs/forward/guides/test-your-project).
3
### Deploy to your Cloud Workspace (https://www.tinybird.co/docs/forward/development-workflow#deploy-to-your-cloud-workspace)
Push your changes to Tinybird Cloud with automatic schema migrations. Run a deployment check first when the change affects Data Source schemas, Materialized Views, engine settings, or connector configuration.
- [Examples](/docs/forward/development-workflow/examples): See concrete workflows for Data Source evolution, Pipe iteration, SDK projects, and deployments.
- [Evolve Data Sources](/docs/forward/guides/evolve-data-source): Understand how schema, engine, and Materialized View changes are migrated during deployments.
- [Manual deployment](/docs/forward/development-workflow/manual-deployment): Deploy directly from the CLI.
- [CI/CD](/docs/forward/development-workflow/cicd): Automate testing and deployment with GitHub Actions or GitLab CI.
## Examples (https://www.tinybird.co/docs/forward/development-workflow#examples)
See [Development workflow examples](/docs/forward/development-workflow/examples) for common loops with datafiles, the TypeScript SDK, and the Python SDK.
## Core concepts (https://www.tinybird.co/docs/forward/development-workflow#core-concepts)
- [Tinybird Local](/docs/forward/core-concepts/tinybird-local) explains the local runtime, services, installation, Docker usage, persistence, and local Tokens.
- [Branches](/docs/forward/core-concepts/branches) explains the Branch resource model and data isolation semantics.
---
URL: https://www.tinybird.co/docs/forward/dev-reference
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Tinybird developer reference \| Tinybird"
theme-color: "#171612"
description: "The Tinybird developer reference covers all available commands and tools for developers."
inkeep:version: "forward"
---
# Developer reference (https://www.tinybird.co/docs/forward/dev-reference#developer-reference)
Use this reference to understand the commands and tools available in Tinybird, including their features, options, and arguments.
## Config files (https://www.tinybird.co/docs/forward/dev-reference#config-files)
- Configure Tinybird projects: [Config files](/docs/forward/dev-reference/config-files)
## Tinybird CLI (https://www.tinybird.co/docs/forward/dev-reference#tinybird-cli)
- Learn the datafile formats: [Datafiles](/docs/forward/dev-reference/datafiles)
- Browse CLI commands: [CLI commands](/docs/forward/dev-reference/commands)
## TypeScript SDK (https://www.tinybird.co/docs/forward/dev-reference#typescript-sdk)
- Define resources in TypeScript: [TypeScript SDK resources](/docs/forward/dev-reference/typescript-sdk-resources)
- Browse SDK CLI commands: [TypeScript SDK CLI commands](/docs/forward/dev-reference/commands/typescript-sdk-cli)
## Python SDK (https://www.tinybird.co/docs/forward/dev-reference#python-sdk)
- Define resources in Python: [Python SDK resources](/docs/forward/dev-reference/python-sdk-resources)
- Browse SDK CLI commands: [Python SDK CLI commands](/docs/forward/dev-reference/commands/python-sdk-cli)
- Follow the full setup flow: [Quick start: Python SDK](/docs/forward/quickstarts/python-sdk)
---
URL: https://www.tinybird.co/docs/forward/core-concepts
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Core concepts of Tinybird \| Tinybird"
theme-color: "#171612"
description: "Learn the core concepts of Tinybird, how it works, and how to use it."
inkeep:version: "forward"
---
# Core concepts of Tinybird (https://www.tinybird.co/docs/forward/core-concepts#core-concepts-of-tinybird)
Tinybird gives you the tooling and infrastructure you need to ship analytics features in your app:
- An OLAP database (ClickHouse®)
- An API gateway
- A real-time ingestion system
- An auth system
- Tooling to build, iterate, and deploy data projects
Tinybird is built around several core concepts that abstract the underlying infrastructure and data storage. To make the most of Tinybird, it's important to understand these core concepts.
## Develop anywhere, deploy to Tinybird Cloud (https://www.tinybird.co/docs/forward/core-concepts#develop-anywhere-deploy-to-tinybird-cloud)
Tinybird is a platform for building analytics features in your app. You can build with:
- Tinybird CLI (datafiles)
- TypeScript SDK (typed resources)
- Python SDK (typed resources)
There are three main environments: Tinybird Cloud for production workloads, Tinybird Local for local development, and Tinybird Branches for shared preview environments.
- Tinybird Cloud is the hosted platform (see available [regions](/docs/api-reference#regions-and-endpoints)) where your production workspace lives.
- [Tinybird Local](/docs/forward/core-concepts/tinybird-local) is a local runtime for development and testing.
- Tinybird Branches are cloud environments for development, collaboration, and pre-production validation.
The typical workflow is as follows:
1. Define your resources using [datafiles](/docs/forward/dev-reference/datafiles) or the TypeScript SDK.
2. Iterate and test [locally](/docs/forward/development-workflow/local-development) or in [Cloud Branches](/docs/forward/development-workflow/using-cloud-branches).
3. [Deploy your data project](/docs/forward/core-concepts/deployments) to Tinybird Cloud so your endpoints and ingest pipelines are live. Strongly recommended via [CI/CD](/docs/forward/development-workflow/cicd).
4. Use [Tinybird Cloud](https://cloud.tinybird.co/) to inspect data, manage deployments, and monitor usage.
The following diagram illustrates the development and usage flow, from local development or branches to deployment in Tinybird Cloud.
## Your projects live in workspaces (https://www.tinybird.co/docs/forward/core-concepts#your-projects-live-in-workspaces)
A [workspace](/docs/forward/core-concepts/workspaces) contains the resources, data, and state of your Tinybird project. You can have as many workspaces as you need and invite users to collaborate.
At a minimum, a workspace contains a data source (a database table). You write data to it and read data from it. The following diagram illustrates the relationship between resources in a workspace.
## Data enters through data sources (https://www.tinybird.co/docs/forward/core-concepts#data-enters-through-data-sources)
[Data Sources](/docs/forward/core-concepts/data-sources) are where you ingest and store data in Tinybird. All your data lives inside a data source, and you write SQL queries against data sources. You can bulk upload or stream data into a data source.
You can bring data in from the following sources:
- Events sent to the [Events API](/docs/forward/ingest-data/events-api).
- Events streamed from Kafka. See [Ingest data from Kafka](/docs/forward/ingest-data/connectors/kafka).
- Files in your local file system. See [Ingest data from files](/docs/forward/ingest-data/local-file).
- Files in your cloud storage bucket. See [Connections](/docs/forward/core-concepts/connections).
Data sources can be defined in datafiles or with the SDKs. See [Data Sources](/docs/forward/core-concepts/data-sources) for more information.
## Use pipes to process your data (https://www.tinybird.co/docs/forward/core-concepts#use-pipes-to-process-your-data)
[Pipes](/docs/forward/core-concepts/pipes) are how you write SQL logic in Tinybird. They're a collection of one or more SQL queries chained together, or nodes. Pipes let you break larger queries down into smaller queries that are easier to read.
With pipes you can:
- Publish Endpoints.
- Create optimized Data Sources with Materialized Views or Copy Pipes.
- Send data to Cloud Storage via Sinks.
- Create Views.
[Pipes](/docs/forward/core-concepts/pipes) can be defined in datafiles or with the SDKs. See [Pipes](/docs/forward/core-concepts/pipes) for more information.
## Query your data (https://www.tinybird.co/docs/forward/core-concepts#query-your-data)
When your data is ready to be consumed, expose it through Endpoints or query it directly from Tinybird Cloud, SQL clients, BI tools, and agents.
To query from outside Tinybird:
- [Endpoints](/docs/forward/core-concepts/api-endpoints), which you can call using custom Query Parameters from any app.
- [Query API](/docs/api-reference/query-api) for running SQL directly against Data Sources and Pipes over HTTP.
- [ClickHouse interface](/docs/forward/query-data/clickhouse-interface) for SQL clients and BI tools.
- [MCP server](/docs/forward/query-data/mcp) for agent and tool integrations.
To query inside Tinybird Cloud:
- [Playgrounds](/docs/forward/query-data/playgrounds) for writing and testing SQL interactively.
- [Explorations](/docs/forward/query-data/explorations) for AI-assisted data exploration.
- [Time Series](/docs/forward/query-data/time-series) for visualizing time-based metrics.
## Tokens protect your data (https://www.tinybird.co/docs/forward/core-concepts#tokens-protect-your-data)
[Tokens](/docs/forward/core-concepts/tokens) are how you authenticate and control access to your Tinybird resources. Without tokens, nobody can send data to your data sources or query your endpoints.
There are two types of tokens:
- **[Static tokens](/docs/forward/core-concepts/static-tokens)**: Permanent tokens for backend integrations. Use them to send data, manage resources via CLI, or read data directly from your backend applications.
- **[JSON Web Tokens, JWTs](/docs/forward/core-concepts/jwt)**: Short-lived tokens. Use them for browser access and multi-tenant applications where each user or tenant needs scoped endpoint access, optional filtering, and per-user rate limits.
When you create a workspace, Tinybird automatically creates default tokens for you, including:
- **Workspace admin token**: Full access to everything in your workspace
- **Your personal admin token**: Stored in your `.tinyb` file when you run `tb login`
For development, you can use these admin tokens. For production, create specific tokens with limited permissions (scopes) for security.
### Token workflow (https://www.tinybird.co/docs/forward/core-concepts#token-workflow)
1. **Development**: Use admin tokens to build and test your project locally
2. **Production**: Create resource-specific tokens with minimal required permissions
3. **Frontend access**: Generate JWTs from your backend for browser-based queries
See [Tokens](/docs/forward/core-concepts/tokens) for detailed token management.
## Templates make resources dynamic (https://www.tinybird.co/docs/forward/core-concepts#templates-make-resources-dynamic)
Use the [templating language](/docs/forward/core-concepts/templating-language) to add typed parameters, conditional SQL, settings, and secrets to resources defined as code. Use templates to create parameterized Endpoints, run Copy Pipes with runtime values, and keep connector credentials out of source control.
## To go live, create and promote a deployment (https://www.tinybird.co/docs/forward/core-concepts#to-go-live-create-and-promote-a-deployment)
After you develop your project, you can deploy it to Tinybird Cloud. This creates a deployment, which is a version of your project resources and data running on local or cloud infrastructure.
When you're ready to go live, you can promote your deployment in Tinybird Cloud. This makes your deployment available to your users. In other words, the deployment is live.
See [Deployments](/docs/forward/core-concepts/deployments) for more information.
## Monitor with service data sources (https://www.tinybird.co/docs/forward/core-concepts#monitor-with-service-data-sources)
Tinybird exposes built-in data sources for monitoring usage and performance:
- **`tinybird.*` service data sources** for workspace-level metrics and logs.
- **`system.*` tables** for ClickHouse system-level information.
- **`organization.*` service data sources** for org-wide usage and billing signals.
See [Service data sources](/docs/forward/monitoring/service-datasources) and [Organizations](/docs/forward/core-concepts/organizations) for details.
---
URL: https://www.tinybird.co/docs/forward/copy-export-data
Last update: 2026-05-29T14:27:44.000Z
Content:
---
title: "Copy and export data \| Tinybird"
theme-color: "#171612"
description: "Move data inside Tinybird with Copy Pipes and export query results to Kafka, S3, or GCS with Sinks."
inkeep:version: "forward"
---
# Copy and export data (https://www.tinybird.co/docs/forward/copy-export-data#copy-and-export-data)
After data is in Tinybird, you often need to reshape it, store query results, or send processed data to another system. Use this section when you want to move data after ingestion.
## Move data internally (https://www.tinybird.co/docs/forward/copy-export-data#move-data-internally)
Use [Copy Pipes](/docs/forward/core-concepts/copy-pipes) to capture the result of a Pipe at a specific point in time and write it to a target Data Source. Copy Pipes can run on a schedule or on demand.
Copy Pipes are useful when you need to:
- Build scheduled snapshots of source data.
- Backfill or rebuild a derived Data Source.
- Deduplicate data with a replace strategy.
- Run heavy transformations outside request-time query paths.
- Test a new shape before replacing an existing Data Source.
For continuously updated derived Data Sources, use [Materialized Views](/docs/forward/core-concepts/materialized-views). Materialized Views process new rows as they arrive. Copy Pipes rerun a query and write the result when the copy job executes.
## Sinks (https://www.tinybird.co/docs/forward/copy-export-data#sinks)
Use [Sinks](/docs/forward/core-concepts/sinks) to export query results from Tinybird to external systems. A Sink is a Pipe with `TYPE SINK` and destination-specific export settings.
Tinybird supports these Sink destinations:
- [Kafka Sink](/docs/forward/copy-export-data/kafka-sink): Export events to a Kafka topic.
- [S3 Sink](/docs/forward/copy-export-data/s3-sink): Export files to an S3 bucket.
- [GCS Sink](/docs/forward/copy-export-data/gcs-sink): Export files to a Google Cloud Storage bucket.
Use Sinks when another system needs processed Tinybird data, for example to feed a data lake, publish events to a streaming platform, or deliver scheduled extracts to customers or partners.
## Choose the right option (https://www.tinybird.co/docs/forward/copy-export-data#choose-the-right-option)
| Need | Use |
| --- | --- |
| Store query results in another Tinybird Data Source | [Copy Pipes](/docs/forward/core-concepts/copy-pipes) |
| Export query results to Kafka, S3, or GCS | [Sinks](/docs/forward/core-concepts/sinks) |
| Keep a derived Data Source updated as rows are ingested | [Materialized Views](/docs/forward/core-concepts/materialized-views) |
| Query data without storing the result | [API Endpoints](/docs/forward/core-concepts/api-endpoints) or [Query API](/docs/forward/query-data/sql-api) |
---
URL: https://www.tinybird.co/docs/forward/compliance
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Compliance and certifications \| Tinybird"
theme-color: "#171612"
description: "Tinybird is committed to the highest data security and safety. See what compliance certifications are available."
inkeep:version: "forward"
---
# Compliance and certifications (https://www.tinybird.co/docs/forward/compliance#compliance-and-certifications)
Data security and privacy are paramount in today's digital landscape. Tinybird's commitment to protecting your sensitive information is backed by the following compliance certifications, which demonstrate compliance with rigorous industry standards for data security, privacy, and operational excellence.
## SOC 2 Type II (https://www.tinybird.co/docs/forward/compliance#soc-2-type-ii)
Tinybird has obtained a SOC 2 Type II certification, in accordance with attestation standards established by the American Institute of Certified Public Accountants (AICPA), that are relevant to security, availability, processing integrity, confidentiality, and privacy for Tinybird's real-time platform for user-facing analytics. Compliance is monitored continually—with reports published annually—to confirm the robustness of Tinybird's data security. This independent assessment provides Tinybird users with assurance that their sensitive information is being handled responsibly and securely.
## HIPAA (https://www.tinybird.co/docs/forward/compliance#hipaa)
Tinybird supports its customers' Health Insurance Portability and Accountability Act (HIPAA) compliance efforts by offering Business Associate Agreements (BAAs). Additionally, Tinybird's offering allows customers to process their data constituting personal health information (PHI) in AWS, Azure, or Google Cloud—entities which themselves have entered into BAAs with Tinybird.
## Trust center (https://www.tinybird.co/docs/forward/compliance#trust-center)
To learn more about Tinybird security controls and certifications, visit the [Tinybird Trust Center](https://trust.tinybird.co/).
---
URL: https://www.tinybird.co/docs/forward/quickstarts/typescript-sdk
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Quickstart: TypeScript SDK \| Tinybird"
theme-color: "#171612"
description: "Build Tinybird projects using the TypeScript SDK and typed resources."
inkeep:version: "forward"
---
# Quickstart: TypeScript SDK (https://www.tinybird.co/docs/forward/quickstarts/typescript-sdk#quickstart-typescript-sdk)
This quickstart walks through creating a Tinybird project using the TypeScript SDK. You define Data Sources and Endpoints in TypeScript, sync them to Tinybird, and use a typed client for ingest and queries.
Looking for other ways to start? See [Quickstarts](/docs/forward/quickstarts).
If you use a coding agent like Cursor, Claude Code, Amp, or Open Code, install Tinybird agent skills so your agent understands Tinybird project structure and workflows:
```
npx skills add tinybirdco/tinybird-agent-skills
```
## Before you begin (https://www.tinybird.co/docs/forward/quickstarts/typescript-sdk#before-you-begin)
To get started, you need the following:
- Node.js 20 LTS or later
- TypeScript 4.9 or later
## Install the SDK (https://www.tinybird.co/docs/forward/quickstarts/typescript-sdk#install-the-sdk)
Install the SDK globally so you can run the `tinybird` CLI directly.
- pnpm
- yarn
- npm
- bun
```
pnpm add -g @tinybirdco/sdk
```
Alternatively, add it to your repository:
- pnpm
- yarn
- npm
- bun
```
pnpm add @tinybirdco/sdk
```
To follow the quickstart, install it in the repository so that commands work exactly as they are in the examples.
## Initialize a Tinybird TypeScript project (https://www.tinybird.co/docs/forward/quickstarts/typescript-sdk#initialize-a-tinybird-typescript-project)
Create a new directory and initialize the project:
```
mkdir tinybird-typescript-quickstart
cd tinybird-typescript-quickstart
git init
pnpm add @tinybirdco/sdk
npx tinybird init
```
During initialization, the CLI authenticates you against Tinybird Cloud and can optionally add CI/CD workflow files.
This creates:
- `tinybird.config.json` configuration file
- `lib/tinybird.ts` for Data Sources, Pipes, Connectors, and Client definitions
- `.env.local` for authentication against your Tinybird Cloud Workspace
## Check your config file (https://www.tinybird.co/docs/forward/quickstarts/typescript-sdk#check-your-config-file)
In `tinybird.config.json`, the default `"include"` is `lib/tinybird.ts` to generate resources from that file. The default `"devMode"` is `"branch"` to develop in Tinybird Cloud Branches, although you can use `"local"` too with Tinybird Local.
```
{
"include": [
"lib/tinybird.ts"
],
"token": "${TINYBIRD_TOKEN}",
"baseUrl": "${TINYBIRD_URL}",
"devMode": "branch"
}
```
## Check your Tinybird token (https://www.tinybird.co/docs/forward/quickstarts/typescript-sdk#check-your-tinybird-token)
Create a `.env.local` file with your Tinybird token if it doesn't exist yet:
```
TINYBIRD_TOKEN=p.your_token_here
TINYBIRD_URL=https://api.tinybird.co
```
The CLI reads `.env.local` automatically.
## Inspect resource definitions (https://www.tinybird.co/docs/forward/quickstarts/typescript-sdk#inspect-resource-definitions)
### Data sources (https://www.tinybird.co/docs/forward/quickstarts/typescript-sdk#data-sources)
Check the Data Source definition in `lib/tinybird.ts`. The schema and engine stay the same regardless of how you ingest data. The difference is how data arrives: via the Events API, from a Kafka topic, or from an S3 bucket.
- Events
- Kafka
- S3
```
import { defineDatasource, engine, t, type InferRow } from "@tinybirdco/sdk";
export const pageViews = defineDatasource("page_views", {
description: "Page view tracking data",
schema: {
timestamp: t.dateTime(),
session_id: t.string(),
pathname: t.string(),
referrer: t.string().nullable(),
},
engine: engine.mergeTree({
sortingKey: ["pathname", "timestamp"],
}),
});
export type PageViewsRow = InferRow;
```
### API Endpoints (https://www.tinybird.co/docs/forward/quickstarts/typescript-sdk#api-endpoints)
Check the Endpoint definition in `lib/tinybird.ts`:
```
import {
defineEndpoint,
node,
p,
t,
type InferParams,
type InferOutputRow,
} from "@tinybirdco/sdk";
export const topPages = defineEndpoint("top_pages", {
description: "Get the most visited pages",
params: {
start_date: p.dateTime().describe("Start of date range"),
end_date: p.dateTime().describe("End of date range"),
limit: p.int32().optional(10).describe("Number of results"),
},
nodes: [
node({
name: "aggregated",
sql: `
SELECT
pathname,
count() AS views
FROM page_views
WHERE timestamp >= {{DateTime(start_date)}}
AND timestamp <= {{DateTime(end_date)}}
GROUP BY pathname
ORDER BY views DESC
LIMIT {{Int32(limit, 10)}}
`,
}),
],
output: {
pathname: t.string(),
views: t.uint64(),
},
});
export type TopPagesParams = InferParams;
export type TopPagesOutput = InferOutputRow;
```
### Typed client (https://www.tinybird.co/docs/forward/quickstarts/typescript-sdk#typed-client)
Check the client in `lib/tinybird.ts`:
```
import { Tinybird } from "@tinybirdco/sdk";
export const tinybird = new Tinybird({
datasources: { pageViews },
pipes: { topPages },
});
```
## Build the project (https://www.tinybird.co/docs/forward/quickstarts/typescript-sdk#build-the-project)
Check out a new git branch:
```
git add tinybird.config.json && git commit -m "Initial commit" && git checkout -b tinybird_intro
```
Run a build to create your resources in your Tinybird development environment:
```
npx tinybird build
```
If you want your changes to sync automatically while you work, run dev mode instead:
```
npx tinybird dev
```
### About the development flow (https://www.tinybird.co/docs/forward/quickstarts/typescript-sdk#about-the-development-flow)
Your default target (local or branch) is set in `tinybird.config.json`. You can override it with flags on the command line.
`tinybird build` syncs to a Tinybird branch, either in Cloud or in Tinybird Local.
You can also pass a flag to explicitly use the desired development environment.
```
npx tinybird build --local # Tinybird Local Branches
# or
npx tinybird build --branch # Tinybird Cloud Branches
```
If you haven't installed Tinybird Local yet, see [Tinybird Local](/docs/forward/core-concepts/tinybird-local).
When you're ready to promote changes:
- Use `tinybird build` to push to a branch (good for review or CI).
- Use CI/CD to deploy to your main Tinybird Cloud Workspace. You can also use `tinybird deploy` directly.
#### Git integration (https://www.tinybird.co/docs/forward/quickstarts/typescript-sdk#git-integration)
You may have noticed that the quickstart example had a `git checkout -b tinybird_intro`. If Tinybird detects you are in a Git repository, it creates a branch with the git branch name in your Tinybird development environment.
## Ingest data and query (https://www.tinybird.co/docs/forward/quickstarts/typescript-sdk#ingest-data-and-query)
### Ingest (https://www.tinybird.co/docs/forward/quickstarts/typescript-sdk#ingest)
Use the typed client to send events via the Events API:
```
import { tinybird } from "./lib/tinybird";
await tinybird.pageViews.ingest({
timestamp: new Date(),
pathname: "/home",
session_id: "abc123",
referrer: "https://google.com",
});
```
If you configured a Kafka or S3 connector, data is ingested automatically after you deploy. You don't need to call `ingest` manually.
### Query (https://www.tinybird.co/docs/forward/quickstarts/typescript-sdk#query)
Query the Endpoint from your app code:
```
const result = await tinybird.topPages.query({
start_date: new Date("2026-01-01"),
end_date: new Date(),
limit: 5,
});
result.data.forEach((row) => {
console.log(row.pathname, row.views);
});
```
## Iterate in a branch (https://www.tinybird.co/docs/forward/quickstarts/typescript-sdk#iterate-in-a-branch)
Add a materialized view in another branch, test the app against that branch, and deploy to Tinybird Cloud when you're finished.
```
git checkout -b materialization
```
Edit `lib/tinybird.ts` to add a new Data Source, a new materialized view, and update the Endpoint to read from the materialized view.
```
import {
defineDatasource,
defineEndpoint,
defineMaterializedView,
engine,
node,
p,
t,
} from "@tinybirdco/sdk";
export const pageViews = defineDatasource("page_views", {
description: "Page view tracking data",
schema: {
timestamp: t.dateTime(),
session_id: t.string(),
pathname: t.string(),
referrer: t.string().nullable(),
},
engine: engine.mergeTree({
sortingKey: ["pathname", "timestamp"],
}),
});
export const dailyPageViews = defineDatasource("daily_page_views_ds", {
description: "Daily page view counts populated by a materialized view",
schema: {
date: t.date(),
pathname: t.string(),
views: t.simpleAggregateFunction("sum", t.uint64()),
},
engine: engine.aggregatingMergeTree({
sortingKey: ["date", "pathname"],
}),
jsonPaths: false,
});
export const dailyPageViewsMv = defineMaterializedView("daily_page_views_mv", {
datasource: dailyPageViews,
nodes: [
node({
name: "aggregate",
sql: `
SELECT
toDate(timestamp) AS date,
pathname,
count() AS views
FROM page_views
GROUP BY date, pathname
`,
}),
],
});
export const topPages = defineEndpoint("top_pages", {
description: "Get the most visited pages from MV-backed datasource",
params: {
start_date: p.dateTime(),
end_date: p.dateTime(),
limit: p.int32().optional(10),
},
nodes: [
node({
name: "aggregated",
sql: `
SELECT
pathname,
sum(views) AS views
FROM daily_page_views_ds
WHERE date >= toDate({{DateTime(start_date)}})
AND date <= toDate({{DateTime(end_date)}})
GROUP BY pathname
ORDER BY views DESC
LIMIT {{Int32(limit, 10)}}
`,
}),
],
output: {
pathname: t.string(),
views: t.uint64(),
},
});
```
Build against the branch:
```
npx tinybird build
```
Once validated, you can deploy to Tinybird Cloud with `npx tinybird deploy` if you haven't set up a CI/CD script.
## Next steps (https://www.tinybird.co/docs/forward/quickstarts/typescript-sdk#next-steps)
- Review the SDK repository for advanced usage. See [tinybird-sdk-typescript](https://github.com/tinybirdco/tinybird-sdk-typescript).
- Review the TypeScript SDK CLI reference. See [TypeScript SDK CLI commands](/docs/forward/dev-reference/commands/typescript-sdk-cli).
- Learn about core Tinybird concepts. See [Core concepts](/docs/forward/core-concepts).
- Configure deployments and branches. See [Development workflow](/docs/forward/development-workflow).
---
URL: https://www.tinybird.co/docs/forward/quickstarts/python-sdk
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Quickstart: Python SDK \| Tinybird"
theme-color: "#171612"
description: "Build Tinybird projects with the Python SDK and use the bundled CLI commands."
inkeep:version: "forward"
---
# Quickstart: Python SDK (https://www.tinybird.co/docs/forward/quickstarts/python-sdk#quickstart-python-sdk)
This quickstart walks through creating a Tinybird project with the Python SDK. You define Data Sources and Endpoints in Python, use the CLI to build and deploy, and call your Endpoint from app code.
Looking for other ways to start? See [Quickstarts](/docs/forward/quickstarts).
If you use a coding agent like Cursor, Claude Code, Amp, or Open Code, install Tinybird agent skills so your agent understands Tinybird project structure and workflows:
```
npx skills add tinybirdco/tinybird-agent-skills
```
## Before you begin (https://www.tinybird.co/docs/forward/quickstarts/python-sdk#before-you-begin)
To get started, you need:
- Python 3.11, 3.12, or 3.13
- A Tinybird Workspace
- git
- `uv` (recommended [Python package manager](https://docs.astral.sh/uv/))
Python 3.14 is not supported yet for this workflow. The Python SDK relies on the Tinybird CLI under the hood, and current CLI support covers Python 3.11 to 3.13.
## Initialize a Python project and add the SDK (https://www.tinybird.co/docs/forward/quickstarts/python-sdk#initialize-a-python-project-and-add-the-sdk)
Create a new directory and initialize a Python SDK project:
```
mkdir tinybird-python-quickstart
cd tinybird-python-quickstart
git init
uv init --name tinybird-sdk-python-sample --python 3.13
uv add tinybird-sdk python-dotenv
uv run tinybird init --folder src/tinybird/
```
This creates:
- `tinybird.config.json` with an `include` path for Python resources
- `.env.local` for your Tinybird credentials
## Check your config file (https://www.tinybird.co/docs/forward/quickstarts/python-sdk#check-your-config-file)
In `tinybird.config.json`, `"include"` points to the folder with your Python resource definitions. The default `"devMode"` is `"branch"` to develop in Tinybird Cloud Branches, although you can use `"local"` too with Tinybird Local.
## Check your Tinybird token (https://www.tinybird.co/docs/forward/quickstarts/python-sdk#check-your-tinybird-token)
Create a `.env.local` file with your Tinybird token if it doesn't exist yet:
```
TINYBIRD_TOKEN=p.your_token_here
TINYBIRD_URL=https://api.tinybird.co
```
The CLI reads `.env.local` automatically.
## Define resources in Python (https://www.tinybird.co/docs/forward/quickstarts/python-sdk#define-resources-in-python)
Create `src/tinybird/tinybird_resources.py`. The schema and engine stay the same regardless of how you ingest data. The difference is how data arrives: via the Events API, from a Kafka topic, or from an S3 bucket.
### Data source (https://www.tinybird.co/docs/forward/quickstarts/python-sdk#data-source)
- Events
- Kafka
- S3
```
from tinybird_sdk import define_datasource, engine, t
page_views = define_datasource(
"page_views",
{
"description": "Page view tracking data",
"schema": {
"timestamp": t.date_time(),
"session_id": t.string(),
"pathname": t.string(),
"referrer": t.string().nullable(),
},
"engine": engine.merge_tree(
{
"sorting_key": ["pathname", "timestamp"],
}
),
},
)
```
### Endpoint (https://www.tinybird.co/docs/forward/quickstarts/python-sdk#endpoint)
The endpoint reads from the Data Source regardless of how data is ingested:
```
from tinybird_sdk import define_endpoint, node, p, t
top_pages = define_endpoint(
"top_pages",
{
"description": "Get the most visited pages",
"params": {
"start_date": p.date_time(),
"end_date": p.date_time(),
"limit": p.int32().optional(10),
},
"nodes": [
node(
{
"name": "aggregated",
"sql": """
SELECT pathname, count() AS views
FROM page_views
WHERE timestamp >= {{DateTime(start_date)}}
AND timestamp <= {{DateTime(end_date)}}
GROUP BY pathname
ORDER BY views DESC
LIMIT {{Int32(limit, 10)}}
""",
}
)
],
"output": {
"pathname": t.string(),
"views": t.uint64(),
},
},
)
```
Create `src/tinybird/client.py`:
```
import os
from tinybird_sdk import Tinybird
from src.tinybird.tinybird_resources import page_views, top_pages
tinybird = Tinybird(
{
"datasources": {"page_views": page_views},
"pipes": {"top_pages": top_pages},
"base_url": os.getenv("TINYBIRD_URL", "https://api.tinybird.co"),
"token": os.getenv("TINYBIRD_TOKEN"),
}
)
```
## Build the project (https://www.tinybird.co/docs/forward/quickstarts/python-sdk#build-the-project)
Commit your project files and check out a new git branch:
- `pyproject.toml` — project metadata and dependencies
- `uv.lock` — pinned dependency versions (like `package-lock.json`)
- `.python-version` — ensures consistent Python version
- `.gitignore`
- Your source code and tests
```
git add pyproject.toml uv.lock .python-version .gitignore src/ tinybird.config.json main.py
git commit -m "Initial commit"
git checkout -b tinybird_intro
```
Run a build to create your resources in your Tinybird development environment:
```
uv run tinybird build
```
#### Git integration (https://www.tinybird.co/docs/forward/quickstarts/python-sdk#git-integration)
You may have noticed that the quickstart example had a `git checkout -b tinybird_intro`. If Tinybird detects you are in a Git repository, it creates a branch with the git branch name in your Tinybird development environment.
## Ingest data and query (https://www.tinybird.co/docs/forward/quickstarts/python-sdk#ingest-data-and-query)
### Ingest (https://www.tinybird.co/docs/forward/quickstarts/python-sdk#ingest)
Use the runtime client to send events via the Events API:
```
from datetime import datetime, timezone
from src.tinybird.client import tinybird
now = datetime.now(timezone.utc).isoformat(timespec="milliseconds")
tinybird.page_views.ingest(
{
"timestamp": now,
"session_id": "abc123",
"pathname": "/home",
"referrer": "https://google.com",
}
)
```
If you configured a Kafka or S3 connector, data is ingested automatically after you deploy. You don't need to call `ingest` manually.
### Query (https://www.tinybird.co/docs/forward/quickstarts/python-sdk#query)
Query the Endpoint from your app code:
```
result = tinybird.top_pages.query(
{
"start_date": "2026-01-01 00:00:00",
"end_date": now,
"limit": 5,
}
)
for row in result["data"]:
print(row["pathname"], row["views"])
```
## Iterate in a branch (https://www.tinybird.co/docs/forward/quickstarts/python-sdk#iterate-in-a-branch)
Add a materialized view in another branch, test the app against that branch, and deploy to Tinybird Cloud when you're finished.
```
git checkout -b materialization
```
Edit `tinybird_resources.py` to add a new Data Source, a new materialized view, and update the Endpoint to read from the materialized view.
```
from tinybird_sdk import (
define_datasource,
define_endpoint,
define_materialized_view,
engine,
node,
p,
t,
)
page_views = define_datasource(
"page_views",
{
"description": "Page view tracking data",
"schema": {
"timestamp": t.date_time(),
"session_id": t.string(),
"pathname": t.string(),
"referrer": t.string().nullable(),
},
"engine": engine.merge_tree(
{
"sorting_key": ["pathname", "timestamp"],
}
),
},
)
daily_page_views_ds = define_datasource(
"daily_page_views_ds",
{
"description": "Daily page view counts populated by a materialized view",
"schema": {
"date": t.date(),
"pathname": t.string(),
"views": t.simple_aggregate_function("sum", t.uint64()),
},
"engine": engine.aggregating_merge_tree(
{
"sorting_key": ["date", "pathname"],
}
),
"json_paths": False,
},
)
daily_page_views_mv = define_materialized_view(
"daily_page_views_mv",
{
"description": "Materialized rollup of page views by day and pathname",
"datasource": daily_page_views_ds,
"nodes": [
node(
{
"name": "aggregate",
"sql": """
SELECT
toDate(timestamp) AS date,
pathname,
count() AS views
FROM page_views
GROUP BY date, pathname
""",
}
)
],
},
)
top_pages = define_endpoint(
"top_pages",
{
"description": "Get the most visited pages from MV-backed datasource",
"params": {
"start_date": p.date_time(),
"end_date": p.date_time(),
"limit": p.int32().optional(10),
},
"nodes": [
node(
{
"name": "aggregated",
"sql": """
SELECT
pathname,
sum(views) AS views
FROM daily_page_views_ds
WHERE date >= toDate({{DateTime(start_date)}})
AND date <= toDate({{DateTime(end_date)}})
GROUP BY pathname
ORDER BY views DESC
LIMIT {{Int32(limit, 10)}}
""",
}
)
],
"output": {
"pathname": t.string(),
"views": t.uint64(),
},
},
)
```
Build against the branch:
```
uv run tinybird build
```
Once validated, you can deploy to Tinybird Cloud with `uv run tinybird deploy` if you haven't set up a CI/CD script.
## Next steps (https://www.tinybird.co/docs/forward/quickstarts/python-sdk#next-steps)
- Learn Tinybird concepts: [Core concepts](/docs/forward/core-concepts)
- Review Python resource definitions: [Python SDK resources](/docs/forward/dev-reference/python-sdk-resources)
- Review command options: [Python SDK CLI commands](/docs/forward/dev-reference/commands/python-sdk-cli)
- Configure branches and deployments: [Development workflow](/docs/forward/development-workflow)
- Review the SDK repository for advanced usage: [tinybird-sdk-python](https://github.com/tinybirdco/tinybird-sdk-python)
---
URL: https://www.tinybird.co/docs/forward/quickstarts/cli
Last update: 2026-06-02T11:48:14.000Z
Content:
---
title: "Quickstart: Tinybird CLI \| Tinybird"
theme-color: "#171612"
description: "Follow this step-by-step tutorial to get started with Tinybird."
inkeep:version: "forward"
---
# Quickstart: Tinybird CLI (https://www.tinybird.co/docs/forward/quickstarts/cli#quickstart-tinybird-cli)
Follow these steps to install the Tinybird CLI, build your first data project in a Tinybird branch, and deploy it to Tinybird Cloud.
See [Core concepts](/docs/forward/core-concepts) for a complete overview of Tinybird.
This quickstart is based on the Tinybird CLI and datafiles. If you're looking for a different approach with SDKs, see [Quickstarts](/docs/forward/quickstarts).
If you use a coding agent like Cursor, Claude Code, Amp, or Open Code, install Tinybird agent skills so your agent understands Tinybird project structure and workflows:
```
npx skills add tinybirdco/tinybird-agent-skills
```
## Before you begin (https://www.tinybird.co/docs/forward/quickstarts/cli#before-you-begin)
To get started, you need the following:
- A Tinybird account. You can create one at [cloud.tinybird.co](https://cloud.tinybird.co/).
- Git.
## Deploy a new project in five minutes (https://www.tinybird.co/docs/forward/quickstarts/cli#deploy-a-new-project-in-five-minutes)
### Install Tinybird CLI (https://www.tinybird.co/docs/forward/quickstarts/cli#install-tinybird-cli)
Run the following command to install the Tinybird CLI:
- For macOS and Linux
- For Windows
```
curl https://tinybird.co | sh
```
**Already have Tinybird Classic CLI installed?** Both Classic and Forward CLIs use the `tb` command. Run `which tb` to check if you already have a `tb` binary installed. If you do, see [Managing CLI versions](/docs/forward/guides/migrate-from-classic#install-the-tinybird-forward-cli) to avoid conflicts.
### Create a project (https://www.tinybird.co/docs/forward/quickstarts/cli#create-a-project)
Create a new directory and initialize the project:
```
mkdir tinybird-cli-quickstart
cd tinybird-cli-quickstart
git init
tb init
```
This creates the standard project folders (`datasources/`, `pipes/`, `endpoints/`, `fixtures/`, `tests/`, and others), plus other useful resources like CI/CD templates or Agent Skills. It also creates a `tinybird.config.json` containing your project settings, like the folder where the project is located and the default development environment.
```
{
"dev_mode": "branch",
"include": [
"tinybird"
]
}
```
If you log in during this step, a file called `.tinyb` is created. This contains your auth details, so never commit it to git.
First things first, you need a data source. The schema and engine stay the same regardless of how you ingest data. The difference is how data arrives: via the Events API, from a Kafka topic, or from an S3 bucket.
Create the `.datasource` file. If you want to ingest from Kafka or S3 instead of the Events API, add a `.connection` file and reference it in the `.datasource`:
- Events
- Kafka
- S3
datasources/page_views.datasource
```
DESCRIPTION >
Page view tracking data
SCHEMA >
`timestamp` DateTime `json:$.timestamp`,
`session_id` String `json:$.session_id`,
`pathname` String `json:$.pathname`,
`referrer` Nullable(String) `json:$.referrer`
ENGINE "MergeTree"
ENGINE_SORTING_KEY "pathname, timestamp"
```
Data Sources define the schema and properties of the database tables where you store your data. For more information, see [Data Source files](/docs/forward/dev-reference/datafiles/datasource-files).
### Create an endpoint (https://www.tinybird.co/docs/forward/quickstarts/cli#create-an-endpoint)
Create a new file `endpoints/top_pages.pipe`:
endpoints/top_pages.pipe
```
TOKEN top_pages_read READ
DESCRIPTION >
Get the most visited pages
NODE aggregated
SQL >
%
SELECT
pathname,
count() AS views
FROM page_views
WHERE timestamp >= {{DateTime(start_date, '2026-01-01 00:00:00')}}
AND timestamp <= {{DateTime(end_date, '2026-12-31 23:59:59')}}
GROUP BY pathname
ORDER BY views DESC
LIMIT {{Int32(limit, 10)}}
TYPE endpoint
```
### Build the project (https://www.tinybird.co/docs/forward/quickstarts/cli#build-the-project)
Checkout a new git branch:
```
git add tinybird.config.json && git commit -m "initial commit" && git checkout -b tinybird_intro
```
Run a build to create your resources in your Tinybird development environment:
```
tb build
```
#### Git integration (https://www.tinybird.co/docs/forward/quickstarts/cli#git-integration)
You may have noticed that the quickstart example had a `git checkout -b tinybird_intro`. Tinybird detects you're on a git project and creates a branch with the git branch name in your Tinybird development environment.
### Ingest data and query (https://www.tinybird.co/docs/forward/quickstarts/cli#ingest-data-and-query)
#### Ingest (https://www.tinybird.co/docs/forward/quickstarts/cli#ingest)
Send events using the Events API with `curl`:
```
TB_HOST=$(cat .tinyb | jq -r ".host")
tb --branch=tinybird_intro token copy "admin"
TB_TOKEN=""
curl -X POST "$TB_HOST/v0/events?name=page_views&token=$TB_TOKEN" \
-d '{"timestamp": "2026-01-15 10:00:00", "session_id": "s1", "pathname": "/home", "referrer": "https://google.com"}
{"timestamp": "2026-01-15 10:01:00", "session_id": "s2", "pathname": "/docs", "referrer": null}
{"timestamp": "2026-01-15 10:02:00", "session_id": "s1", "pathname": "/pricing", "referrer": "/home"}'
```
If you configured a Kafka or S3 connector, data is ingested automatically after you deploy. You don't need to call the Events API manually.
#### Query (https://www.tinybird.co/docs/forward/quickstarts/cli#query)
Query the endpoint with `curl`. All Tinybird API calls require a **token** for authentication.
```
curl -X GET "$TB_HOST/v0/pipes/top_pages.json?token=$TB_TOKEN"
```
```
{
"meta": [
{ "name": "pathname", "type": "String" },
{ "name": "views", "type": "UInt64" }
],
"data": [
{ "pathname": "/home", "views": 1 },
{ "pathname": "/docs", "views": 1 },
{ "pathname": "/pricing", "views": 1 }
]
}
```
### Deploy to Tinybird Cloud (https://www.tinybird.co/docs/forward/quickstarts/cli#deploy-to-tinybird-cloud)
```
tb deploy
```
Open the project in Tinybird Cloud:
```
tb --cloud open
```
## Iterate in a branch (https://www.tinybird.co/docs/forward/quickstarts/cli#iterate-in-a-branch)
Add a materialized view in another branch, test the app against that branch, and deploy to Tinybird Cloud when you're finished.
```
git checkout -b materialization
```
Create a new Data Source for the materialized rollup in `datasources/daily_page_views.datasource`:
datasources/daily_page_views.datasource
```
DESCRIPTION >
Daily page view counts populated by a materialized view
SCHEMA >
`date` Date,
`pathname` String,
`views` AggregateFunction(count)
ENGINE "AggregatingMergeTree"
ENGINE_SORTING_KEY "date, pathname"
```
Create the materialized view in `pipes/daily_page_views_mv.pipe`:
pipes/daily_page_views_mv.pipe
```
DESCRIPTION >
Materialized rollup of page views by day and pathname
NODE aggregate
SQL >
SELECT
toDate(timestamp) AS date,
pathname,
countState() AS views
FROM page_views
GROUP BY date, pathname
TYPE materialized
DATASOURCE daily_page_views
```
Update the endpoint in `endpoints/top_pages.pipe` to read from the materialized Data Source:
endpoints/top_pages.pipe
```
TOKEN top_pages_read READ
DESCRIPTION >
Get the most visited pages from MV-backed datasource
NODE aggregated
SQL >
%
SELECT
pathname,
countMerge(views) AS views
FROM daily_page_views
WHERE date >= toDate({{DateTime(start_date, '2026-01-01 00:00:00')}})
AND date <= toDate({{DateTime(end_date, '2026-12-31 23:59:59')}})
GROUP BY pathname
ORDER BY views DESC
LIMIT {{Int32(limit, 10)}}
TYPE endpoint
```
Build against the branch:
```
tb build
```
Once validated, you can deploy to Tinybird Cloud with `tb deploy` if you haven't set up a CI/CD script.
## Next steps (https://www.tinybird.co/docs/forward/quickstarts/cli#next-steps)
- Familiarize yourself with Tinybird concepts. See [Core concepts](/docs/forward/core-concepts).
- Learn about datafiles, like .datasource and .pipe files. See [Datafiles](/docs/forward/dev-reference/datafiles).
- Get data into Tinybird from a variety of sources. See [Get data in](/docs/forward/ingest-data).
- Learn about authentication and securing your Endpoints. See [Tokens](/docs/forward/core-concepts/tokens).
- Browse the Tinybird CLI commands reference. See [Commands reference](/docs/forward/dev-reference/commands).
---
URL: https://www.tinybird.co/docs/forward/query-data/time-series
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Time Series \| Tinybird"
theme-color: "#171612"
description: "Visualize time-based data with Tinybird Time Series."
inkeep:version: "forward"
---
# Time series (https://www.tinybird.co/docs/forward/query-data/time-series#time-series)
Time Series help analyze a sequence of data points collected over an interval of time. Use them to visualize any time series Data Source in your Workspace, including Service Data Sources and Endpoints.
## Getting started (https://www.tinybird.co/docs/forward/query-data/time-series#getting-started)
1. Go to [Tinybird Cloud](https://cloud.tinybird.co/) and log in to your account.
2. Select **Time Series** in the sidebar.
3. Select **New time series**.
4. Pick a Data Source and the date or datetime column.
5. Choose the metric and any grouping columns.
Time Series uses SQL under the hood. For statements, functions, and settings, see the [SQL reference](/docs/sql-reference).
## Time series workflow (https://www.tinybird.co/docs/forward/query-data/time-series#time-series-workflow)
**Shape the chart**
Adjust the date range and granularity, then refine metrics, conditions, and groupings to match the story you need to tell. Select bar or line charts and set the number of results to show.
**Zoom and inspect**
Use smart zoom to drill into spikes or drops and keep the time selector in sync as you explore. Activate it by clicking and dragging the mouse over the chart.
**Cancel long runs**
When a query is running, a **Cancel** button appears. Use it to stop long-running queries mid execution when you need to adjust filters or aggregation settings.
**Open from observability**
Open any observability chart in Time Series to investigate trends with the same interactive controls.
**Share with your team**
Use the sidebar to switch between **Favorites**, **Private**, and **Workspace** Time Series.
- **Private**: Time Series are visible only to you.
- **Workspace**: Time Series are visible and editable by any Workspace member, but only the owner can delete them or stop sharing.
**Work with context**
Select the source to open a side panel, then review in-context metadata like columns, settings, and query plans to validate results.
![Video]()
Controls: true
The side panel shows extra context when you click a Data Source.
## Next steps (https://www.tinybird.co/docs/forward/query-data/time-series#next-steps)
- Learn about [Playgrounds](/docs/forward/query-data/playgrounds) to write your own SQL queries.
- Learn about [Pipes](/docs/forward/core-concepts/pipes).
- Learn about [API Endpoints](/docs/forward/core-concepts/api-endpoints).
---
URL: https://www.tinybird.co/docs/forward/query-data/sql-api
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Query API \| Tinybird"
theme-color: "#171612"
description: "Run SQL queries against Tinybird over HTTP."
inkeep:version: "forward"
---
# Query API (https://www.tinybird.co/docs/forward/query-data/sql-api#query-api)
The [Query API](/docs/api-reference/query-api) runs SQL directly against Pipes and Data Sources in your Workspace through the `/v0/sql` endpoint. Use it for internal tools, automation, and ad hoc queries that do not need a published Endpoint contract.
Use API Endpoints for production query contracts that applications call repeatedly. Use the Query API when the SQL is owned by the caller, such as an internal tool, notebook, script, or debugging workflow.
Query API requests can also use typed [Query parameters](/docs/forward/query-data/query-parameters) when you need validated request-time values.
## Run a SQL query (https://www.tinybird.co/docs/forward/query-data/sql-api#run-a-sql-query)
Send a request to `/v0/sql` with a `q` parameter containing the SQL query. Use a Token with read permissions for the resources you query. Use a [static Token](/docs/forward/core-concepts/static-tokens) for backend services and trusted internal tools. Use a [JWT](/docs/forward/core-concepts/jwt) for browser, user-facing, or multi-tenant access where each user or tenant needs scoped access, filters, expiration, or rate limits.
For short queries, use `GET` and pass the SQL in the `q` query parameter:
```
curl \
-H "Authorization: Bearer " \
--get \
--data-urlencode "q=SELECT * FROM FORMAT JSON" \
https://api.tinybird.co/v0/sql
```
Use `POST` when the SQL query or request parameters are large. This avoids URL length limits and keeps long queries out of the URL:
```
curl \
-X POST \
-H "Authorization: Bearer " \
--data-urlencode "q=SELECT * FROM FORMAT JSON" \
https://api.tinybird.co/v0/sql
```
You can query Data Sources and Pipes by name. Add a `FORMAT` clause when you need a specific response format.
## Available response formats (https://www.tinybird.co/docs/forward/query-data/sql-api#available-response-formats)
Set the response format with a ClickHouse® `FORMAT` clause at the end of the SQL query:
```
SELECT * FROM events LIMIT 10 FORMAT JSON
```
Common formats include:
| Format | Use when |
| --- | --- |
| `JSON` | You need the default JSON response with metadata. |
| `CSV` | You need comma-separated values without headers. |
| `CSVWithNames` | You need comma-separated values with column headers. |
| `JSONEachRow` | You need newline-delimited JSON. |
| `Parquet` | You need a binary columnar format for larger exports or downstream analytics tools. |
If you don't specify a `FORMAT`, Tinybird returns the default Query API format. For the full endpoint contract, see the [Query API reference](/docs/api-reference/query-api).
## Use query parameters (https://www.tinybird.co/docs/forward/query-data/sql-api#use-query-parameters)
Query API requests can use Tinybird's [templating language](/docs/forward/core-concepts/templating-language) and receive request parameters the same way API Endpoints do. This is useful when an internal tool or script owns the SQL but still needs typed, validated inputs.
```
curl \
-X POST \
-H "Authorization: Bearer " \
--data-urlencode "q=% SELECT count() FROM events WHERE timestamp >= {{DateTime(start_date)}} FORMAT JSON" \
--data-urlencode "start_date=2026-01-01 00:00:00" \
https://api.tinybird.co/v0/sql
```
For parameter syntax, defaults, and helper functions, see [Templating language](/docs/forward/core-concepts/templating-language) and [Query parameters](/docs/forward/query-data/query-parameters).
## Query a specific Pipe (https://www.tinybird.co/docs/forward/query-data/sql-api#query-a-specific-pipe)
You can also run a SQL query against a specific Pipe. Send the request to `/v0/pipes/.json` and use `_` as the Pipe input in the `q` parameter:
```
curl \
-X POST \
-H "Authorization: Bearer " \
--data "q=SELECT count() FROM _" \
https://api.tinybird.co/v0/pipes/.json
```
This is useful when you want to reuse a Pipe's logic but apply an extra aggregation, filter, or format at request time.
## Track Query API usage (https://www.tinybird.co/docs/forward/query-data/sql-api#track-query-api-usage)
Query API requests are logged as `query_api` in Service Data Sources. Add metadata as request parameters when you want to group requests by app, customer, or workflow.
```
curl \
-X POST \
-H "Authorization: Bearer " \
--data "q=SELECT count() FROM events FORMAT JSON" \
"https://api.tinybird.co/v0/sql?app_name=internal_tool"
```
See [Endpoint performance](/docs/forward/monitoring/analyze-endpoints-performance#use-the-query-api-with-metadata-parameters) for monitoring examples.
## Next steps (https://www.tinybird.co/docs/forward/query-data/sql-api#next-steps)
- Review the [Query API reference](/docs/api-reference/query-api).
- Add typed parameters with [Query parameters](/docs/forward/query-data/query-parameters).
- Use the [ClickHouse® interface](/docs/forward/query-data/clickhouse-interface) for SQL clients and BI tools.
---
URL: https://www.tinybird.co/docs/forward/query-data/query-parameters
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Query parameters \| Tinybird"
theme-color: "#171612"
description: "Add dynamic query parameters to Tinybird queries."
inkeep:version: "forward"
---
# Query parameters (https://www.tinybird.co/docs/forward/query-data/query-parameters#query-parameters)
Query parameters define values that your app can control when it calls an API Endpoint or the Query API. For example, one Endpoint or Query API request can answer different questions by changing a filter, limit, sort order, or grouping value at request time.
Query parameters are part of Tinybird's [templating language](/docs/forward/core-concepts/templating-language). Use the concept page for the mental model, and this page for common patterns and examples.
Use query parameters to do things like:
- Filtering as part of a `WHERE` clause.
- Changing the number of results as part of a `LIMIT` clause.
- Sorting order as part of an `ORDER BY` clause.
- Selecting specific columns for `ORDER BY` or `GROUP BY` clauses.
## Define query parameters (https://www.tinybird.co/docs/forward/query-data/query-parameters#define-query-parameters)
To make a `.pipe` query dynamic, start the query with a `%` character. That signals the engine that it needs to parse potential parameters. In SDK projects, declare parameters in the `params` object and use the same SQL template syntax inside each node.
After you make a query dynamic, define parameters by using the following pattern `{{([,, description=<"This is a description">, required=])}}`. For example:
Simple select clause using dynamic parameters
```
%
SELECT * FROM TR LIMIT {{Int32(lim, 10, description="Limit the number of rows in the response", required=False)}}
```
The previous query returns 10 results by default, or however many are specified on the `lim` parameter when requesting data.
Boolean parameters don't support the `description` or `required` arguments.
## Call a query with parameters (https://www.tinybird.co/docs/forward/query-data/query-parameters#call-a-query-with-parameters)
When using an Endpoint that uses parameters, pass in the desired parameters.
Using the previous example where `lim` sets the maximum number of rows, the request looks like this:
- cURL
- TypeScript SDK
- Python SDK
```
curl "/v0/pipes/tr_pipe.json?lim=20&token="
```
You can specify parameters in more than one node in a Pipe. When invoking the Endpoint, the passed parameters are included in the request.
When using the Query API, pass parameters together with the `q` parameter. See [Query API](/docs/forward/query-data/sql-api#use-query-parameters).
You can't use query parameters in Materialized Views.
## Use parameters for observability (https://www.tinybird.co/docs/forward/query-data/query-parameters#use-parameters-for-observability)
In addition to using query parameters in your Endpoints and Query API requests, you can use them downstream for monitoring.
When you pass a parameter to your queries, you can build Pipes to reference the parameters and query the Service Data Sources with them, even if you don't use them in the Endpoints themselves.
Review [Service Data Sources](/docs/forward/monitoring/service-datasources) to use the available options. For example, the `user_agent` column on `pipe_stats_rt` shows which user agent made the request. Pass any additional values you need as parameters to improve visibility into incidents and Workspace performance.
Example query to the pipe_stats_rt Service data source leveraging a passed 'referrer' parameter
```
SELECT
toStartOfMinute(start_datetime) AS date,
count() AS requests,
parameters['referrer'] AS referrer
FROM tinybird.pipe_stats_rt
WHERE
status_code != 429
AND start_datetime > now() - INTERVAL 1 HOUR
AND (
pipe_id = ''
OR pipe_name = ''
)
GROUP BY date, referrer
ORDER BY requests DESC, date DESC
```
## Available parameter types (https://www.tinybird.co/docs/forward/query-data/query-parameters#available-parameter-types)
Tinybird supports the following data types for query parameters:
- `Boolean`: Accepts `True` and `False` as values, as well as strings like `'TRUE'`, `'FALSE'`, `'true'`, `'false'`, `'1'`, or `'0'`, or the integers `1` and `0`.
- `String`: For any string values.
- `DateTime64`, `DateTime` and `Date`: Accepts values like `YYYY-MM-DD HH:MM:SS.MMM`, `YYYY-MM-DD HH:MM:SS` and `YYYYMMDD` respectively.
- `Float32` and `Float64`: Accepts floating point numbers of either 32 or 64 bit precision.
- `Int` or `Integer`: Accepts integer numbers of any precision.
- `Int8`, `Int16`, `Int32`, `Int64`, `Int128`, `Int256` and `UInt8`, `UInt16`, `UInt32`, `UInt64`, `UInt128`, `UInt256`: Accepts signed or unsigned integer numbers of the specified precision.
### Column parameters (https://www.tinybird.co/docs/forward/query-data/query-parameters#column-parameters)
You can use `column` to pass along column names of a defined type as parameters, like:
Using column dynamic parameters
```
%
SELECT * FROM TR
ORDER BY {{column(order_by, 'timestamp')}}
LIMIT {{Int32(lim, 10)}}
```
Always define the `column` function's second argument, the one for the default value. The alternative for not defining the argument is to validate that the first argument is defined, but this only has an effect on query execution. A placeholder is used in the development of the Pipes.
Validate the column parameter when not defining a default value
```
%
SELECT * FROM TR
{% if defined(order_by) %}
ORDER BY {{column(order_by)}}
{% end %}
```
### Array parameters (https://www.tinybird.co/docs/forward/query-data/query-parameters#array-parameters)
You can pass along a list of values with the `Array` function for parameters, like so:
Passing arrays as dynamic parameters
```
%
SELECT * FROM TR WHERE
access_type IN {{Array(access_numbers, 'Int32', default='101,102,110')}}
```
### JSON parameters (https://www.tinybird.co/docs/forward/query-data/query-parameters#json-parameters)
Consider the following stringified JSON:
```
"filters": [
{
"operand": "date",
"operator": "equals",
"value": "2018-01-02"
},
{
"operand": "high",
"operator": "greater_than",
"value": "100"
},
{
"operand": "symbol",
"operator": "in_list",
"value": "AAPL,AMZN"
}
]
```
You can use the `JSON()` function to use `filters` as a query parameter. The following example shows to use the `filters` field from the JSON snippet with the stock_prices_1m sample dataset.
```
%
SELECT symbol, date, high
FROM stock_prices_1m
WHERE
1
{% if defined(filters) %}
{% for item in JSON(filters, '[]') %}
{% if item.get('operator', '') == 'equals' %}
AND {{ column(item.get('operand', '')) }} == {{ item.get('value', '') }}
{% elif item.get('operator') == 'greater_than' %}
AND {{ column(item.get('operand', '')) }} > {{ item.get('value', '') }}
{% elif item.get('operator') == 'in_list' %}
AND {{ column(item.get('operand', '')) }} IN splitByChar(',',{{ item.get('value', '') }})
{% end %}
{% end %}
{% end %}
```
When accessing the fields in a JSON object, use the following syntax:
```
item.get('Field', 'Default value to avoid SQL errors').
```
## Pagination (https://www.tinybird.co/docs/forward/query-data/query-parameters#pagination)
You paginate results by adding `LIMIT` and `OFFSET` clauses to your query. You can parameterize the values of these clauses, allowing you to pass pagination values as query parameters to your Endpoint or Query API request.
Use the `LIMIT` clause to select only the first `n` rows of a query result. Use the `OFFSET` clause to skip `n` rows from the beginning of a query result. Together, you can dynamically chunk the results of a query up into pages.
For example, the following query introduces two dynamic parameters, `page_size` and `page`. Use them to control the pagination of a query result.
Paging results using dynamic parameters
```
%
SELECT * FROM TR
LIMIT {{Int32(page_size, 100)}}
OFFSET {{Int32(page, 0) * Int32(page_size, 100)}}
```
You can also use pages to perform calculations such as `count()`. The following example counts the total number of pages:
Operation on a paginated endpoint
```
%
SELECT count() as total_rows, ceil(total_rows/{{Int32(page_size, 100)}}) pages FROM endpoint_to_paginate
```
The addition of a `LIMIT` clause to a query also adds the `rows_before_limit_at_least` field to the response metadata. `rows_before_limit_at_least` is the lower bound on the number of rows returned by the query after transformations but before the limit was applied, and can be useful for response handling calculations.
**Example:**
```
SELECT * FROM users WHERE active = true LIMIT 10
```
Response might include:
- `rows`: 10 (actual rows returned)
- `rows_before_limit_at_least`: 1,000 (at least this many active users exist)
**Why "at least"?** ClickHouse® can stop counting once it knows the answer exceeds the `LIMIT` value. This makes it a lower bound, not an exact count.
**Useful for:**
- Pagination logic ("showing 10 of at least 1,247 results")
- Determining if more results exist without running a separate `COUNT()` query
This metadata is returned automatically with any `LIMIT` query, saving you from running an additional `COUNT(*)` query to check if more results are available.
To get consistent pagination results, add an `ORDER BY` clause to your paginated queries.
## Advanced templating (https://www.tinybird.co/docs/forward/query-data/query-parameters#advanced-templating)
To build more complex queries, use flow control operators like `if`, `else` and `elif` in combination with the `defined()` function, which helps you check whether a parameter has been received and act accordingly.
Tinybird's templating system is based on the [Tornado Python framework](https://github.com/tornadoweb/tornado), and uses Python syntax. You must enclose control statements in curly brackets with percentages `{%..%}` as in the following example:
Advanced templating using dynamic parameters
```
%
SELECT
toDate(start_datetime) as day,
countIf(status_code < 400) requests,
countIf(status_code >= 400) errors,
avg(duration) avg_duration
FROM
log_events
WHERE
endsWith(user_email, {{String(email, 'gmail.com')}}) AND
start_datetime >= {{DateTime(start_date, '2019-09-20 00:00:00')}} AND
start_datetime <= {{DateTime(end_date, '2019-10-10 00:00:00')}}
{% if method != 'All' %} AND method = {{String(method,'POST')}} {% end %}
GROUP BY
day
ORDER BY
day DESC
```
### Validate required parameters (https://www.tinybird.co/docs/forward/query-data/query-parameters#validate-required-parameters)
Validate if a param is in the query
```
%
SELECT * FROM table
{% if defined(my_filter) %}
WHERE attr > {{Int32(my_filter)}}
{% end %}
```
When you call the query with `my_filter=20`, it applies the filter.
### Set default values and understand placeholders (https://www.tinybird.co/docs/forward/query-data/query-parameters#set-default-values-and-understand-placeholders)
Following best practices, you should set default parameter values as follows:
Default parameter values
```
%
SELECT * FROM table
WHERE attr > {{Int32(my_filter, 10)}}
```
When you call the query without setting any value to `my_filter`, it automatically applies the default value of 10.
If you don't set a default value for a parameter, you should validate that the parameter is defined before using it in the query as explained previously.
If you don't validate the parameter and it's not defined, the query might fail. Tinybird populates the parameter with a placeholder value based on the data type. For instance, numerical data types are populated with 0, strings with `__no_value__`, and date and timestamps with `2019-01-01` and `2019-01-01 00:00:00` respectively. You could try yourself with a query like this:
Get placeholder values
```
%
SELECT
{{String(param)}} as placeholder_string,
{{Int32(param)}} as placeholder_num,
{{Boolean(param)}} as placeholder_bool,
{{Float32(param)}} as placeholder_float,
{{Date(param)}} as placeholder_date,
{{DateTime(param)}} as placeholder_ts,
{{Array(param)}} as placeholder_array
```
This returns the following values:
```
{
"placeholder_string": "__no_value__",
"placeholder_num": 0,
"placeholder_bool": 0,
"placeholder_float": 0,
"placeholder_date": "2019-01-01",
"placeholder_ts": "2019-01-01 00:00:00",
"placeholder_array": ["__no_value__0","__no_value__1"]
}
```
### Test parameters in the UI (https://www.tinybird.co/docs/forward/query-data/query-parameters#test-parameters-in-the-ui)
Any dynamic parameters you create appear in the UI. Select **Test new values** to open a test dialog populated with the default value of your parameters. The test dialog helps you test different Pipe values than the default ones without impacting production environments.
Use the View API page to see Endpoint metrics resulting from that specific combination of parameters. Close the dialog to bring the Pipe back to its default production state.
When testing parameters, you can modify both the SQL code and the parameters.
### Encode special characters in test parameters (https://www.tinybird.co/docs/forward/query-data/query-parameters#encode-special-characters-in-test-parameters)
When using the Test UI for parameters, certain special characters need to be encoded:
| Character | Status in Test UI | URI Encoding |
| --- | --- | --- |
| `#` | Truncates the value if not encoded | `%23` |
| `&` | Used as a parameter separator | `%26` |
| `+` | Interpreted as a space character | `%2B` |
| `%` | Reserved for escape sequences | `%25` |
| `"` | Needs encoding | `%22` |
The Forward UI supports special characters without encoding, except `%` and `"`. These character values are not supported.
### Cascade parameters across Pipes (https://www.tinybird.co/docs/forward/query-data/query-parameters#cascade-parameters-across-pipes)
Parameters with the same name in different Pipes are cascaded down the dependency chain.
For example, publish Pipe A with the parameter `foo`, then publish Pipe B with the same parameter and Pipe A as a Data Source. When you call Pipe B's API Endpoint with `foo=bar`, the value of `foo` is `bar` in both Pipes.
### Return custom errors (https://www.tinybird.co/docs/forward/query-data/query-parameters#return-custom-errors)
The following example stops query processing and returns a 400 error:
Validate if a param is defined and throw an error if it's not defined
```
%
{% if not defined(my_filter) %}
{{ error('my_filter (int32) query param is required') }}
{% end %}
SELECT * FROM table
WHERE attr > {{Int32(my_filter)}}
```
The `custom_error` function is an advanced version of `error` where you can customize the response and other aspects. The function gets an object as the first argument, which is sent as JSON, and the status_code as a second argument, which defaults to 400.
Validate if a param is defined and throw an error if it's not defined
```
%
{% if not defined(my_filter) %}
{{ custom_error({'error_id': 10001, 'error': 'my_filter (int32) query param is required'}) }}
{% end %}
SELECT * FROM table
WHERE attr > {{Int32(my_filter)}}
```
## Limits (https://www.tinybird.co/docs/forward/query-data/query-parameters#limits)
You can't use query parameters in nodes that are published as [Materialized Views](/docs/forward/core-concepts/materialized-views), only in Endpoints, Query API requests, or on-demand copies or sinks.
You can use query parameters in scheduled sinks and copies, but must have a default. That default is used in the scheduled execution. The preview step fails if the default doesn't exist.
---
URL: https://www.tinybird.co/docs/forward/query-data/playgrounds
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Playgrounds \| Tinybird"
theme-color: "#171612"
description: "Prototype and test SQL queries in Tinybird Playgrounds."
inkeep:version: "forward"
---
# Playgrounds (https://www.tinybird.co/docs/forward/query-data/playgrounds#playgrounds)
Playgrounds are interactive SQL workspaces in Tinybird Cloud. Use them to explore data, iterate on query logic, and validate results before turning the SQL into a Pipe or Endpoint.
## Getting started (https://www.tinybird.co/docs/forward/query-data/playgrounds#getting-started)
1. Go to [Tinybird Cloud](https://cloud.tinybird.co/) and log in to your account.
2. Select **Playgrounds** in the sidebar.
3. Select **New playground**.
4. Give the playground a title and optional description.
5. Start writing SQL.
For the statements, functions, and settings you can use in queries, see the [SQL reference](/docs/sql-reference).
## Playground workflow (https://www.tinybird.co/docs/forward/query-data/playgrounds#playground-workflow)
**Organize your playground**
Use the list on the left to switch between playgrounds. Mark favorites for quick access and use filters like **Favorites** or **Private** to focus the list.
**Build queries with nodes**
Each playground is made of nodes. Keep nodes focused and readable so you can build multi-step logic and iterate quickly.
**Run and review results**
Use **Run** to execute a node and preview results. Adjust the row limit (for example, **Limited to 100**) or export results when you need to validate outputs outside Tinybird.
**Format and document**
Use **Format** to keep SQL readable. Add node descriptions, or use **Update node description using AI** to keep your work self-documented as it grows.
**Share with your team**
Use **Share** to make a playground available to your workspace when it is ready to review.
**Work with context**
Select the source to open a side panel, then review in-context metadata like columns, settings, and query plans to validate results.
![Video]()
Controls: true
The side panel shows extra context when you click a Data Source.
## From playground to production (https://www.tinybird.co/docs/forward/query-data/playgrounds#from-playground-to-production)
When a query is ready, move the SQL into a Pipe so it can be versioned, deployed, and used by Endpoints or other resources.
## Next steps (https://www.tinybird.co/docs/forward/query-data/playgrounds#next-steps)
- Learn about [Time Series](/docs/forward/query-data/time-series) to quickly visualize data.
- Learn about [Pipes](/docs/forward/core-concepts/pipes).
- Learn about [API Endpoints](/docs/forward/core-concepts/api-endpoints).
---
URL: https://www.tinybird.co/docs/forward/query-data/mcp
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "MCP \| Tinybird"
theme-color: "#171612"
description: "Connect AI agents to Tinybird with the remote MCP server."
inkeep:version: "forward"
---
# MCP (https://www.tinybird.co/docs/forward/query-data/mcp#mcp)
The Tinybird remote MCP server lets AI agents query your Workspace and use published API Endpoints as tools. MCP stands for [Model Context Protocol](https://modelcontextprotocol.io/), a standard interface for exposing tools and context to agents.
Use MCP when you want an agent to explore data, call analytics Endpoints, or answer questions over Tinybird data without writing a custom API integration.
The Tinybird MCP server supports Streamable HTTP. If your MCP client doesn't support Streamable HTTP, use `mcp-remote` as a bridge.
## Before you start (https://www.tinybird.co/docs/forward/query-data/mcp#before-you-start)
You need:
- A Tinybird account and Workspace.
- A Token with the scopes required by the tools you want to expose.
- An MCP client that supports Streamable HTTP, or a client that can run `mcp-remote`.
## Connect to the MCP server (https://www.tinybird.co/docs/forward/query-data/mcp#connect-to-the-mcp-server)
Use this URL in your MCP client:
```
https://mcp.tinybird.co?token=TINYBIRD_TOKEN
```
Replace `TINYBIRD_TOKEN` with a Tinybird Token.
Use a resource-scoped static Token or a JWT when you want to restrict which Endpoints or Data Sources the agent can access. Use an admin Token only for trusted internal tools.
### Clients that need a bridge (https://www.tinybird.co/docs/forward/query-data/mcp#clients-that-need-a-bridge)
For MCP clients that do not support Streamable HTTP directly, configure `mcp-remote`:
```
{
"mcpServers": {
"tinybird": {
"command": "npx",
"args": [
"-y",
"mcp-remote",
"https://mcp.tinybird.co?token=TINYBIRD_TOKEN"
]
}
}
}
```
New Tinybird Tokens include the Tinybird API host. For older Tokens, pass the API host as a query parameter: `https://mcp.tinybird.co?token=TINYBIRD_TOKEN&host=https://api.tinybird.co`. See [Regions and endpoints](/docs/api-reference#regions-and-endpoints).
### JWTs and non-default regions (https://www.tinybird.co/docs/forward/query-data/mcp#jwts-and-non-default-regions)
JWTs do not contain region information. When you use a JWT outside the default region, include the `host` parameter:
```
https://mcp.tinybird.co?token=JWT_TOKEN&host=https://api.REGION.PROVIDER.tinybird.co
```
Examples:
- EU Central 1 AWS: `https://mcp.tinybird.co?token=JWT_TOKEN&host=https://api.eu-central-1.aws.tinybird.co`
- US East 4 GCP: `https://mcp.tinybird.co?token=JWT_TOKEN&host=https://api.us-east.tinybird.co`
See [Regions and endpoints](/docs/api-reference#regions-and-endpoints) for the full list.
## Authentication and Token scopes (https://www.tinybird.co/docs/forward/query-data/mcp#authentication-and-token-scopes)
The tools exposed by the MCP server depend on the Token scopes.
Use static Tokens for backend and internal tools. An admin Token exposes all available tools. Resource-scoped Tokens expose only the resources allowed by the Token.
Use JWTs for user-facing or multi-tenant agents. JWTs can restrict access to specific Endpoints, add fixed parameters, filter Data Sources, and set rate limits. See [Tokens](/docs/forward/core-concepts/tokens) and [JWTs](/docs/forward/core-concepts/jwt).
| Tool | JWT | Admin Token |
| --- | --- | --- |
| Endpoint tools | Specific Endpoints only | ✓ |
| `list_endpoints` | ✓ | ✓ |
| `list_datasources` | ✓ | ✓ |
| `list_service_datasources` | ✓ | ✓ |
| `execute_query` | ✓ | ✓ |
| `explore_data` | ✓ | ✓ |
| `text_to_sql` | ✓ | ✓ |
## Available tools (https://www.tinybird.co/docs/forward/query-data/mcp#available-tools)
The MCP server exposes Endpoint tools and core tools.
### Endpoint tools (https://www.tinybird.co/docs/forward/query-data/mcp#endpoint-tools)
Every published Endpoint in your Workspace can become a tool with the Endpoint name.
Endpoint tools:
- Accept the same parameters as the Endpoint.
- Return the same response as direct API calls.
- Return CSV by default. JSON is also supported.
- Respect Endpoint rate limits and Token scopes.
For example, an Endpoint named `daily_active_users` becomes a tool named `daily_active_users`.
### Core tools (https://www.tinybird.co/docs/forward/query-data/mcp#core-tools)
`explore_data`
: Ask questions about your data and get a natural language answer.
`text_to_sql`
: Convert a natural language question into a SQL query.
`execute_query`
: Run SQL against the Tinybird Query API. The `sql` parameter is required. The optional `format` parameter controls the response format. The default is `CSVWithNames`.
`list_datasources`
: List Data Sources in your Workspace.
`list_service_datasources`
: List Organization and Tinybird service Data Sources available to your Workspace.
`list_endpoints`
: List published Endpoints with names, parameters, and descriptions.
## Use MCP or direct APIs (https://www.tinybird.co/docs/forward/query-data/mcp#use-mcp-or-direct-apis)
Use MCP when:
- You are building an AI agent that needs access to analytics data.
- You want a conversational interface over Data Sources or Endpoints.
- You are prototyping data analysis workflows with an LLM.
- You want Endpoints to become tools without writing tool wrappers.
Use direct APIs when:
- You are building a production app with fixed query patterns.
- You need the lowest possible latency.
- You need custom caching, retries, observability, or request handling.
## Monitor MCP usage (https://www.tinybird.co/docs/forward/query-data/mcp#monitor-mcp-usage)
MCP requests include `from=mcp`. Use Tinybird service Data Sources to monitor queries executed by agents.
```
SELECT *
FROM tinybird.pipe_stats_rt
WHERE url LIKE '%from=mcp%'
AND start_datetime > now() - INTERVAL 1 HOUR
```
See [Service Data Sources](/docs/forward/monitoring/service-datasources).
## Troubleshooting (https://www.tinybird.co/docs/forward/query-data/mcp#troubleshooting)
### Workspace not found with a JWT (https://www.tinybird.co/docs/forward/query-data/mcp#workspace-not-found-with-a-jwt)
If you receive an error like this:
```
Error calling TinybirdAPI: invalid authentication token. Workspace with ID ... not found
```
You might be using a JWT without a `host` parameter. JWTs do not include region information. Add the regional API host to the MCP URL:
```
https://mcp.tinybird.co?token=JWT_TOKEN&host=https://api.REGION.PROVIDER.tinybird.co
```
See [JWTs and non-default regions](https://www.tinybird.co/docs/forward/query-data/mcp#jwts-and-non-default-regions).
## See also (https://www.tinybird.co/docs/forward/query-data/mcp#see-also)
- [Tokens](/docs/forward/core-concepts/tokens)
- [Endpoints](/docs/forward/core-concepts/api-endpoints)
- [Query API](/docs/api-reference/query-api)
- [Model Context Protocol documentation](https://modelcontextprotocol.io/)
---
URL: https://www.tinybird.co/docs/forward/query-data/explorations
Last update: 2026-07-31T11:10:11.000Z
Content:
---
title: "Explorations \| Tinybird"
theme-color: "#171612"
description: "Explore and understand your data with natural language in Tinybird."
inkeep:version: "forward"
---
# Explorations (https://www.tinybird.co/docs/forward/query-data/explorations#explorations)
Explorations is a conversational UI feature in Tinybird. With Explorations, explore and interact with your data using natural language.
## Getting started (https://www.tinybird.co/docs/forward/query-data/explorations#getting-started)
To start using Explorations:
1. Go to [Tinybird Cloud](https://cloud.tinybird.co/) and log in to your account
2. Select **Explorations** in the sidebar
3. Start with a suggested prompt or ask your own question
4. Begin exploring your data
## How it works (https://www.tinybird.co/docs/forward/query-data/explorations#how-it-works)
Explorations provides conversational analysis through a chat interface. Ask questions in natural language and receive structured, step-by-step responses. The AI breaks down complex analyses into clear, practical steps that build upon each other. The chat also suggests follow-up questions that reference previous results, creating a natural flow where each question deepens your understanding of the data.
If you want to write your own SQL queries, use [Playgrounds](/docs/forward/query-data/playgrounds). For time-based data visualization, use Time Series.
## Get the most out of Explorations (https://www.tinybird.co/docs/forward/query-data/explorations#get-the-most-out-of-explorations)
### Custom workspace rules (https://www.tinybird.co/docs/forward/query-data/explorations#custom-workspace-rules)
Fine-tune how the AI assistant responds to get faster and more relevant results. Inside an Exploration, select the **workspace rules** link in the chat prompt input area to add your custom rules. The assistant uses this context in all your prompts.
### Add context (https://www.tinybird.co/docs/forward/query-data/explorations#add-context)
Select the `@` icon near the submit button to add specific resources to your query. The LLM prioritizes them for that specific prompt.
### Getting better results (https://www.tinybird.co/docs/forward/query-data/explorations#getting-better-results)
Start with specific questions rather than broad exploratory queries. Instead of asking "show the data," try "what are the top 5 pages by traffic in the last 7 days?" The AI performs better with clear, focused requests that have measurable outcomes.
## Usage and billing (https://www.tinybird.co/docs/forward/query-data/explorations#usage-and-billing)
At the moment, Explorations usage is free for Developer and Enterprise organizations. Free organizations receive a limited number of credits per month per organization. Tinybird restricts them once they consume all credits. Read more in [Free plan LLM usage](/docs/forward/pricing/free#llm-usage).
Usage can be tracked using the `llm_usage` service data source.
## Limitations and privacy (https://www.tinybird.co/docs/forward/query-data/explorations#limitations-and-privacy)
Explorations are available to all roles within a Workspace.
The Explorations AI chat is only available in Tinybird Cloud. Tinybird anonymously processes a limited sample of your data to deliver faster, more accurate results.
Currently, Explorations cannot be shared or exported. Each user can only access their own Explorations.
## Next steps (https://www.tinybird.co/docs/forward/query-data/explorations#next-steps)
- Learn about [Playgrounds](/docs/forward/query-data/playgrounds) to write your own SQL queries.
- Learn about [Time Series](/docs/forward/query-data/time-series) to quickly visualize data.
- Learn about [Pipes](/docs/forward/core-concepts/pipes).
- Learn about [Materialized views](/docs/forward/core-concepts/materialized-views).
---
URL: https://www.tinybird.co/docs/forward/query-data/clickhouse-interface
Last update: 2026-08-07T12:15:37.000Z
Content:
---
title: "ClickHouse® interface \| Tinybird"
theme-color: "#171612"
description: "Learn how to connect BI tools and SQL clients to Tinybird using the ClickHouse HTTP interface for data visualization and analysis."
inkeep:version: "forward"
---
# ClickHouse® interface (https://www.tinybird.co/docs/forward/query-data/clickhouse-interface#clickhouse-interface)
Tinybird is compatible with the ClickHouse® HTTP interface, so BI tools, SQL clients, and data visualization platforms can query Tinybird Data Sources directly.
Use the ClickHouse interface when a tool already supports ClickHouse and needs read-only SQL access to Tinybird.
The ClickHouse connection to Tinybird is read-only. You can use it to query, visualize, and analyze data from your Tinybird Data Sources, but you cannot modify data through this connection.
## Connection parameters (https://www.tinybird.co/docs/forward/query-data/clickhouse-interface#connection-parameters)
To connect any ClickHouse®-compatible tool to Tinybird, use these standard connection parameters:
### Basic Connection settings (https://www.tinybird.co/docs/forward/query-data/clickhouse-interface#basic-connection-settings)
```
Protocol: HTTPS
Host: clickhouse..tinybird.co
Port: 443 (HTTPS)
SSL/TLS: Required (enabled)
```
Replace `` with your workspace region (e.g., `eu-west-1.aws`, `us-east-1`).
### ClickHouse interface hosts (https://www.tinybird.co/docs/forward/query-data/clickhouse-interface#clickhouse-interface-hosts)
Here are the ClickHouse HTTP hosts for each Tinybird region:
| Region | Provider | Provider region | ClickHouse interface URL |
| --- | --- | --- | --- |
| Europe | GCP | europe-west2 | clickhouse.europe-west2.gcp.tinybird.co |
| Europe | GCP | europe-west3 | clickhouse.tinybird.co |
| US East | GCP | us-east4 | clickhouse.us-east.tinybird.co |
| North America | GCP | northamerica-northeast2 | clickhouse.northamerica-northeast2.gcp.tinybird.co |
| Europe | AWS | eu-central-1 | clickhouse.eu-central-1.aws.tinybird.co |
| Europe | AWS | eu-west-1 | clickhouse.eu-west-1.aws.tinybird.co |
| US East | AWS | us-east-1 | clickhouse.us-east.aws.tinybird.co |
| US West | AWS | us-west-2 | clickhouse.us-west-2.aws.tinybird.co |
### Authentication (https://www.tinybird.co/docs/forward/query-data/clickhouse-interface#authentication)
```
Username: # Optional, for identification purposes
Password: # Your Tinybird Auth Token
```
The username field is not used for authentication but helps identify the connection. The actual authentication is done via the Auth Token in the password field.
### Auth Token requirements (https://www.tinybird.co/docs/forward/query-data/clickhouse-interface#auth-token-requirements)
Your Auth Token must have the following permissions:
- **Read access** to Workspace Data Sources
- **Read access** to Service Data Sources (if needed)
- **Read access** to Organization Data Sources (if needed)
Create a token with these permissions:
```
tb --cloud token create static --scope "WORKSPACE:READ_ALL" --scope "ORG_DATASOURCES:READ"
```
### Tinybird Local (https://www.tinybird.co/docs/forward/query-data/clickhouse-interface#tinybird-local)
To connect to your local development environment using Tinybird Local, use these connection parameters:
```
Protocol: HTTP
Host: localhost
Port: 7182
SSL/TLS: Disabled
```
You can test the connection using curl:
```
curl -H "X-ClickHouse-Key: your_token_here" \
"http://localhost:7182/?query=SELECT%20*%20FROM%20system.tables"
```
## Available databases (https://www.tinybird.co/docs/forward/query-data/clickhouse-interface#available-databases)
The ClickHouse interface exposes four databases:
- **``**: Contains your Workspace's Data Sources.
- **`tinybird`**: Contains [Workspace Service Data Sources](/docs/forward/monitoring/service-datasources#service-data-sources), such as `tinybird.datasources_ops_log` and `tinybird.pipe_stats_rt`.
- **`organization`**: Contains [Organization Service Data Sources](/docs/forward/monitoring/service-datasources#organization-service-data-sources), such as `organization.workspaces` and `organization.pipe_stats_rt`.
- **`system`**: Contains a subset of system tables, including `system.tables`, `system.columns`, `system.parts`, `system.part_log`, `system.query_views_log`, and `system.query_log`.
Querying `system.query_log`, `system.query_views_log` or `system.part_log` without filters is too resource-intensive and times out. Always apply a time filter on the `event_time` column to query a reasonable time frame.
## Compatible tools (https://www.tinybird.co/docs/forward/query-data/clickhouse-interface#compatible-tools)
Tinybird's ClickHouse® interface works with BI tools and SQL clients that support ClickHouse:
### Business intelligence platforms (https://www.tinybird.co/docs/forward/query-data/clickhouse-interface#business-intelligence-platforms)
- **[Grafana](/docs/forward/guides/connect-grafana)**
- **[Hex](/docs/forward/guides/connect-hex)**
- **[Metabase](/docs/forward/guides/connect-metabase)**
- **[Superset](/docs/forward/guides/connect-superset)**
- **[Power BI](/docs/forward/guides/connect-powerbi)**
- **[Redash](/docs/forward/guides/connect-redash)**
- **[Chartbrew](/docs/forward/guides/connect-chartbrew)**
- **[Deepnote](/docs/forward/guides/connect-deepnote)**
- **[Draxlr](/docs/forward/guides/connect-draxlr)**
- **[Fabi.ai](/docs/forward/guides/connect-fabi)**
- **[Holistics](/docs/forward/guides/connect-holistics)**
- **[Luzmo](/docs/forward/guides/connect-luzmo)**
- **[Mitzu](/docs/forward/guides/connect-mitzu)**
- **[Tableau](/docs/forward/guides/connect-tableau)**
### SQL clients and IDEs (https://www.tinybird.co/docs/forward/query-data/clickhouse-interface#sql-clients-and-ides)
- **[DataGrip](/docs/forward/guides/connect-datagrip)**
- **[DBeaver](/docs/forward/guides/connect-dbeaver)**
### Application clients (https://www.tinybird.co/docs/forward/query-data/clickhouse-interface#application-clients)
- **[ClickHouse JS Client](/docs/forward/guides/connect-clickhouse-js)**
- **[ClickHouse Go Client](/docs/forward/guides/connect-clickhouse-go)**
- **[ClickHouse Python Client](/docs/forward/guides/connect-clickhouse-python)**
### Other compatible tools (https://www.tinybird.co/docs/forward/query-data/clickhouse-interface#other-compatible-tools)
Any tool that supports ClickHouse® HTTP interface can potentially work with Tinybird, including:
- Looker (via custom connection)
- Other JetBrains IDEs
- Retool and other low-code platforms
## Limitations (https://www.tinybird.co/docs/forward/query-data/clickhouse-interface#limitations)
The interface is **read-only**. You cannot perform `INSERT`, `UPDATE`, `DELETE`, or any DDL operations (`CREATE`, `ALTER`, `DROP`).
## Observability (https://www.tinybird.co/docs/forward/query-data/clickhouse-interface#observability)
Queries made through the ClickHouse® interface are logged as [Query API](/docs/api-reference/query-api) requests in the pipe statistics Data Sources. You can monitor these queries using:
- `tinybird.pipe_stats_rt` and `tinybird.pipe_stats` - Workspace-level statistics
- `organization.pipe_stats_rt` and `organization.pipe_stats` - Organization-level statistics
ClickHouse® queries appear with `pipe_id` and `pipe_name` as `'query_api'`. The actual SQL query is available in the `parameters` column under the `q` field.
For more details, see [Service Data Sources](/docs/forward/monitoring/service-datasources).
## Troubleshooting (https://www.tinybird.co/docs/forward/query-data/clickhouse-interface#troubleshooting)
**Invalid host or region**
```
Error: Connection refused
```
Verify your region and use the correct host format: `clickhouse..tinybird.co`
**Authentication failures**
```
Error: Authentication failed
```
- Check your Auth Token is valid and not expired
- Ensure the token has read permissions for the workspace
- Verify the token is entered in the password field
**SSL/TLS errors**
```
Error: SSL connection error
```
- Ensure SSL/TLS is enabled in your client
- Use port 443 with HTTPS protocol
- Check that your client supports TLS 1.2 or higher.
**Query timeouts**
```
Error: Query timeout
```
- Add appropriate WHERE clauses to filter data
- Use LIMIT to reduce result set size
- Consider querying Materialized Views instead of raw data
### Getting help (https://www.tinybird.co/docs/forward/query-data/clickhouse-interface#getting-help)
If you encounter issues not covered here:
1. Check the specific tool integration guide
2. Review your [Auth Token](https://cloud.tinybird.co/tokens) permissions
3. Test the connection with a simple query
4. Contact [Tinybird support](/docs/forward/support) for additional assistance
## Next steps (https://www.tinybird.co/docs/forward/query-data/clickhouse-interface#next-steps)
Connect your BI tool and follow the specific integration guide to set up dashboards and visualizations with your Tinybird data.
- [ClickHouse JS](/docs/forward/guides/connect-clickhouse-js)
- [ClickHouse Go](/docs/forward/guides/connect-clickhouse-go)
- [ClickHouse Python](/docs/forward/guides/connect-clickhouse-python)
- [Grafana](/docs/forward/guides/connect-grafana)
- [Metabase](/docs/forward/guides/connect-metabase)
---
URL: https://www.tinybird.co/docs/forward/query-data/api-endpoints
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "API Endpoints \| Tinybird"
theme-color: "#171612"
description: "Query Tinybird API Endpoints from applications and tools."
inkeep:version: "forward"
---
# API Endpoints (https://www.tinybird.co/docs/forward/query-data/api-endpoints#api-endpoints)
[Endpoints](/docs/forward/core-concepts/api-endpoints) publish the result of a Pipe as a REST API. Use them when an app, dashboard, agent, or external service needs a stable query contract with predictable inputs and outputs.
Use Endpoints when the query belongs to your Tinybird project and callers should only provide request parameters. Use the [Query API](/docs/forward/query-data/sql-api) when the caller owns the SQL, such as an internal tool, notebook, script, or debugging workflow.
## Call an Endpoint (https://www.tinybird.co/docs/forward/query-data/api-endpoints#call-an-endpoint)
Call an Endpoint by name and include a Token with read permissions for the Endpoint. Use a [static Token](/docs/forward/core-concepts/static-tokens) for backend services and trusted internal tools. Use a [JWT](/docs/forward/core-concepts/jwt) for browser, user-facing, or multi-tenant access where each user or tenant needs scoped access, fixed parameters, filters, expiration, or rate limits.
```
curl \
-H "Authorization: Bearer " \
"https://api.tinybird.co/v0/pipes/top_pages.json"
```
You can also pass the Token or JWT as a `token` query parameter when needed:
```
curl "https://api.tinybird.co/v0/pipes/top_pages.json?token="
```
## Use query parameters (https://www.tinybird.co/docs/forward/query-data/api-endpoints#use-query-parameters)
Endpoints can use Tinybird's [templating language](/docs/forward/core-concepts/templating-language) to receive typed, validated request parameters. Use parameters for filters, date ranges, pagination, tenant IDs, sorting, and limits.
```
curl \
-H "Authorization: Bearer " \
"https://api.tinybird.co/v0/pipes/top_pages.json?start_date=2026-01-01%2000:00:00&limit=20"
```
For parameter syntax, defaults, helper functions, pagination, and advanced patterns, see [Query parameters](/docs/forward/query-data/query-parameters).
## Choose a response format (https://www.tinybird.co/docs/forward/query-data/api-endpoints#choose-a-response-format)
Choose the response format by appending the format extension to the Endpoint URL.
```
curl \
-H "Authorization: Bearer " \
"https://api.tinybird.co/v0/pipes/top_pages.ndjson"
```
Supported formats include:
- `.json`: JSON response. This is the default format.
- `.csv`: CSV response.
- `.csvwithnames`: CSV response with column names.
- `.ndjson`: Newline-delimited JSON response.
- `.parquet`: Parquet response.
- `.prometheus`: Prometheus metrics response.
See [Endpoint response formats](/docs/forward/core-concepts/api-endpoints#response-formats) and the [Pipe API reference](/docs/api-reference/pipe-api/api-endpoints#response-formats) for more details.
## Errors and retries (https://www.tinybird.co/docs/forward/query-data/api-endpoints#errors-and-retries)
Endpoints return standard HTTP success or error codes. Error responses include extra information about what went wrong, encoded as JSON.
### Error codes (https://www.tinybird.co/docs/forward/query-data/api-endpoints#error-codes)
Endpoints might return the following HTTP error codes:
| Code | Description |
| --- | --- |
| 400 | Bad request. A `HTTP400` can be returned in several scenarios and typically represents a malformed request, such as errors in your SQL queries or missing query parameters. |
| 403 | Forbidden. The auth Token doesn't have the correct scopes. |
| 404 | Not found. This usually occurs when the name of the Endpoint is wrong or hasn't been published. |
| 405 | HTTP method not allowed. The request method isn't allowed for this Endpoint. |
| 408 | Request timeout. This occurs when the query takes too long to complete. By default, this is 10 seconds. |
| 414 | Request-URI Too Large. Not all APIs have the same limit, but it's usually 2KB for GET requests. Reduce the URI length or use a POST request to avoid the limit. |
| 429 | Too many requests. Usually occurs when an Endpoint hits rate limits. |
| 499 | Connection closed. This occurs if the client closes the connection after 1 second. If this is unexpected, increase the connection timeout on your end. |
| 500 | Internal Server Error. Usually an unexpected transient service error. |
Errors when running a query are usually reported as `400 Bad request` or `500 Internal Server Error`, depending on whether the caller can fix the error.
In those cases, the API response includes an additional `X-DB-Exception-Code` HTTP header with the internal database error as a stringified number.
### Retries (https://www.tinybird.co/docs/forward/query-data/api-endpoints#retries)
When you call an Endpoint from an API Gateway, backend service, or job, handle potential errors and implement retry strategies where appropriate.
Implement automatic retries for the following errors:
- HTTP 429: Too many requests
- HTTP 500: Internal Server Error
Use exponential backoff when retrying requests that return these errors.
## Track Endpoint usage (https://www.tinybird.co/docs/forward/query-data/api-endpoints#track-endpoint-usage)
Endpoint requests are logged in Service Data Sources. Add metadata as request parameters when you want to group requests by app, customer, workflow, or agent.
```
curl \
-H "Authorization: Bearer " \
"https://api.tinybird.co/v0/pipes/top_pages.json?tenant_id=acme&app_name=dashboard"
```
See [Endpoint performance](/docs/forward/monitoring/analyze-endpoints-performance) for monitoring examples.
## Next steps (https://www.tinybird.co/docs/forward/query-data/api-endpoints#next-steps)
- Learn the [Endpoints concept](/docs/forward/core-concepts/api-endpoints).
- Add dynamic inputs with [Query parameters](/docs/forward/query-data/query-parameters).
- Use the [Query API](/docs/forward/query-data/sql-api) for caller-owned SQL.
- Review the [Pipe API reference](/docs/api-reference/pipe-api/api-endpoints).
---
URL: https://www.tinybird.co/docs/forward/pricing/saas
Last update: 2026-08-07T12:15:37.000Z
Content:
---
title: "SaaS plan \| Tinybird"
theme-color: "#171612"
description: "The Tinybird SaaS plan — shared infrastructure with 4+ vCPUs, credit-based billing, and mandatory support."
inkeep:version: "forward"
---
# SaaS plan (https://www.tinybird.co/docs/forward/pricing/saas#saas-plan)
The SaaS plan runs on **shared infrastructure** with a commitment. It starts at 4 vCPUs and uses a credit-based billing system with a minimum 12-month commitment. Billing uses active minutes, QPS overages, storage overages, and related SKUs. Minimum storage is 1 TB. A support package is mandatory.
For plans on **dedicated infrastructure** with your own ClickHouse® cluster, see [Enterprise](./enterprise). For shared infrastructure without a commitment, see [Developer](./developer).
## What's included (https://www.tinybird.co/docs/forward/pricing/saas#whats-included)
SaaS plan sizes start at 4 vCPUs. For available sizes and what's included in each, refer to your contract or [contact sales](https://www.tinybird.co/contact-us).
## Billing (https://www.tinybird.co/docs/forward/pricing/saas#billing)
SaaS plans are billed based on the following resources:
| Resource | Billing | Type |
| --- | --- | --- |
| vCPUs | Number of vCPUs available. Starting at 4 vCPUs. Each vCPU size includes a number of active minutes. | Fixed. |
| Active minutes | An active minute is when any operation used a vCPU for a minute. See [Active minutes](https://www.tinybird.co/docs/forward/pricing/saas#active-vcpu-minutes--hours). | Usage based. |
| Queries per second (QPS) | Queries per second, or QPS. Each vCPU size includes a QPS allowance. See [Queries per second](https://www.tinybird.co/docs/forward/pricing/saas#queries-per-second). | Usage based when usage exceeds the QPS included in the plan. |
| Storage | Average of daily maximum usage of the compressed disk storage of all your data, in gigabytes. | Usage based when usage exceeds the minimum 1 TB of storage included in your contract. |
| Data transfer | When using [Sinks](/docs/api-reference/sink-pipes-api), usage is billed depending on the destination, which can be the same cloud provider and region as your Tinybird cluster, or a different one. | Usage based. |
| Support | Standard or Premier support package as agreed in your contract. Standard is included. Premier pricing is defined in your agreement. | Fixed. |
| LLM usage | LLM usage in AI tools within MCP and Explorations UI. Includes $200 per billing cycle. | Usage based when usage exceeds the $200 included in the plan. |
The same QPS, active minutes, and storage overage rates apply in every region. See [Pricing](https://www.tinybird.co/pricing) for current rates.
### Active vCPU minutes / hours (https://www.tinybird.co/docs/forward/pricing/saas#active-vcpu-minutes--hours)
An active minute is any minute where at least one operation used a vCPU, regardless of actual vCPU time consumed during that minute. Whether you use 1 vCPU second or 60 vCPU seconds within a minute, it counts as 1 active minute. Multiple operations executed within the same minute still count as a single active minute. When using fractioned vCPUs, an active minute is proportional to the fraction. For example, 30 seconds of 0.5 vCPU.
Plan sizes include a number of active hours, calculated as active minutes divided by 60. If you consume all your active minutes, the overage is billed at a fixed rate per minute. Usage bursts allow you to temporarily exceed the vCPU usage limit. See [vCPU burst mode](https://www.tinybird.co/docs/forward/pricing/saas#vcpu-burst-mode).
### Queries per second (https://www.tinybird.co/docs/forward/pricing/saas#queries-per-second)
Queries per second, or QPS, is the number of queries per second that your plan includes. Calls to API endpoints and queries sent to the Query API count towards your QPS allowance. The limit excludes queries from the UI.
If you exceed the QPS allowance, the platform supports traffic peaks up to a ceiling of 4x the QPS included in your plan. Requests that exceed the allowance incur a fixed charge per request. Requests beyond the ceiling are rate limited. See [QPS overages and QPS ceiling](https://www.tinybird.co/docs/forward/pricing/saas#qps-overages-and-qps-ceiling).
### QPS overages and QPS ceiling (https://www.tinybird.co/docs/forward/pricing/saas#qps-overages-and-qps-ceiling)
Once you are over your plan's QPS allowance, you can keep making requests up to the plan's QPS ceiling of 4 times your plan's QPS. Requests that exceed the allowance incur a fixed charge per request. This applies to API endpoint and Query API requests.
While your organization is temporarily limited due to vCPU usage over the vCPU burst capacity, you don't have access to the QPS ceiling of QPS allowance × 4. Request rates are limited to your QPS allowance.
### vCPU burst mode (https://www.tinybird.co/docs/forward/pricing/saas#vcpu-burst-mode)
You can't exceed the vCPU time limit indefinitely. The limit relates to vCPU time used during a minute. Tinybird allows usage bursts of 2x the vCPU time per minute. If you reach the vCPU limit, you can use the base QPS limit of your plan size during the next 5 minutes without QPS overages.
### Max threads (https://www.tinybird.co/docs/forward/pricing/saas#max-threads)
Max threads is the maximum number of concurrent threads used to execute queries in API Endpoints and Query API calls. Your SaaS plan's thread allowance determines this limit.
While more threads can improve concurrent query processing, final performance also depends on your vCPU limit and query complexity.
### Data storage (https://www.tinybird.co/docs/forward/pricing/saas#data-storage)
You are charged for persistent data stored in ClickHouse® per Data Source. SaaS plans have a minimum of 1 TB of storage as defined in your contract. Storage beyond the included amount is billed at a fixed rate per GB. Rates aren't tied to region.
### Data transfer (https://www.tinybird.co/docs/forward/pricing/saas#data-transfer)
The data transfer SKU covers data sent out of the platform, primarily through [Sinks](/docs/forward/core-concepts/sinks). Data transfer is charged per gigabyte. The rate depends on the transfer type. Intra transfers stay within the same cloud provider and region. Inter transfers cross providers or regions.
### Support SKU (https://www.tinybird.co/docs/forward/pricing/saas#support-sku)
The Support SKU applies to SaaS plans. A support package is mandatory. Standard is included. Premier is available for SaaS S-24+. Support charges don't depend on ticket volume or platform activity. Premier pricing and final terms are defined in your customer agreement. See [Support](../support) for details.
### LLM usage (https://www.tinybird.co/docs/forward/pricing/saas#llm-usage)
LLM usage in AI tools within MCP and Explorations UI includes $200 of credits per billing cycle. Usage beyond included credits is billed at cost.
You can monitor usage through the `tinybird.llm_usage` Service Data Source.
## Credits and billing cycle (https://www.tinybird.co/docs/forward/pricing/saas#credits-and-billing-cycle)
SaaS plans are billed using credits. A credit is a unit of resource usage that can be consumed across Tinybird SKUs. The minimum commitment period is 12 months.
Bills are issued monthly based on credits consumed during the billing cycle (first to last day of the month). Invoices are issued when credits are purchased (for example, at contract signature or when buying additional credits).
## Monitor usage and invoices (https://www.tinybird.co/docs/forward/pricing/saas#monitor-usage-and-invoices)
You can monitor credit consumption, remaining commitment, and infrastructure usage from your organization's dashboard. You can also review monthly usage receipts.
## Support (https://www.tinybird.co/docs/forward/pricing/saas#support)
A support package is mandatory on SaaS plans. Standard is included. Premier is available for SaaS S-24+. Premier pricing and final terms are defined in your customer agreement. See [Support](../support) for full details.
## Resize and manage (https://www.tinybird.co/docs/forward/pricing/saas#resize-and-manage)
Resizing or making other changes to a SaaS plan requires [contacting sales](https://www.tinybird.co/contact-us).
## Next steps (https://www.tinybird.co/docs/forward/pricing/saas#next-steps)
- [Developer plan](./developer): Shared infrastructure without a commitment.
- [Enterprise plan](./enterprise) — Dedicated ClickHouse® cluster with no resource sharing.
- [Limits](./limits): Platform and operational limits.
---
URL: https://www.tinybird.co/docs/forward/pricing/limits
Last update: 2026-08-07T12:15:37.000Z
Content:
---
title: "Limits \| Tinybird"
theme-color: "#171612"
description: "Tinybird has limits on certain operations and processes to ensure the highest performance."
inkeep:version: "forward"
---
# Limits (https://www.tinybird.co/docs/forward/pricing/limits#limits)
Tinybird has limits on certain operations and processes to ensure the highest performance. All limits are applied at the organization level, meaning resources and limits are shared across all workspaces within your organization.
Each plan page documents plan-specific billing limits and SKUs:
- [Free plan](./free)
- [Developer plan](./developer)
- [SaaS plan](./saas)
- [Enterprise plan](./enterprise)
## Rate limiter (https://www.tinybird.co/docs/forward/pricing/limits#rate-limiter)
The rate limiter monitors cluster health and may temporarily limit concurrent requests to prevent instability due to memory or CPU pressure.
It activates when both conditions are met:
- The cluster is under resource pressure, either:
- Memory usage exceeds the threshold: greater than 70% of cluster capacity on Free and Developer plans, or greater than 80% on Enterprise (dedicated) plans.
- CPU is saturated: the 1-minute load average exceeds 3 times the number of available CPU cores.
- The percentage of `408 Timeout Exceeded` and `500 Internal Server Error` responses for a Pipe Endpoint exceeds 10% of its successful requests.
When active, the maximum concurrent requests for affected Pipe Endpoints is reduced proportionally to the observed error percentage. Under CPU pressure, reduced limits only apply to the Pipe Endpoints consuming a significant share of the cluster's CPU. Workspace administrators on dedicated infrastructure receive an email with affected Endpoints and applied concurrency limits.
Once Pipes stop receiving `408` or `500` errors, concurrency limits are gradually relaxed until removed.
While the rate limiter is active, Endpoints can return HTTP `429`. Clients should retry with exponential backoff.
## Notifications by email (https://www.tinybird.co/docs/forward/pricing/limits#email-notifications)
When reaching or temporarily exceeding your plan's limits, you receive one of the following emails.
### Free plans (https://www.tinybird.co/docs/forward/pricing/limits#free-plans)
The following table shows the limits' notifications you might receive for the Free plan.
| Limit | Warning email | Alert email |
| --- | --- | --- |
| vCPU usage (1/4 vCPU) | | Exceeded plan's base vCPU allowance within the past hour. |
| Storage Usage | Used more than 75% of allocated storage. | Used more than 100% of allocated storage. |
| Requests per day (1,000 queries) | Used more than 75% of daily query limit. | Used more than 100% of daily query limit. |
| LLM usage ($25 LLM credits per cycle) | | Used 100% of LLM credits in current billing cycle. |
### Developer and SaaS plans (https://www.tinybird.co/docs/forward/pricing/limits#developer-and-saas-plans)
The following table shows the limits' notifications you might receive for plans on shared infrastructure.
| Limit | Warning email | Alert email |
| --- | --- | --- |
| vCPU usage | Used more vCPU time than your plan includes, resulting in overages. This notification is sent once per billing cycle. | Exceeded your plan's vCPU time limit within the last hour. |
| Extra costs | Overage costs in vCPU time exceed 33% of your plan's base fee. | Daily email with a breakdown of extra costs when vCPU overages occur. |
| Storage Usage | Used more than 75% of allocated storage. | Used more than 100% of allocated storage. |
| LLM usage ($100 LLM credits per cycle) | | Used 100% of LLM credits in current billing cycle. |
## Workspace limits (https://www.tinybird.co/docs/forward/pricing/limits#workspace-limits)
The following limits apply to all workspaces in an organization, regardless of the plan.
| Description | Limit |
| --- | --- |
| Number of Workspaces | 90 Workspaces. Soft limit, contact support to increase it. |
| Number of members | 90 members. Soft limit, contact support to increase it. |
| Number of branches, including `main` | Default 4. Hard limit on shared infrastructure. Soft limit on dedicated infrastructure, contact support to increase it. |
| Number of Data Sources | 100 Data Sources. Soft limit, contact support to increase it. |
| Number of Copy Pipes | Up to 200 Copy Pipes, depending on the plan. See [Copy Pipe limits](https://www.tinybird.co/docs/forward/pricing/limits#copy-pipe-limits) |
| Number of Sink Pipes | Up to 10 Pipes, depending on the plan. See [Sink Pipe limits](https://www.tinybird.co/docs/forward/pricing/limits#sink-pipe-limits). |
| Number of Tokens | 100,000 tokens. If you need more you should take a look at [JWT tokens](/docs/forward/core-concepts/jwt) |
| Number of secrets | 100 secrets. |
See [Rate limits for JWTs](/docs/forward/core-concepts/jwt#rate-limits-for-jwts) for more detail specifically on JWT limits.
## Ingestion limits (https://www.tinybird.co/docs/forward/pricing/limits#ingestion-limits)
The following ingestion limits apply to all workspaces in an organization.
| Description | Limit |
| --- | --- |
| Data Source max columns | 500 |
| Full body upload | 8 MB |
| Multipart upload - CSV and NDJSON | 500 MB |
| Multipart upload - Parquet | 50 MB |
| Max file size - Parquet - Free plan | 1 GB |
| Max file size - Parquet - Developer, SaaS, and Enterprise plans | 5 GB |
| Max file size (uncompressed) - Free plan | 10 GB |
| Max uncompressed file size - Developer, SaaS, and Enterprise plans | 32 GB |
| Kafka topics | 5 topics. Enterprise plan users can contact support to increase. |
| Max parts created at once - NDJSON/Parquet jobs and Events API | 12 parts |
### Ingestion limits (API) (https://www.tinybird.co/docs/forward/pricing/limits#ingestion-limits-api)
Tinybird throttles requests based on the capacity. So if your queries are using 100% resources you might not be able to run more queries until the running ones finish.
| Description | Limit and time window |
| --- | --- |
| Request size - Events API | 10 MB |
| Response size | 100 MB |
| Create Data Source from schema | 25 times per minute |
| Create Data Source from file or URL * | 5 times per minute |
| Append data to Data Source `POST /v0/datasources` * | 5 times per minute |
| Append data to Data Source using `POST /v0/events` | 100 times per second |
| Create Data Source using `POST /v0/events` | 5 times per minute |
| Replace data in a Data Source * | 5 times per minute |
* The quota is shared at Workspaces level when creating, appending data, or replacing data. For example, you can't do 5 requests of each type per minute, for a total of 15 requests. You can do at most a grand total of 5 requests of those types combined. Import jobs using append mode, [S3 connector](/docs/forward/ingest-data/connectors/s3) ingests, [Datasource API](/docs/api-reference/datasource-api) posts, and [CLI datasource commands](/docs/forward/dev-reference/commands/tb-datasource#tb-datasource-append) all use the `/v0/datasources` endpoint and are subject to the limit.
The number of rows in append requests doesn't impact the ingestion limit; each request counts as a single ingestion.
If you exceed your rate limit, your request is throttled and you receive *HTTP 429 Too Many Requests* response codes from the API. Each response contains a set of HTTP headers with your current rate limit status.
| Header Name | Description |
| --- | --- |
| `X-RateLimit-Limit` | The maximum number of requests you're permitted to make in the current limit window. |
| `X-RateLimit-Remaining` | The number of requests remaining in the current rate limit window. |
| `X-RateLimit-Reset` | The time in seconds after the current rate limit window resets. |
| `Retry-After` | The time to wait before making a another request. Only present on 429 responses. |
## Query limits (https://www.tinybird.co/docs/forward/pricing/limits#query-limits)
The following query limits apply to all workspaces in an organization, regardless of the plan.
| Description | Limit |
| --- | --- |
| SQL length | 128KB |
| Result length | 100 MB |
| Query execution time | 10 seconds |
If you exceed your rate limit, your request is throttled and you receive *HTTP 429 Too Many Requests* response codes from the API. Each response contains a set of HTTP headers with your current rate limit status.
| Header Name | Description |
| --- | --- |
| `X-RateLimit-Limit` | The maximum number of requests you're permitted to make in the current limit window. |
| `X-RateLimit-Remaining` | The number of requests remaining in the current rate limit window. |
| `X-RateLimit-Reset` | The time in seconds after the current rate limit window resets. |
| `Retry-After` | The time to wait before making a another request. Only present on 429 responses. |
### Query timeouts (https://www.tinybird.co/docs/forward/pricing/limits#query-timeouts)
If query execution time exceeds the default limit of 10 seconds, an error message appears. Long execution times hint at issues that need to be fixed in the query or the Data Source schema.
To avoid query timeouts, you can:
- Optimize your queries to remove inefficiencies and common mistakes.
- Increase the number of available threads in your plan to process queries concurrently. See [Max threads](./developer#max-threads) on the Developer plan or the equivalent section on [SaaS](./saas#max-threads) or [Free](./free#max-threads).
If you still need to increase the timeout limit, contact support. See [Get help](/docs/forward/support#get-help).
Only paid accounts can raise the timeout limit.
## Infrastructure limits (https://www.tinybird.co/docs/forward/pricing/limits#infrastructure-limits)
The following infrastructure limits apply to all plans and workspaces.
| Description | Limit |
| --- | --- |
| Concurrent connections per IP address | 250 active connections from the same IP address per region |
If you exceed the concurrent connection limit, requests are throttled and you receive *HTTP 429 Too Many Requests* response codes. This limit is enforced to ensure service availability for all users.
This limit applies per region. If you need higher connection limits, consider distributing your traffic across multiple IP addresses.
## Publishing limits (https://www.tinybird.co/docs/forward/pricing/limits#publishing-limits)
The following publishing limits apply to all workspaces in an organization, regardless of the plan.
### Materialized Views limits (https://www.tinybird.co/docs/forward/pricing/limits#materialized-views-limits)
No numerical limits. Certain operations are [inadvisable when using Materialized Views](/docs/forward/core-concepts/materialized-views#limitations).
### Copy pipe limits (https://www.tinybird.co/docs/forward/pricing/limits#copy-pipe-limits)
Copy Pipes have the following limits per Workspace, depending on your billing plan:
| Plan | Active jobs (running or queued) | Execution time | Minimum scheduled frequency | Copy Pipes |
| --- | --- | --- | --- | --- |
| Free | 1 | 20 s | Once an hour | 20 |
| Developer | 3 | 30 s | Up to every 10 minutes | 100 |
| Enterprise | 6 | 50% of the scheduling period, 30 minutes max | Up to every minute | 200 |
The 6 active Copy Pipe jobs includes:
- 2 concurrently running jobs, and
- 4 additional jobs in the queue (pending execution).
This limit ensures fair resource usage and system stability across all users. If your workflows require higher throughput, contact [Tinybird support](/docs/forward/support) so the team can evaluate your use case and scale as needed.
### Sink pipe limits (https://www.tinybird.co/docs/forward/pricing/limits#sink-pipe-limits)
Sink Pipes have the following limits, depending on your billing plan:
| Plan | Sink Pipes per Workspace | Execution time | Frequency | Memory usage per query | Active jobs (running or queued) |
| --- | --- | --- | --- | --- | --- |
| Developer | 3 | 30 s | Up to every 10 min | 10 GB | 3 |
| Enterprise | 10 | 300 s | Up to every minute | 10 GB | 6 |
## Delete limits (https://www.tinybird.co/docs/forward/pricing/limits#delete-limits)
Delete jobs have the following limits, depending on your billing plan:
| Plan | Active delete jobs per Workspace |
| --- | --- |
| Free | 1 |
| Developer | 3 |
| Enterprise | 6 |
## Next steps (https://www.tinybird.co/docs/forward/pricing/limits#next-steps)
- [Pricing plans](./): Overview of Free, Developer, SaaS, and Enterprise.
---
URL: https://www.tinybird.co/docs/forward/pricing/free
Last update: 2026-07-31T11:10:11.000Z
Content:
---
title: "Free plan \| Tinybird"
theme-color: "#171612"
description: "The Tinybird Free plan — what's included, limits, support, and how to upgrade."
inkeep:version: "forward"
---
# Free plan (https://www.tinybird.co/docs/forward/pricing/free#free-plan)
The Free plan provides a production-grade instance of Tinybird with no time limit: ingest connectors, real-time querying, and managed endpoints are all available. It runs on **shared infrastructure**.
## What's included (https://www.tinybird.co/docs/forward/pricing/free#whats-included)
| Resource | Included |
| --- | --- |
| vCPU | 0.25 vCPU (with burst up to 3x) |
| Requests | 1,000 requests per day per organization |
| Storage | 10 GB |
| Max threads | 1 |
| LLM credits | $25 per billing cycle |
## Resource limits (https://www.tinybird.co/docs/forward/pricing/free#resource-limits)
- **vCPU**: 0.25 vCPU seconds per second, with up to 3x burst. This is a hard limit. Requests that exceed the burst capacity return HTTP 429.
- **Max threads**: Queries through API Endpoints and the Query API use at most 1 thread. See [Max threads](https://www.tinybird.co/docs/forward/pricing/free#max-threads).
- **LLM usage**: Soft stop when you exhaust the $25 included credits. LLM requests stop until the next billing cycle. See [LLM usage](https://www.tinybird.co/docs/forward/pricing/free#llm-usage).
For Workspace, ingestion, query, and publishing limits, see [Limits](./limits).
### Max threads (https://www.tinybird.co/docs/forward/pricing/free#max-threads)
Max threads is the maximum number of concurrent threads that execute queries in API Endpoints and Query API calls. The Free plan limits usage to 1 thread.
### LLM usage (https://www.tinybird.co/docs/forward/pricing/free#llm-usage)
Large Language Model usage, or LLM usage, in AI tools within MCP and Explorations UI includes $25 of credits per billing cycle. When you exhaust the included credits, LLM requests stop until the next billing cycle.
You can monitor usage through the `tinybird.llm_usage` Service Data Source.
## Support (https://www.tinybird.co/docs/forward/pricing/free#support)
Support is available through the [Community Slack](/docs/community), monitored by the Tinybird team on a best-effort basis.
## Upgrade (https://www.tinybird.co/docs/forward/pricing/free#upgrade)
To upgrade to a Developer plan:
1. Go to **Organization settings** → **Billing**.
2. Select **Upgrade**.
3. Select your vCPU size and fill out the details.
4. Confirm the upgrade.
To upgrade to a SaaS or Enterprise plan, [contact sales](https://www.tinybird.co/contact-us).
## Next steps (https://www.tinybird.co/docs/forward/pricing/free#next-steps)
- [Developer plan](./developer) — The next step up from Free.
- [Limits](./limits): Platform and operational limits.
---
URL: https://www.tinybird.co/docs/forward/pricing/enterprise
Last update: 2026-08-07T12:15:37.000Z
Content:
---
title: "Enterprise plan \| Tinybird"
theme-color: "#171612"
description: "The Tinybird Enterprise plan. Dedicated ClickHouse® cluster, replica SKUs, credit-based billing, and Standard or Premier support."
inkeep:version: "forward"
---
# Enterprise plan (https://www.tinybird.co/docs/forward/pricing/enterprise#enterprise-plan)
The Enterprise plan provides **dedicated infrastructure**: your own isolated ClickHouse® cluster where you are the only customer. Your queries and ingestion run on your own replicas with no resource sharing from other customers. This gives you compute and network isolation and fine-tuned control over the infrastructure required for your workloads.
Minimum storage for Enterprise plans is 1 TB. A support package is mandatory. Billing is credit-based with a minimum 12-month commitment.
## Managing your cluster (https://www.tinybird.co/docs/forward/pricing/enterprise#managing-your-cluster)
- **Add or remove replicas**: Use **Organization settings** → **Plan & Billing** to add replicas, remove them, or change their size.
- **Traffic distribution**: Set read and write weights per replica to control how query and ingestion traffic is distributed. See [Cluster management](/docs/forward/guides/cluster-management#how-weight-distribution-works).
- **Monitoring**: Monitor cluster usage, remaining credits, and current commitment through your organization's dashboard. See [Enterprise infrastructure monitoring](/docs/forward/core-concepts/organizations#dedicated-infrastructure-monitoring).
For other adjustments to your Enterprise plan or to discuss moving to SaaS (shared infrastructure), [contact sales](https://www.tinybird.co/contact-us).
## Billing (https://www.tinybird.co/docs/forward/pricing/enterprise#billing)
Enterprise plans are billed using credits based on the following SKUs:
| SKU | What it covers | Billing type |
| --- | --- | --- |
| Dedicated ClickHouse® cluster CPUs | The total active CPUs in your dedicated ClickHouse® cluster. Billed by CPU cores at any given time. | Usage based. Rates may vary by region. |
| On-demand CPUs | Temporary compute for jobs that run outside the baseline cluster, for example populate jobs. Billed by CPU cores and duration. | Usage based. Rates may vary by region. |
| Real-time ingestion | Data ingested via HFI API, Kafka, and DynamoDB connectors. Measured as compressed bytes expressed as CPU core-hours (P95). | Usage based. Rates may vary by region and contract. |
| Workers | Batch and async operations: batch ingestion, populates, deletes, replaces, copies. Billed by peak concurrent workers per day. | Usage based. |
| Data storage | Persistent data in ClickHouse® per Data Source, including primary and quarantine data. Based on peak daily storage. | Usage based. |
| Backup storage | ClickHouse® backups retained for your organization. Based on total backup size retained per day. | Usage based. |
| Data transfer | Data transferred out of the platform via Sinks. Rate varies by transfer type (intra vs. inter cloud provider/region) and region. | Usage based. |
| Private networking | Private network connectivity for your Workspaces, such as private links. Billed as a markup on the underlying connectivity cost. | Usage based. |
| Support | Standard or Premier support as agreed in your contract. Standard is included. Not usage-based. | Fixed. |
| Enterprise authentication | SSO with SAML. Flat daily rate when turned on; no charge when off. | Fixed daily rate. |
| Private region | Private region deployment. Flat daily rate when turned on; no charge when off. | Fixed daily rate. |
| LLM usage | LLM usage in AI tools within MCP and Explorations UI. Includes $200 per billing cycle. | Usage based when usage exceeds the included credits. |
The minimum commitment period is 12 months. Bills are issued monthly based on credits consumed (first to last day of the month). Invoices are issued upon credits purchase, which can happen when signing the contract or when purchasing additional credits.
Where applicable, rates depend on your [region](/docs/api-reference#regions-and-endpoints) and contract.
### Dedicated ClickHouse® cluster CPUs (https://www.tinybird.co/docs/forward/pricing/enterprise#dedicated-clickhouse-cluster-cpus)
The Dedicated ClickHouse® cluster CPUs SKU covers the cost of running the dedicated ClickHouse® cluster that Tinybird provisions exclusively for your organization. This cluster stores your data, runs your queries, and provides isolated compute and storage according to your topology and capacity. It includes ClickHouse® nodes, coordination services when applicable, and the underlying compute resources. Pricing depends on the cluster's [region](/docs/api-reference#regions-and-endpoints).
You can upgrade or downgrade your cluster at any time, immediately affecting billing for this SKU. See [Cluster management](/docs/forward/guides/cluster-management).
Billing unit: CPU cores/hour.
### On-demand CPUs (https://www.tinybird.co/docs/forward/pricing/enterprise#on-demand-cpus)
The On-demand CPUs SKU covers temporary compute capacity for jobs that need dedicated compute instances. This currently applies to populate jobs that run on on-demand compute. These jobs run outside the baseline resources of your dedicated ClickHouse® cluster and use short-lived, single-host instances during the job. Billing is based on the CPU cores used and the duration of the on-demand instance. Rates may vary by [region](/docs/api-reference#regions-and-endpoints). See also [On-demand CPUs](./#on-demand-cpus) on the plans hub.
Billing unit: OnDemand CPU cores/hour.
### Real-time ingestion (https://www.tinybird.co/docs/forward/pricing/enterprise#real-time-ingestion)
The Real-time ingestion SKU covers real-time data ingestion into the platform. This includes data ingested using the High-Frequency Ingestion API, or HFI API, Kafka connectors, and DynamoDB connectors. This SKU doesn't include other ingestion mechanisms. Billing is based on the compute capacity required to sustain your ingestion throughput. Throughput is measured as decompressed bytes sent to the platform and expressed as CPU core-hours using P95 over time windows. Rates may vary by [region](/docs/api-reference#regions-and-endpoints) and contract.
Only data that arrives at Tinybird is considered for this SKU. After the data is written, the Dedicated ClickHouse® cluster CPUs SKU covers downstream operations such as Materialized Views and Copy Pipes.
Billing unit: MB/s/CPU/hour using P95 over 15 min windows.
### Workers (https://www.tinybird.co/docs/forward/pricing/enterprise#workers)
The Workers SKU covers batch and asynchronous operations that the platform runs on your behalf. These operations include batch ingestion from sources such as S3, CSV, or Parquet, populate jobs, deletes, replaces, copies, and other job processor operations, but exclude real-time ingestion. Billing is based on the maximum number of jobs you can run in parallel, called concurrent workers, during the billing period. For each day, the observed peak concurrency determines the number of workers charged at a fixed daily rate per worker.
Billing unit: number of concurrent workers.
### Data storage (https://www.tinybird.co/docs/forward/pricing/enterprise#data-storage)
The Data Storage SKU covers persistent data in ClickHouse® for your Workspaces. It accounts for the total amount of data on disk per Data Source, including both primary data and quarantine data. Storage is measured at the Data Source level according to peak storage usage during each billing period. Billing uses a daily storage rate according to the observed maximum storage.
Billing unit: TBs/day.
### Backup storage (https://www.tinybird.co/docs/forward/pricing/enterprise#backup-storage)
The Backup Storage SKU covers ClickHouse® backups for organizations on dedicated infrastructure. It accounts for the total size of backups retained for your organization during the billing period, including full and incremental backups, aggregated across all your dedicated clusters. Backup usage is measured at the organization level. Billing uses a daily storage rate according to the total backup size retained. For backup retention and related terms, see the [Terms and Conditions](https://www.tinybird.co/terms-and-conditions).
Billing unit: TBs/day.
### Data transfer (https://www.tinybird.co/docs/forward/pricing/enterprise#data-transfer)
The Data Transfer SKU covers data transferred out of the platform, primarily through [Sinks](/docs/forward/core-concepts/sinks) that export data to external storage. It accounts for data moved across cloud providers and regions. Intra transfers stay within the same provider and region, while inter transfers cross providers or regions. Usage is measured at the Workspace level and aggregated daily. The per-gigabyte billing rate varies by transfer type and [region](/docs/api-reference#regions-and-endpoints).
Billing unit: GBs transferred.
### Private networking (https://www.tinybird.co/docs/forward/pricing/enterprise#private-networking)
The Private Networking SKU covers activating and operating private network connectivity for your Workspaces, for example private links between services, regions, or infrastructure components that aren't exposed over the public internet. Billing is applied as a markup on the underlying connectivity cost.
Billing unit: GBs transferred.
### Support (https://www.tinybird.co/docs/forward/pricing/enterprise#support)
The Support SKU covers ongoing support services for your organization. Standard is included. Premier is available and adds a designated Customer Success engineer, prioritized SLA, on-call requests, monthly partnership meetings, and onboarding support. The applicable tier and price depend on your agreement and aren't usage-based. See [Support](../support) for full details.
### Enterprise authentication (https://www.tinybird.co/docs/forward/pricing/enterprise#enterprise-authentication)
The Enterprise authentication SKU covers activating Single Sign-On, or SSO, with SAML for your organization. This is a flat-rate SKU. When SAML is on, a fixed rate is charged regardless of usage. If SAML is off, no charges apply. Billing is a fixed daily rate for each day the feature is active.
### Private region (https://www.tinybird.co/docs/forward/pricing/enterprise#private-region)
The Private Region SKU covers activating a private region deployment for your organization. When Private Region is on, a flat rate is charged regardless of workload. If the feature is off, no charges apply. Billing is a fixed daily rate for each day the feature is active.
### LLM usage (https://www.tinybird.co/docs/forward/pricing/enterprise#llm-usage)
LLM usage in AI tools within MCP and Explorations UI includes $200 of credits per billing cycle. Usage beyond included credits is billed at cost.
## Support (https://www.tinybird.co/docs/forward/pricing/enterprise#support)
A support package is mandatory on Enterprise plans. Standard is included and provides priority email, live chat, and a dedicated customer Slack channel. Premier is available and adds a designated Customer Success engineer, prioritized SLA, on-call requests, monthly partnership meetings, and onboarding support. Your customer agreement defines your support plan, pricing, and terms. See [Support](../support) for full details.
## Next steps (https://www.tinybird.co/docs/forward/pricing/enterprise#next-steps)
- [Cluster management](/docs/forward/guides/cluster-management): Add replicas, select SKUs, set traffic weights.
- [SaaS plan](./saas) — Shared infrastructure alternative for 4+ vCPU workloads.
- [Limits](./limits): Platform and operational limits.
- [Regions and endpoints](/docs/api-reference#regions-and-endpoints): Available regions and regional pricing differences.
---
URL: https://www.tinybird.co/docs/forward/pricing/developer
Last update: 2026-08-07T12:15:37.000Z
Content:
---
title: "Developer plan \| Tinybird"
theme-color: "#171612"
description: "The Tinybird Developer plan — plan sizes, billing, resize, downgrade, and upgrade."
inkeep:version: "forward"
---
# Developer plan (https://www.tinybird.co/docs/forward/pricing/developer#developer-plan)
The Developer plan runs on **shared infrastructure** without a commitment. You select a plan size when signing up or changing plans. Each size has a fixed set of included resources. Developer plans include 25 GB of storage. Usage beyond this and other included amounts is billed as overage. Plans support up to 8 vCPUs. For more capacity on shared infrastructure, see [SaaS](./saas), or for a dedicated cluster, see [Enterprise](./enterprise).
Each organization is tied to a single region. You can create multiple workspaces within one organization, all covered by the same billing plan. If you need workspaces in a different region, you must create a separate organization, which requires its own plan.
## Billing (https://www.tinybird.co/docs/forward/pricing/developer#billing)
Developer plans are billed based on the following resources:
| Resource | Billing | Type |
| --- | --- | --- |
| vCPUs | Number of vCPUs available per second. More than 8 vCPUs require a SaaS or Enterprise plan. | Fixed minimum. Usage based when usage exceeds the vCPU included in the plan. |
| On-demand CPU | Temporary compute for jobs that run outside the baseline cluster, for example populate jobs. Billed by CPU cores and duration. | Usage based. Rates may vary by region. See [On-demand CPUs](./#on-demand-cpus). |
| Storage | Average of daily maximum usage of the compressed disk storage of all your data, in gigabytes. | Usage based when usage exceeds the 25 GB included in the plan. |
| Data transfer | When using [Sinks](/docs/api-reference/sink-pipes-api), usage is billed depending on the destination, which can be the same cloud provider and region as your Tinybird cluster, or a different one. | Usage based. |
| LLM usage | LLM usage in AI tools within MCP and Explorations UI. Includes $100 per billing cycle. | Usage based when usage exceeds the $100 included in the plan. |
Developer plans are billed monthly. Each invoice contains fixed costs plus usage-based costs calculated from the previous period. The billing period starts on the subscription date.
The same vCPU and storage overage rates apply in every region. See [Pricing](https://www.tinybird.co/pricing) for current rates.
### vCPU seconds and overages (https://www.tinybird.co/docs/forward/pricing/developer#vcpu-seconds)
Each plan includes a baseline **vCPU** allocation. For example, if your plan includes 1 vCPU, you can consume 1 vCPU second per second without overage.
Usage exceeding your plan allocation counts as **vCPU overage** and is billed at a fixed rate per second. This calculation includes both read and write operations.
### vCPU burst mode (https://www.tinybird.co/docs/forward/pricing/developer#vcpu-burst-mode)
You can use burst mode to temporarily exceed your base vCPU allocation:
- Developer plans S-1/4, S-1/2, S-1, and S-2 can burst up to 3× their vCPU capacity.
- Plans S-4 and higher can burst up to 2×.
Usage beyond the base allowance incurs vCPU overage charges. Requests that exceed the burst capacity return HTTP 429 from API Endpoints or the Query API.
For batch operations such as populates or copies, the operation can run until it hits a platform limit such as maximum available memory.
### Max threads (https://www.tinybird.co/docs/forward/pricing/developer#max-threads)
Max threads is the maximum number of concurrent threads used to execute queries in API Endpoints and Query API calls. The number of max threads available depends on your Developer plan size.
While more threads can improve concurrent query processing, final performance also depends on your vCPU limit and query complexity.
### Data storage (https://www.tinybird.co/docs/forward/pricing/developer#data-storage)
You are charged for persistent data stored in ClickHouse® per Data Source. Developer plans include 25 GB. Storage beyond the included amount is billed at a fixed rate per GB. Rates aren't tied to region.
### Data transfer (https://www.tinybird.co/docs/forward/pricing/developer#data-transfer)
The data transfer SKU covers data sent out of the platform, primarily through [Sinks](/docs/forward/core-concepts/sinks). Data transfer is charged per gigabyte. The rate depends on the transfer type. Intra transfers stay within the same cloud provider and region. Inter transfers cross providers or regions.
### LLM usage (https://www.tinybird.co/docs/forward/pricing/developer#llm-usage)
LLM usage in AI tools within MCP and Explorations UI includes $100 of credits per billing cycle. Usage beyond included credits is billed at cost.
You can monitor usage through the `tinybird.llm_usage` Service Data Source.
### Self-service (https://www.tinybird.co/docs/forward/pricing/developer#self-service)
After adding billing details, you can select and resize any Developer plan up to 8 vCPUs in self-service mode. To use more than 8 vCPUs, upgrade through sales to SaaS for shared infrastructure or Enterprise for dedicated infrastructure.
## Support (https://www.tinybird.co/docs/forward/pricing/developer#support)
Community Slack is available to everyone. Developer also includes email support. See [Support](../support) for details.
## Resize and upgrade (https://www.tinybird.co/docs/forward/pricing/developer#resize-and-upgrade)
You can resize Developer plans at any time. Increases and reductions in capacity take effect instantly.
To resize your plan:
1. Go to **Organization settings** → **Billing**.
2. Select **Change size**.
3. Select a bigger or a smaller size. Self-service plans can be up to 8 vCPUs.
4. Confirm the resize.
To upgrade to a SaaS or Enterprise plan, [contact sales](https://www.tinybird.co/contact-us).
## Downgrade (https://www.tinybird.co/docs/forward/pricing/developer#downgrade)
You can downgrade back to the Free plan by canceling your Developer plan.
To cancel your Developer plan:
1. Go to **Organization settings** → **Billing**.
2. Select the **More actions (⋯)** button and then select **Cancel plan**.
3. Review the information and confirm. Remaining costs are charged automatically upon cancellation.
After canceling your Developer plan, Free plan limits apply to your Workspaces.
## Next steps (https://www.tinybird.co/docs/forward/pricing/developer#next-steps)
- [Limits](./limits): Platform and operational limits.
- [SaaS plan](./saas): Shared infrastructure with 4+ vCPUs and a commitment.
- [Enterprise plan](./enterprise) — Dedicated ClickHouse® cluster.
---
URL: https://www.tinybird.co/docs/forward/monitoring/service-datasources
Last update: 2026-08-04T10:22:54.000Z
Content:
---
title: "Service data sources \| Tinybird"
theme-color: "#171612"
description: "In addition to the data sources you upload, Tinybird provides other \"Service data sources\" that allow you to inspect what's going on in your account."
inkeep:version: "forward"
---
# Service data sources (https://www.tinybird.co/docs/forward/monitoring/service-datasources#service-data-sources)
Tinybird provides Service data sources that you can use to inspect what's going on in your Tinybird account, diagnose issues, monitor usage, and so on.
For example, you can get real time stats about API calls or a log of every operation over your data sources. This is similar to using system tables in a database, although Service data sources contain information about the usage of the service itself.
Queries made to Service data sources are free of charge and don't count towards your usage. However, calls to API endpoints that use Service data sources do count towards API rate limits. See [Billing](../pricing).
## Considerations (https://www.tinybird.co/docs/forward/monitoring/service-datasources#considerations)
- You can't use Service data sources in materialized view queries.
- Pass dynamic query parameters to API endpoints to then query Service data sources.
## Service data sources (https://www.tinybird.co/docs/forward/monitoring/service-datasources#service-data-sources-2)
The following Service data sources are available.
### tinybird.pipe_stats_rt (https://www.tinybird.co/docs/forward/monitoring/service-datasources#tinybird-pipe-stats-rt)
Contains information about all requests made to your [API endpoints](/docs/forward/core-concepts/api-endpoints) in real time. This data source has a TTL of 7 days. If you need to query data older than 7 days you must use the aggregated by day data available at [tinybird.pipe_stats](https://www.tinybird.co/docs/forward/monitoring/service-datasources#tinybird-pipe-stats).
Calls made against Service data sources are not logged and don't count towards usage limits.
| Field | Type | Description |
| --- | --- | --- |
| `start_datetime` | `DateTime` | API call start date and time. |
| `pipe_id` | `String` | Pipe ID as returned in the [Pipes API](/docs/api-reference/pipe-api). The value is `query_api` for a Query API request. |
| `pipe_name` | `String` | Pipe name as returned in the [Pipes API](/docs/api-reference/pipe-api). The value is `query_api` for a Query API request. |
| `duration` | `Float` | API call duration, in seconds. |
| `read_bytes` | `UInt64` | API call read data, in bytes. |
| `read_rows` | `UInt64` | API call rows read. |
| `result_rows` | `UInt64` | Rows returned by the API call. |
| `url` | `String` | URL (`token` param is removed for security reasons). |
| `error` | `UInt8` | `1` if query returned error, else `0`. |
| `request_id` | `String` | API call identifier returned in `x-request-id` header. Format is ULID string. |
| `token` | `String` | API call token identifier used. |
| `token_name` | `String` | API call token name used. |
| `status_code` | `Int32` | API call returned status code. |
| `method` | `String` | API call method POST or GET. |
| `parameters` | `Map(String, String)` | API call parameters used. |
| `release` | `String` | Semantic version of the release (deprecated). |
| `user_agent` | `Nullable(String)` | User Agent HTTP header from the request. |
| `resource_tags` | `Array(String)` | Tags associated with the pipe when the request was made. |
| `cpu_time` | `Float` | CPU time used by the query, in seconds. |
| `memory_usage` | `UInt64` | Memory consumption by the query, in bytes. High memory usage indicates the query may not be optimized efficiently. |
### tinybird.pipe_stats (https://www.tinybird.co/docs/forward/monitoring/service-datasources#tinybird-pipe-stats)
Aggregates the request stats in [tinybird.pipe_stats_rt](https://www.tinybird.co/docs/forward/monitoring/service-datasources#tinybird-pipe-stats-rt) by day.
Calls made against Service data sources are not logged and don't count towards usage limits.
| Field | Type | Description |
| --- | --- | --- |
| `date` | `Date` | Request date and time. |
| `pipe_id` | `String` | Pipe ID as returned in the [Pipes API](/docs/api-reference/pipe-api). |
| `pipe_name` | `String` | Name of the pipe. |
| `view_count` | `UInt64` | Request count. |
| `error_count` | `UInt64` | Number of requests with error. |
| `avg_duration_state` | `AggregateFunction(avg, Float32)` | Average duration state, in seconds (see [Querying _state columns](https://www.tinybird.co/docs/forward/monitoring/service-datasources#querying-state-columns)). |
| `quantile_timing_state` | `AggregateFunction(quantilesTiming(0.9, 0.95, 0.99), Float64)` | 0.9, 0.95 and 0.99 quantiles state. Time, in milliseconds (see [Querying _state columns](https://www.tinybird.co/docs/forward/monitoring/service-datasources#querying-state-columns)). |
| `read_bytes_sum` | `UInt64` | Total bytes read. |
| `read_rows_sum` | `UInt64` | Total rows read. |
| `resource_tags` | `Array(String)` | All the tags associated with the resource when the aggregated requests were made. |
### tinybird.block_log (https://www.tinybird.co/docs/forward/monitoring/service-datasources#tinybird-block-log)
The data source contains details about how Tinybird ingests data into your data sources. You can use this Service data source to spot problematic parts of your data.
| Field | Type | Description |
| --- | --- | --- |
| `timestamp` | `DateTime` | Date and time of the block ingestion. |
| `import_id` | `String` | Id of the import operation. |
| `job_id` | `Nullable(String)` | Id of the job that ingested the block of data, if it was ingested by URL. In this case, `import_id` and `job_id` must have the same value. |
| `request_id` | `String` | Id of the request that performed the operation. In this case, `import_id` and `job_id` must have the same value. Format is ULID string. |
| `source` | `String` | Either the URL or `stream` or `body` keywords. |
| `block_id` | `String` | Block identifier. You can cross this with the `blocks_ids` column from the [tinybird.datasources_ops_log](https://www.tinybird.co/docs/forward/monitoring/service-datasources#tinybird-datasources-ops-log) Service data source. |
| `status` | `String` | `done` \| `error`. |
| `datasource_id` | `String` | data source consistent id. |
| `datasource_name` | `String` | data source name when the block was ingested. |
| `start_offset` | `Nullable(Int64)` | The starting byte of the block, if the ingestion was split, where this block started. |
| `end_offset` | `Nullable(Int64)` | If split, the ending byte of the block. |
| `rows` | `Nullable(Int32)` | How many rows it ingested. |
| `parser` | `Nullable(String)` | Whether the native block parser or falling back to row by row parsing is used. |
| `quarantine_lines` | `Nullable(UInt32)` | If any, how many rows went into the quarantine data source. |
| `empty_lines` | `Nullable(UInt32)` | If any, how many empty lines were skipped. |
| `bytes` | `Nullable(UInt32)` | How many bytes the block had. |
| `processing_time` | `Nullable(Float32)` | How long it took, in seconds. |
| `processing_error` | `Nullable(String)` | Detailed message in case of error. |
When Tinybird ingests data from a URL, it splits the download in several requests, resulting in different ingestion blocks. The same happens when the data upload happens with a multipart request.
### tinybird.datasources_ops_log (https://www.tinybird.co/docs/forward/monitoring/service-datasources#tinybird-datasources-ops-log)
Contains all operations performed to your data sources. Tinybird tracks the following operations:
| Event | Description |
| --- | --- |
| `create` | A data source is created. |
| `append` | Append operation. |
| `append-hfi` | Append operation using the [High-frequency Ingestion API](/docs/forward/ingest-data/events-api). |
| `append-kafka` | Append operation using the [Kafka Connector](/docs/forward/ingest-data/connectors/kafka). |
| `replace` | A replace operation took place in the data source. |
| `delete` | A data source is deleted. |
| `delete_data` | A delete operation took place in the data source. |
| `truncate` | A truncate operation took place in the data source. |
| `rename` | The data source was renamed. |
| `populateview-queued` | A populate operation was queued for execution. |
| `populateview` | A finished populate operation (up to 8 hours after it started). |
| `copy` | A copy operation took place in the data source. |
| `alter` | An alter operation took place in the data source. |
Materializations are logged with same `event_type` and `operation_id` as the operation that triggers them. You can track the materialization pipe with `pipe_id` and `pipe_name`.
Tinybird logs all operations with the following information in this data source:
| Field | Type | Description |
| --- | --- | --- |
| `timestamp` | `DateTime` | Date and time when the operation started. |
| `event_type` | `String` | Operation being logged. |
| `operation_id` | `String` | Groups rows affected by the same operation. Useful for checking materializations triggered by an append operation. |
| `datasource_id` | `String` | Id of your data source. The data source id is consistent after renaming operations. You should use the id when you want to track name changes. |
| `datasource_name` | `String` | Name of your data source when the operation happened. |
| `result` | `String` | `ok` \| `error` |
| `elapsed_time` | `Float32` | How much time the operation took, in seconds. |
| `error` | `Nullable(String)` | Detailed error message if the result was error. |
| `import_id` | `Nullable(String)` | Id of the import operation, if data has been ingested using one of the following operations: `create`, `append` or `replace` |
| `job_id` | `Nullable(String)` | Id of the job that performed the operation, if any. If data has been ingested, `import_id` and `job_id` must have the same value. |
| `request_id` | `String` | Id of the request that performed the operation. If data has been ingested, `import_id` and `request_id` must have the same value. Format is ULID string. |
| `rows` | `Nullable(UInt64)` | How many rows the operations affected. This depends on `event_type`: for the `append` event, how many rows got inserted; for `delete` or `truncate` events, how many rows the data source had; for `replace`, how many rows the data source has after the operation. |
| `rows_quarantine` | `Nullable(UInt64)` | How many rows went into the quarantine data source, if any. |
| `blocks_ids` | `Array(String)` | List of blocks ids used for the operation. See the [tinybird.block_log](https://www.tinybird.co/docs/forward/monitoring/service-datasources#tinybird-block-log) Service data source for more details. |
| `options` | `Nested(Names String, Values String)` | Tinybird stores key-value pairs with extra information for some operations. For the `replace` event, Tinybird uses the `rows_before_replace` key to track how many rows the data source had before the replacement happened, the `replace_condition` key shows what condition was used. For `append` and `replace` events, Tinybird stores the data `source`, for example the URL, or body/stream keywords. For `rename` event, `old_name` and `new_name`. For `populateview` you can find there the whole populate `job` metadata as a JSON string. For `alter` events, Tinybird stores `operations`, and dependent pipes as `dependencies` if they exist. |
| `read_bytes` | `UInt64` | Read bytes in the operation. |
| `read_rows` | `UInt64` | Read rows in the operation. |
| `written_rows` | `UInt64` | Written rows in the operation. |
| `written_bytes` | `UInt64` | Written bytes in the operation. |
| `written_rows_quarantine` | `UInt64` | Quarantined rows in the operation. |
| `written_bytes_quarantine` | `UInt64` | Quarantined bytes in the operation. |
| `pipe_id` | `String` | If present, Materialization Pipe ID as returned in the [Pipes API](/docs/api-reference/pipe-api). |
| `pipe_name` | `String` | If present, Materialization Pipe name as returned in the [Pipes API](/docs/api-reference/pipe-api). |
| `release` | `String` | Semantic version of the release (deprecated). |
| `resource_tags` | `Array(String)` | Tags associated with the pipe when the request was made. |
| `cpu_time` | `Float32` | CPU time used by the operation, in seconds. |
| `memory_usage` | `UInt64` | Memory consuptiom by the operation, in bytes. |
### tinybird.datasource_ops_stats (https://www.tinybird.co/docs/forward/monitoring/service-datasources#tinybird-datasource-ops-stats)
Data from `datasource_ops_log`, aggregated by day.
| Field | Type | Description |
| --- | --- | --- |
| `event_date` | `Date` | Date of the event. |
| `workspace_id` | `String` | Unique identifier for the workspace. |
| `event_type` | `String` | Name of your data source. |
| `pipe_id` | `String` | Identifier of the pipe. |
| `pipe_name` | `String` | Name of the pipe. |
| `error_count` | `UInt64` | Number of requests with an error. |
| `executions` | `UInt64` | Number of executions. |
| `avg_elapsed_time_state` | `Float32` | Average time spent in elapsed state. |
| `quantiles_state` | `Float32` | 0.9, 0.95 and 0.99 quantiles state. Time in milliseconds (see [Querying _state columns](https://www.tinybird.co/docs/forward/monitoring/service-datasources#querying-state-columns)). |
| `read_bytes` | `UInt64` | Read bytes in the operation. |
| `read_rows` | `UInt64` | Read rows in the Sink operation. |
| `written_rows` | `UInt64` | Written rows in the Sink operation. |
| `read_bytes` | `UInt64` | Read bytes in the operation. |
| `written_bytes` | `UInt64` | Written bytes in the operation. |
| `written_rows_quarantine` | `UInt64` | Quarantined rows in the operation. |
| `written_bytes_quarantine` | `UInt64` | Quarantined bytes in the operation. |
| `resource_tags` | `Array(String)` | Tags associated with the pipe when the request was made. |
### tinybird.endpoint_errors (https://www.tinybird.co/docs/forward/monitoring/service-datasources#tinybird-endpoint-errors)
It provides the last 30 days errors of your published endpoints. Tinybird logs all errors with additional information in this data source.
| Field | Type | Description |
| --- | --- | --- |
| `start_datetime` | `DateTime` | Date and time when the API call started. |
| `request_id` | `String` | The id of the request that performed the operation. Format is ULID string. |
| `pipe_id` | `String` | If present, Pipe ID as returned in the [Pipes API](/docs/api-reference/pipe-api). |
| `pipe_name` | `String` | If present, Pipe name as returned in the [Pipes API](/docs/api-reference/pipe-api). |
| `params` | `Nullable(String)` | URL query params included in the request. |
| `url` | `Nullable(String)` | URL path. |
| `status_code` | `Nullable(Int32)` | HTTP error code. |
| `error` | `Nullable(String)` | Error message. |
| `resource_tags` | `Array(String)` | Tags associated with the pipe when the request was made. |
### tinybird.kafka_ops_log (https://www.tinybird.co/docs/forward/monitoring/service-datasources#tinybird-kafka-ops-log)
Contains all operations performed to your Kafka data sources during the last 30 days.
| Field | Type | Description |
| --- | --- | --- |
| `timestamp` | `DateTime` | Date and time when the operation took place. |
| `datasource_id` | `String` | Id of your data source. The data source id is consistent after renaming operations. You should use the id when you want to track name changes. |
| `topic` | `String` | Kafka topic. |
| `partition` | `Int16` | Partition number, or `-1` for all partitions. |
| `msg_type` | `String` | 'info' for regular messages, 'warning' for issues related to the user's Kafka cluster, deserialization or materialized views, and 'error' for other issues. |
| `lag` | `Int64` | Number of messages behind for the partition. This is the difference between the high-water mark and the last commit offset. |
| `processed_messages` | `Int32` | Messages processed for a topic and partition. |
| `processed_bytes` | `Int32` | Amount of bytes processed. |
| `committed_messages` | `Int32` | Messages ingested for a topic and partition. |
| `time_read` | `Float32` | Time taken to read messages from Kafka, in seconds. |
| `time_process` | `Float32` | Time taken to process the messages, in seconds. |
| `time_write` | `Float32` | Time taken to write messages to ClickHouse®, in seconds. |
| `msg` | `String` | Information in the case of warnings or errors. Empty otherwise. |
### tinybird.datasources_storage (https://www.tinybird.co/docs/forward/monitoring/service-datasources#tinybird-datasources-storage)
Contains stats about your data sources storage.
Tinybird logs maximum values per hour, the same as when it calculates storage consumption.
| Field | Type | Description |
| --- | --- | --- |
| `datasource_id` | `String` | Id of your data source. The data source id is consistent after renaming operations. You should use the id when you want to track name changes. |
| `datasource_name` | `String` | Name of your data source. |
| `timestamp` | `DateTime` | When storage was tracked. By hour. |
| `bytes` | `UInt64` | Max number of bytes the data source has, not including quarantine. |
| `rows` | `UInt64` | Max number of rows the data source has, not including quarantine. |
| `bytes_quarantine` | `UInt64` | Max number of bytes the data source has in quarantine. |
| `rows_quarantine` | `UInt64` | Max number of rows the data source has in quarantine. |
### tinybird.releases_log (deprecated) (https://www.tinybird.co/docs/forward/monitoring/service-datasources#tinybird-releases-log-deprecated)
Contains operations performed to your releases. Tinybird tracks the following operations:
| Event | Description |
| --- | --- |
| `init` | First Release is created on Git sync. |
| `override` | Release commit is overridden. `tb init --override-commit {{commit}}`. |
| `deploy` | Resources from a commit are deployed to a Release. |
| `preview` | Release status is changed to preview. |
| `promote` | Release status is changed to live. |
| `post` | Resources from a commit are deployed to the live Release. |
| `rollback` | Rollback is done a previous Release is now live. |
| `delete` | Release is deleted. |
Tinybird logs all operations with additional information in this data source.
| Field | Type | Description |
| `timestamp` | `DateTime64` | Date and time when the operation took place. |
| `event_type` | `String` | Name of your data source. |
| `semver` | `String` | Semantic version identifies a release. |
| `commit` | `String` | Git sha commit related to the operation. |
| `token` | `String` | API call token identifier used. |
| `token_name` | `String` | API call token name used. |
| `result` | `String` | `ok` \| `error` |
| `error` | `String` | Detailed error message. |
### tinybird.sinks_ops_log (https://www.tinybird.co/docs/forward/monitoring/service-datasources#tinybird-sinks-ops-log)
Contains all operations performed to your Sink pipes.
| Field | Type | Description |
| `timestamp` | `DateTime64` | Date and time when the operation took place. |
| `service` | `LowCardinality(String)` | Type of Sink (GCS, S3, and so on). |
| `pipe_id` | `String` | The ID of the Sink pipe. |
| `pipe_name` | `String` | the name of the Sink pipe. |
| `token_name` | `String` | Token name used. |
| `result` | `LowCardinality(String)` | `ok` \| `error` |
| `error` | `Nullable(String)` | Detailed error message. |
| `elapsed_time` | `Float64` | The duration of the operation in seconds. |
| `job_id` | `Nullable(String)` | ID of the job that performed the operation, if any. |
| `read_rows` | `UInt64` | Read rows in the Sink operation. |
| `written_rows` | `UInt64` | Written rows in the Sink operation. |
| `read_bytes` | `UInt64` | Read bytes in the operation. |
| `written_bytes` | `UInt64` | Written bytes in the operation. |
| `output` | `Array(String)` | The outputs of the operation. In the case of writing to a bucket, the name of the written files. |
| `parameters` | `Map(String, String)` | The parameters used. Useful to debug the parameter query values. |
| `options` | `Map(String, String)` | Extra information. You can access the values with `options['key']` where key is one of: file_template, file_format, file_compression, bucket_path, execution_type. |
| `cpu_time` | `Float64` | The CPU time used by the sinks, in seconds. |
### tinybird.data_transfer (https://www.tinybird.co/docs/forward/monitoring/service-datasources#tinybird-data-transfer)
Stats of data transferred per hour by a workspace.
| Field | Type | Description |
| `timestamp` | `DateTime` | Date and time data transferred is tracked. By hour. |
| `event` | `LowCardinality(String)` | Type of operation generated the data (ie. `sink`). |
| `origin_provider` | `LowCardinality(String)` | Provider data was transferred from. |
| `origin_region` | `LowCardinality(String)` | Region data was transferred from. |
| `destination_provider` | `LowCardinality(String)` | Provider data was transferred to. |
| `destination_region` | `LowCardinality(String)` | Region data was transferred to. |
| `kind` | `LowCardinality(String)` | `intra` \| `inter` depending if the data moves within or outside the region. |
### tinybird.llm_usage (https://www.tinybird.co/docs/forward/monitoring/service-datasources#tinybird-llm-usage)
Contains LLM usage metrics from Tinybird AI features including token consumption, costs, and model usage for each request in the workspace.
| Field | Type | Description |
| --- | --- | --- |
| `start_time` | `DateTime` | Request start timestamp. |
| `end_time` | `DateTime` | Request end timestamp. |
| `organization_id` | `String` | Organization identifier. |
| `organization_name` | `String` | Organization name. |
| `workspace_id` | `String` | Workspace identifier. |
| `workspace_name` | `String` | Workspace name. |
| `user_email` | `String` | email of the user who made the request. |
| `request_id` | `String` | Unique identifier for the LLM request. |
| `prompt_tokens` | `UInt32` | Number of tokens in the prompt. |
| `completion_tokens` | `UInt32` | Number of tokens in the completion. |
| `total_tokens` | `UInt32` | Total number of tokens used. |
| `duration` | `Float32` | Request duration in seconds. |
| `cost` | `Float32` | Cost of the LLM request. |
| `origin` | `String` | Origin of the request. |
| `feature` | `String` | Tinybird AI feature used. |
### tinybird.hfi_requests (https://www.tinybird.co/docs/forward/monitoring/service-datasources#tinybird-hfi-requests)
Contains telemetry for High-Frequency Ingestion (HFI) requests in your workspace.
| Field | Type | Description |
| --- | --- | --- |
| `event_time` | `DateTime` | Time slot for the ingestion event. |
| `workspace_id` | `String` | Workspace identifier. |
| `datasource_id` | `String` | Data Source identifier. |
| `successful_requests` | `UInt64` | Number of successful ingestion requests. |
| `error_requests` | `UInt64` | Number of failed ingestion requests. |
| `rows_total` | `UInt64` | Total number of rows received. |
| `rows_successful` | `UInt64` | Number of successfully ingested rows. |
| `rows_quarantined` | `UInt64` | Number of quarantined rows. |
| `bytes_received` | `UInt64` | Number of bytes received. |
| `bytes_decompressed` | `UInt64` | Number of bytes after decompression. |
| `ch_written_rows` **(deprecated)** | `UInt64` | **Deprecated.** |
| `ch_written_bytes` **(deprecated)** | `UInt64` | **Deprecated.** |
| `ch_cpu_time` **(deprecated)** | `Float32` | **Deprecated.** |
| `rate_limited` | `UInt8` | `1` if the request was rate-limited, `0` otherwise. |
| `ips` | `Array(String)` | Deduplicated list of source IPs seen for the aggregated bucket. |
| `tokens` | `Array(String)` | Deduplicated list of token identifiers seen for the aggregated bucket. |
### tinybird.jobs_log (https://www.tinybird.co/docs/forward/monitoring/service-datasources#tinybird-jobs-log)
Contains all job executions performed in your workspace. Tinybird logs all jobs with extra information in this data source:
| Field | Type | Description |
| --- | --- | --- |
| `job_id` | `String` | Unique identifier for the job. |
| `job_type` | `LowCardinality(String)` | Type of job execution: `copy`, `delete`, `deployment`, `dynamodb_sync`, `gcs_sync`, `import`, `populate`, `s3_sync`, `sink`. |
| `workspace_id` | `String` | Unique identifier for the workspace. |
| `pipe_id` | `String` | Unique identifier for the pipe. |
| `pipe_name` | `String` | Name of the pipe. |
| `created_at` | `DateTime` | Timestamp when the job was created. |
| `updated_at` | `DateTime` | Timestamp when the job was last updated. |
| `started_at` | `DateTime` | Timestamp when the job execution started. |
| `status` | `LowCardinality(String)` | Current status of the job. `waiting`, `working`, `done`, `error`, `cancelled`. |
| `error` | `Nullable(String)` | Detailed error message if the result was error. |
| `job_metadata` | `JSON String` | Additional metadata related to the job execution. |
Learn more about how to track background jobs execution in the [Jobs monitoring guide](./jobs).
### tinybird.query_validator_log (https://www.tinybird.co/docs/forward/monitoring/service-datasources#tinybird-query-validator-log)
Contains information about query validation executions for Pipes and Materialized Views in your Workspace.
| Field | Type | Description |
| --- | --- | --- |
| `host` | `LowCardinality(String)` | Name of the host the query was extracted from. It might differ from the host the query was executed on. |
| `version` | `LowCardinality(String)` | The version in which the query was tested. |
| `stable_version` | `LowCardinality(String)` | The version that was running in the cluster when the query was validated. |
| `query_hash` | `UInt64` | The normalized query hash. |
| `query_last_execution` | `DateTime` | The last execution time of this query at the time when it was extracted. |
| `region` | `String` | The region in which the cluster is deployed. |
| `workspace` | `String` | The name of the Workspace, for Pipe queries. |
| `database` | `String` | The internal database name for this Workspace. |
| `pipe_name` | `String` | The name of the Pipe, for Pipe queries. |
| `error_code` | `Int16` | Error code returned by ClickHouse®. |
| `error` | `String` | Detailed error message if the query failed. |
| `query_id` | `String` | ClickHouse® query ID. For Pipe queries, this can be matched with `system.query_log`. For Materialized Views, it contains the qualified internal table name. |
| `query` | `String` | The full query. |
| `fix_suggestion` | `String` | Optionally, a suggestion on how to fix the query. |
| `run_validation` | `DateTime` | The time when this query was validated. |
Learn more about query validation in `tinybird.query_validator_log`.
## Query _state columns (https://www.tinybird.co/docs/forward/monitoring/service-datasources#query-state-columns)
Several of the Service data sources include columns suffixed with `_state`. This suffix identifies columns with values that are in an intermediate aggregated state. When reading these columns, merge the intermediate states to get the final value.
To merge intermediate states, wrap the column in the original aggregation function and apply the `-Merge` combinator.
For example, to finalize the value of the `avg_duration_state` column, you use the `avgMerge` function:
finalize the value for the avg_duration_state column
```
SELECT
date,
avgMerge(avg_duration_state) avg_time,
quantilesTimingMerge(0.9, 0.95, 0.99)(quantile_timing_state) quantiles_timing_in_ms_array
FROM tinybird.pipe_stats
where pipe_id = 'PIPE_ID'
group by date
```
See [Combinators](/docs/sql-reference/functions/aggregate-functions#aggregate-function-combinators) to learn more about the `-Merge` combinator.
## Organization service data sources (https://www.tinybird.co/docs/forward/monitoring/service-datasources#organization-service-data-sources)
The following is a complete list of available Organization Service data sources:
| Field | Description |
| --- | --- |
| `organization.workspaces` | Lists all Organization workspaces and related information, including name, IDs, databases, plan, when it was created, and whether it has been soft-deleted. |
| `organization.processed_data` **(deprecated)** | **Deprecated and no longer works as of April 1, 2026.** Use `system.query_log` instead. See the [changelog](/docs/changelog/2026-03-27-march-week-four#deprecation-of-bi-and-billing-service-data-sources) for migration examples. |
| `organization.datasources_storage` | Equivalent to `tinybird.datasources_storage` but with data for all Organization workspaces. |
| `organization.pipe_stats` | Equivalent to `tinybird.pipe_stats` but with data for all Organization workspaces. |
| `organization.pipe_stats_rt` | Equivalent to `tinybird.pipe_stats_rt` but with data for all Organization workspaces. |
| `organization.datasources_ops_log` | Equivalent to `tinybird.datasources_ops_log` but with data for all Organization workspaces. |
| `organization.data_transfer` | Equivalent to `tinybird.data_transfer` but with data for all Organization workspaces. |
| `organization.jobs_log` | Equivalent to `tinybird.jobs_log` but with data for all Organization workspaces. |
| `organization.sinks_ops_log` | Equivalent to `tinybird.sinks_ops_log` but with data for all Organization workspaces. |
| `organization.endpoint_errors` | Equivalent to `tinybird.endpoint_errors` but with data for all Organization workspaces. |
| `organization.shared_infra_active_minutes` | Contains information about vCPU active minutes consumption aggregated by minute for all Organization workspaces. Used for SaaS plan billing on shared infrastructure. Deprecated soon for other shared plans. |
| `organization.shared_infra_active_seconds` | Contains the total vCPU time in seconds consumed by all operations in the organization during each natural second. Available for Developer and SaaS plans on shared infrastructure. |
| `organization.shared_infra_qps_overages` | Contains information about QPS consumption and overages aggregated by second for all Organization workspaces. Used for SaaS plan billing on shared infrastructure. |
| `organization.llm_usage` | Equivalent to `tinybird.llm_usage` but with data for all Organization workspaces. |
| `organization.backups` | Contains backup metadata for dedicated clusters in the organization, including size and status. |
| `organization.private_link` | Contains Private Link usage and cost data for observability, associated with your organization's workspaces. |
To query Organization Service data sources, go to any workspace that belongs to the Organization and use them as regular Service data sources from the Playground or within pipes. You can also make queries using your preferred method, like `tb sql`.
### metrics_logs service data source (https://www.tinybird.co/docs/forward/monitoring/service-datasources#metrics-logs-service-data-source)
The `metrics_logs` Service data source is available in all the organization's workspaces to all Organization members. New records for each of the metrics monitored are added every minute with the following schema:
The `metrics_logs` Service Data Source is only populated for Organizations with **dedicated infrastructure**. It contains hardware-level measurements (CPU, memory, disk) of dedicated cluster nodes. If your Organization is on shared infrastructure, this table is empty. For monitoring on shared infrastructure, use other Service Data Sources like `datasources_ops_log`, `pipe_stats`, and `datasources_storage`.
| Field | Type | Description |
| --- | --- | --- |
| `timestamp` | `DateTime` | Timestamp of the metric |
| `cluster` | `LowCardinality(String)` | Name of the cluster |
| `host` | `LowCardinality(String)` | Name of the host |
| `metric` | `LowCardinality(String)` | Name of the metric |
| `value` | `String` | Value of the metric |
| `description` | `LowCardinality(String)` | Description of the metric |
| `organization_id` | `String` | ID of your organization |
The available metrics are the following:
| Metric | Description |
| --- | --- |
| MemoryTracking | Total amount of memory, in bytes, allocated by the server. |
| OSMemoryTotal | The total amount of memory on the host system, in bytes. |
| InstanceType | Instance type of the host. |
| Query | Number of executing queries. |
| NumberCPU | Number of CPUs. |
| LoadAverage1 | The whole system load, averaged with exponential smoothing over 1 minute. The load represents the number of threads across all the processes (the scheduling entities of the OS kernel), that are currently running by CPU or waiting for IO, or ready to run but not being scheduled at this point of time. This number includes all the processes, not only the server. The number can be greater than the number of CPU cores, if the system is overloaded, and many processes are ready to run but waiting for CPU or IO. |
| LoadAverage15 | The whole system load, averaged with exponential smoothing over 15 minutes. The load represents the number of threads across all the processes (the scheduling entities of the OS kernel), that are currently running by CPU or waiting for IO, or ready to run but not being scheduled at this point of time. This number includes all the processes, not only the server. The number can be greater than the number of CPU cores, if the system is overloaded, and many processes are ready to run but waiting for CPU or IO. |
| CPUUsage | The ratio of time the CPU core was running OS kernel (system) code or userspace code. This is a system-wide metric, it includes all the processes on the host machine, not only the server. This includes also the time when the CPU was under-utilized due to the reasons internal to the CPU (memory loads, pipeline stalls, branch mispredictions, running another SMT core). |
| ReplicasSumQueueSize | Sum queue size (in the number of operations like get, merge) across Replicated tables. |
| ReplicasSumMergesInQueue | Sum of merge operations in the queue (still to be applied) across Replicated tables. |
| ReplicasSumInsertsInQueue | Sum of insert operations in the queue (still to be applied) across Replicated tables. |
---
URL: https://www.tinybird.co/docs/forward/monitoring/query-log
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "How to debug queries using system.query\_log \| Tinybird"
theme-color: "#171612"
description: "Learn how to use system.query_log to debug slow queries, analyze memory usage, and troubleshoot issues in your Tinybird Workspace."
inkeep:version: "forward"
---
# How to debug queries using system.query_log (https://www.tinybird.co/docs/forward/monitoring/query-log#how-to-debug-queries-using-system-query-log)
When troubleshooting performance issues or investigating query failures, you often need execution details beyond what standard monitoring provides. The `system.query_log` table is a native ClickHouse® system table that records query execution at the database engine level beneath Tinybird. It stores detailed information for every query executed in your Workspace, including execution time, memory usage, error codes, and failure reasons. For a complete description of available columns and data types, see the ClickHouse® documentation on the `system.query_log` table.
**When to use system.query_log**
Use `system.query_log` when you need to:
- Debug queries that are slower than expected
- Investigate why specific queries are failing
- Analyze memory consumption patterns
- Track which Endpoints consume the most resources
- Correlate query execution with Endpoint performance metrics
For routine monitoring of Endpoint performance, use `tinybird.pipe_stats_rt` instead.
Read on to learn how to safely query `system.query_log` and use it to solve real-world debugging challenges.
## Before you start (https://www.tinybird.co/docs/forward/monitoring/query-log#before-you-start)
The `system.query_log` table is a ClickHouse® system table that contains extensive historical data. Before querying it, understand:
- How to [analyze Endpoint performance](./analyze-endpoints-performance) using `tinybird.pipe_stats_rt`
- Basic SQL query patterns in Tinybird
The `system.query_log` table contains all historical query data. **Always filter by `event_time`** to avoid slow queries or timeouts. Queries without time filters can take hundreds of seconds or timeout completely. **Never run** `SELECT * FROM system.query_log` without filtering by time.
Safe query pattern with 1-hour lookback
```
SELECT *
FROM system.query_log
WHERE event_time >= now() - INTERVAL 1 HOUR
```
Start with a 1-hour window (typically 1-2 seconds to execute) and expand the time range only if needed.
## Understand query lifecycle (https://www.tinybird.co/docs/forward/monitoring/query-log#understand-query-lifecycle)
Every query in Tinybird generates two entries in `system.query_log`:
1. **QueryStart** - Logged when query execution begins
2. **QueryFinish** - Logged when the query completes successfully
If a query fails, the log shows a different type instead of `QueryFinish`:
- **ExceptionBeforeStart** - Query failed before execution started (usually syntax errors)
- **ExceptionWhileProcessing** - Query failed during execution (timeouts, memory exceeded, runtime errors)
Here's how the query lifecycle flows:
```
┌─────────────────┐
│ Query Begins │
└────────┬────────┘
│
▼
QueryStart ──────────────────┐
│ │
│ │
▼ ▼
┌─────────────────┐ ┌──────────────────────┐
│ Query Executes │ │ Syntax/Init Error │
└────────┬────────┘ └──────────┬───────────┘
│ │
│ ▼
│ ExceptionBeforeStart
│
┌────┴────┐
│ │
▼ ▼
Success Runtime Error
│ │
▼ ▼
QueryFinish ExceptionWhileProcessing
```
This two-entry pattern helps you distinguish between queries that are still running and those that have finished or failed.
## Common debugging scenarios (https://www.tinybird.co/docs/forward/monitoring/query-log#common-debugging-scenarios)
### Find slow queries (https://www.tinybird.co/docs/forward/monitoring/query-log#find-slow-queries)
Identify queries taking longer than expected:
Queries taking more than 5 seconds
```
SELECT
event_time,
query_id,
JSONExtractString(log_comment, 'pipe') as pipe_name,
query_duration_ms / 1000.0 as duration_seconds,
formatReadableSize(memory_usage) as peak_memory,
substring(query, 1, 100) as query_preview
FROM system.query_log
WHERE event_time >= now() - INTERVAL 1 HOUR
AND type = 'QueryFinish'
AND query_duration_ms > 5000
ORDER BY query_duration_ms DESC
LIMIT 20
```
This helps you identify performance bottlenecks in your Endpoints or scheduled operations.
### Investigate query failures (https://www.tinybird.co/docs/forward/monitoring/query-log#investigate-query-failures)
Track which queries are failing and why:
Failed queries in the last hour
```
SELECT
event_time,
query_id,
type,
JSONExtractString(log_comment, 'pipe') as pipe_name,
exception
FROM system.query_log
WHERE event_time >= now() - INTERVAL 1 HOUR
AND type IN ('ExceptionBeforeStart', 'ExceptionWhileProcessing')
ORDER BY event_time DESC
```
The `exception` column contains the error message, helping you quickly identify syntax errors, timeouts, or resource limits.
### Analyze memory consumption (https://www.tinybird.co/docs/forward/monitoring/query-log#analyze-memory-consumption)
Find queries consuming the most memory:
Memory-intensive queries
```
SELECT
event_time,
query_id,
JSONExtractString(log_comment, 'pipe') as pipe_name,
formatReadableSize(memory_usage) as peak_memory,
query_duration_ms / 1000.0 as duration_seconds
FROM system.query_log
WHERE event_time >= now() - INTERVAL 1 HOUR
AND type = 'QueryFinish'
ORDER BY memory_usage DESC
LIMIT 20
```
The `memory_usage` column shows peak memory in bytes. High memory usage may indicate opportunities for query optimization.
### Track query failure rates (https://www.tinybird.co/docs/forward/monitoring/query-log#track-query-failure-rates)
Monitor how often queries are failing over time:
Query failures by hour
```
SELECT
toStartOfHour(event_time) as hour,
type,
count() as failure_count
FROM system.query_log
WHERE event_time >= now() - INTERVAL 24 HOUR
AND type IN ('ExceptionBeforeStart', 'ExceptionWhileProcessing')
GROUP BY hour, type
ORDER BY hour DESC
```
## Connect query_log with Endpoint stats (https://www.tinybird.co/docs/forward/monitoring/query-log#connect-query-log-with-endpoint-stats)
The `query_id` in `system.query_log` matches the `request_id` from `tinybird.pipe_stats_rt`. Use these fields to combine high-level request metrics with detailed query execution data.
Combine endpoint stats with query details
```
SELECT
ps.pipe_name,
ps.start_datetime,
ps.duration as request_duration,
formatReadableSize(ql.memory_usage) as query_memory,
ql.type as execution_status
FROM tinybird.pipe_stats_rt ps
JOIN system.query_log ql ON ps.request_id = ql.query_id
WHERE ps.start_datetime >= now() - INTERVAL 1 HOUR
AND ql.type = 'QueryFinish'
ORDER BY ps.start_datetime DESC
LIMIT 100
```
The `event_time` from `system.query_log` and `start_datetime` from `pipe_stats_rt` may differ by a few milliseconds. Always join using `query_id` and `request_id` for accurate matching.
## Extract pipe names from log_comment (https://www.tinybird.co/docs/forward/monitoring/query-log#extract-pipe-names-from-log-comment)
Endpoint queries include metadata in the `log_comment` column as JSON. Extract the pipe name to filter by specific Endpoints:
Filter by Endpoint name
```
SELECT
event_time,
query_id,
JSONExtractString(log_comment, 'pipe') as pipe_name,
JSONExtractString(log_comment, 'workspace') as workspace,
formatReadableSize(memory_usage) as memory_usage,
query_duration_ms / 1000.0 as duration_seconds
FROM system.query_log
WHERE event_time >= now() - INTERVAL 1 HOUR
AND log_comment != ''
AND JSONExtractString(log_comment, 'pipe') = 'your_pipe_name'
AND type = 'QueryFinish'
ORDER BY event_time DESC
```
The `log_comment` JSON structure looks like:
```
{"workspace":"prod","pipe":"last_impressions__v7"}
```
## Find memory-intensive endpoints (https://www.tinybird.co/docs/forward/monitoring/query-log#find-memory-intensive-endpoints)
Identify which Endpoints consistently use the most memory:
Average memory by Endpoint
```
SELECT
JSONExtractString(log_comment, 'pipe') as pipe_name,
formatReadableSize(avg(memory_usage)) as avg_memory,
formatReadableSize(max(memory_usage)) as max_memory,
formatReadableSize(quantile(0.9)(memory_usage)) as p90_memory,
count() as query_count
FROM system.query_log
WHERE event_time >= now() - INTERVAL 24 HOUR
AND type = 'QueryFinish'
AND log_comment != ''
GROUP BY pipe_name
ORDER BY avg(memory_usage) DESC
LIMIT 20
```
## Common mistakes (https://www.tinybird.co/docs/forward/monitoring/query-log#common-mistakes)
Avoid these common pitfalls when working with `system.query_log`:
### Running queries without time filters (https://www.tinybird.co/docs/forward/monitoring/query-log#running-queries-without-time-filters)
**Never do this:**
```
SELECT * FROM system.query_log
WHERE query LIKE '%my_datasource_id%'
```
This scans the entire table history and times out or takes hundreds of seconds.
**Always include time filters:**
```
SELECT * FROM system.query_log
WHERE event_time >= now() - INTERVAL 1 HOUR
AND query LIKE '%my_datasource_id%'
```
### Selecting all columns with SELECT * (https://www.tinybird.co/docs/forward/monitoring/query-log#selecting-all-columns-with-select)
Avoid `SELECT *` in production queries. The `query` column can be large, and many columns aren't needed for most debugging tasks.
**Instead, select only what you need:**
```
SELECT
event_time,
query_id,
type,
query_duration_ms,
memory_usage,
exception
FROM system.query_log
WHERE event_time >= now() - INTERVAL 1 HOUR
```
### Forgetting to filter by query type (https://www.tinybird.co/docs/forward/monitoring/query-log#forgetting-to-filter-by-query-type)
When analyzing completed queries, include `type = 'QueryFinish'` to exclude `QueryStart` entries and avoid counting queries twice:
```
-- Without type filter: counts each query twice (Start + Finish)
SELECT count(*) FROM system.query_log
WHERE event_time >= now() - INTERVAL 1 HOUR
-- With type filter: accurate count
SELECT count(*) FROM system.query_log
WHERE event_time >= now() - INTERVAL 1 HOUR
AND type = 'QueryFinish'
```
### Using overly broad time windows (https://www.tinybird.co/docs/forward/monitoring/query-log#using-overly-broad-time-windows)
Start with 1-hour windows and expand only if needed. A 24-hour query takes significantly longer than a 1-hour query:
- **1 hour**: ~1-2 seconds
- **24 hours**: ~15-30 seconds
- **7 days**: May timeout or take minutes
## Best practices (https://www.tinybird.co/docs/forward/monitoring/query-log#best-practices)
When querying `system.query_log`, follow these guidelines:
1. **Always filter by time** - Use `event_time >= now() - INTERVAL X HOUR` in every query
2. **Filter by query type** - Add `AND type = 'QueryFinish'` if you only need completed queries
3. **Limit your results** - Use `LIMIT` to avoid processing unnecessary data
4. **Start small** - Begin with 1-hour windows, then expand as needed
5. **Create monitoring Pipes** - Turn useful debug queries into Endpoints for dashboards
Convert your most useful `system.query_log` queries into Pipes and expose them as Endpoints. This makes it easy to build real-time monitoring dashboards or set up alerts.
## Next steps (https://www.tinybird.co/docs/forward/monitoring/query-log#next-steps)
- Learn how to [analyze Endpoint performance](./analyze-endpoints-performance) using `tinybird.pipe_stats_rt`
- Monitor [Jobs](./jobs) in your Workspace
---
URL: https://www.tinybird.co/docs/forward/monitoring/organization-consumption
Last update: 2026-08-04T10:22:54.000Z
Content:
---
title: "Advanced organization consumption monitoring \| Tinybird"
theme-color: "#171612"
description: "Learn how to monitor your organization's resource consumption in detail using SQL queries"
inkeep:version: "forward"
---
# Advanced organization consumption monitoring (https://www.tinybird.co/docs/forward/monitoring/organization-consumption#advanced-organization-consumption-monitoring)
While Tinybird provides built-in graphs and metrics in the UI for monitoring your organization's resource consumption, some use cases require more detailed information. This guide explains how to use SQL queries to monitor your consumption in detail, specifically for organizations using shared infrastructure.
This monitoring approach applies only to Developer and SaaS customers on shared infrastructure. If you're on a dedicated infrastructure plan, contact your account manager for specific monitoring solutions.
## Prerequisites (https://www.tinybird.co/docs/forward/monitoring/organization-consumption#prerequisites)
- Your organization must be on shared infrastructure (Developer or SaaS plans)
- Basic understanding of SQL and Tinybird's Data Sources
## Understanding organization usage (https://www.tinybird.co/docs/forward/monitoring/organization-consumption#understanding-organization-usage)
By default, Tinybird provides usage graphs in the UI that show:
- Resource consumption over the last 7 days
- A usage table displaying resources consuming the most vCPU
While these built-in visualizations are sufficient for most use cases, you might need more granular control and insight into your consumption patterns.
## Advanced vCPU monitoring (https://www.tinybird.co/docs/forward/monitoring/organization-consumption#advanced-vcpu-monitoring)
Tinybird uses vCPU active minutes for SaaS plan billing on shared infrastructure. Developer plans bill vCPU overages using vCPU seconds instead. See `organization.shared_infra_active_seconds`.
### Important notes about vCPU metrics (https://www.tinybird.co/docs/forward/monitoring/organization-consumption#important-notes-about-vcpu-metrics)
- Materialized Views currently show 0 for direct vCPU time
- The landing Data Source includes both its own CPU time and the CPU time of its associated Materialized Views
### Detailed CPU usage analysis (https://www.tinybird.co/docs/forward/monitoring/organization-consumption#detailed-cpu-usage-analysis)
You can analyze detailed CPU consumption across different operations using several service data sources:
#### API and SQL operations (https://www.tinybird.co/docs/forward/monitoring/organization-consumption#api-and-sql-operations)
Using `pipe_stats` and `pipe_stats_rt` service data sources, you can monitor CPU usage for API endpoints and SQL queries:
Here's an example query that shows vCPU consumption by Pipe for the last 7 days:
Monitor vCPU consumption by Pipe for last 7 days
```
SELECT
pipe_name,
round(sum(cpu_time), 2) as total_cpu_seconds,
count() as total_requests
FROM organization.pipe_stats_rt
WHERE
start_datetime >= (now() - interval 7 day)
GROUP BY pipe_name
ORDER BY total_cpu_seconds DESC
```
#### Data Source operations (https://www.tinybird.co/docs/forward/monitoring/organization-consumption#data-source-operations)
The `datasources_ops_log` service data source provides CPU metrics for operations on your Data Sources.
Here's an example query that shows vCPU consumption by Data Source and operation type:
Monitor Data Source operations vCPU consumption
```
SELECT
datasource_name,
event_type,
round(sum(cpu_time), 2) as total_cpu_seconds,
count() as total_operations
FROM organization.datasources_ops_log
WHERE
timestamp >= (now() - interval 7 day)
GROUP BY
datasource_name,
event_type
ORDER BY total_cpu_seconds DESC
```
#### Sinks operations (https://www.tinybird.co/docs/forward/monitoring/organization-consumption#sinks-operations)
Monitor CPU usage in your Sinks using the `sinks_ops_log` service data source:
Here's an example query that shows vCPU consumption by Sink:
Monitor Sinks operations vCPU consumption
```
SELECT
pipe_name,
round(sum(cpu_time), 2) as total_cpu_seconds,
count() as total_operations
FROM organization.sinks_ops_log
WHERE
timestamp >= (now() - interval 7 day)
GROUP BY pipe_name
ORDER BY total_cpu_seconds DESC
```
### Developer vCPU seconds (https://www.tinybird.co/docs/forward/monitoring/organization-consumption#developer-vcpu-seconds)
For Developer plan vCPU overage billing, use the `shared_infra_active_seconds` service data source:
Monitor vCPU seconds for current day
```
SELECT * FROM organization.shared_infra_active_seconds
WHERE
toStartOfDay(second) = today()
ORDER BY second DESC
```
### SaaS vCPU active minutes (https://www.tinybird.co/docs/forward/monitoring/organization-consumption#saas-vcpu-active-minutes)
For SaaS plan active minutes billing, use the `shared_infra_active_minutes` service data source:
Monitor active minutes for current day
```
SELECT * FROM organization.shared_infra_active_minutes
WHERE
toStartOfDay(minute) = today()
ORDER BY minute DESC
```
## Storage monitoring (https://www.tinybird.co/docs/forward/monitoring/organization-consumption#storage-monitoring)
Storage consumption is a key billing metric that measures the amount of data stored in your Data Sources. You can monitor storage usage using the `datasources_storage` service data source.
Storage is billed based on two factors:
1. The maximum total storage used by your organization each day (including quarantined data)
2. The average of those daily maximums throughout your billing cycle
Storage for data in quarantine is included in your billing calculations. When monitoring storage for costs, always consider both regular and quarantined data.
### Current storage usage (https://www.tinybird.co/docs/forward/monitoring/organization-consumption#current-storage-usage)
Here's an example query that shows current storage usage by Data Source, including both regular and quarantined data:
Monitor current storage usage by Data Source
```
SELECT
datasource_name,
round((bytes + bytes_quarantine)//1000000000, 2) as total_storage_gb,
round(bytes_quarantine//1000000000, 2) as quarantine_storage_gb,
rows + rows_quarantine as total_rows,
rows_quarantine as quarantine_rows
FROM organization.datasources_storage
WHERE timestamp >= (now() - interval 2 hour)
ORDER BY total_storage_gb DESC
LIMIT 1 BY datasource_name
```
### Billing period storage analysis (https://www.tinybird.co/docs/forward/monitoring/organization-consumption#billing-period-storage-analysis)
To analyze your storage consumption for billing purposes, use this query that calculates the average of daily maximum storage across your billing period:
Calculate average storage for billing period
```
SELECT
greatest(avg(daily_max_org_storage_gb), 0) as avg_storage_gb,
greatest(avg(daily_max_org_storage_rows), 0) as avg_storage_rows
FROM (
SELECT
sum(floor(max_total_bytes_by_ds/1000000000, 6)) as daily_max_org_storage_gb,
sum(max_total_rows_by_ds) as daily_max_org_storage_rows
FROM (
SELECT
toDate(timestamp) as date,
max(bytes + bytes_quarantine) as max_total_bytes_by_ds,
max(rows + rows_quarantine) AS max_total_rows_by_ds
FROM organization.datasources_storage
WHERE 1= 1
AND date >= '2025-04-XX' -- beginning of term
AND date <= '2025-04-xx' -- end of term
GROUP BY date, datasource_id
)
GROUP BY date
)
```
Replace the date placeholders (`'2025-04-XX'`) with your actual billing period start and end dates to get accurate billing metrics.
## QPS (queries per second) monitoring (https://www.tinybird.co/docs/forward/monitoring/organization-consumption#qps-queries-per-second-monitoring)
You can monitor your QPS consumption using two different data sources, each providing different information about your usage:
### Detailed query analysis (last 7 days) (https://www.tinybird.co/docs/forward/monitoring/organization-consumption#detailed-query-analysis-last-7-days)
Using the `pipe_stats_rt` service data source, you can analyze detailed information about your API endpoints and SQL queries usage. This data source provides rich information about each query but is limited to the last 7 days due to TTL.
The `pipe_stats_rt` data source has a 7-day TTL (Time To Live), so historical analysis is limited to this timeframe.
Here's an example query that shows the number of requests per Pipe over the last hour:
Monitor QPS by pipe for the last hour
```
SELECT
start_datetime,
pipe_name,
count() total
FROM organization.pipe_stats_rt
WHERE
start_datetime BETWEEN (now() - interval 1 hour) AND now()
GROUP BY
start_datetime, pipe_name
ORDER BY
start_datetime DESC
```
### Historical QPS and overages (https://www.tinybird.co/docs/forward/monitoring/organization-consumption#historical-qps-and-overages)
For longer-term analysis of QPS consumption and overages, you can use the `shared_infra_qps_overages` service data source. This provides aggregated QPS data and overage information per second, though with less detail about individual queries.
Here's an example query that shows daily QPS overages for the current month:
Monitor daily QPS overages for current month
```
SELECT
toStartOfDay(start_datetime) day,
sum(overages) total_overages
FROM organization.shared_infra_qps_overages
WHERE
toStartOfMonth(start_datetime) = toStartOfMonth(now())
GROUP BY day
ORDER BY day DESC
```
## Data transfer monitoring (https://www.tinybird.co/docs/forward/monitoring/organization-consumption#data-transfer-monitoring)
Data transfer metrics track the amount of data moved through Sinks in your organization. The cost varies depending on whether data is transferred within the same region (Intra) or between different regions (Inter).
### Sinks data transfer (https://www.tinybird.co/docs/forward/monitoring/organization-consumption#sinks-data-transfer)
Monitor data transfer costs for Sinks using the `data_transfer` service data source:
Monitor Sinks data transfer by type
```
SELECT
toStartOfDay(timestamp) as day,
workspace_id,
kind,
round(sum(bytes)/1000000000, 2) as transferred_gb,
count() as operations
FROM organization.data_transfer
WHERE
timestamp >= (now() - interval 30 day)
AND kind IN ('intra', 'inter')
GROUP BY
day,
workspace_id,
kind
ORDER BY
day DESC,
kind ASC
```
## Best practices (https://www.tinybird.co/docs/forward/monitoring/organization-consumption#best-practices)
1. **Regular Monitoring**: Set up a routine to check these metrics, especially if you're approaching your plan limits
2. **Alert Setup**: Consider setting up alerts using these queries to proactively monitor consumption
3. **Resource Optimization**: Use these metrics to identify opportunities for query optimization or resource reallocation
## Additional resources (https://www.tinybird.co/docs/forward/monitoring/organization-consumption#additional-resources)
- [Organizations](/docs/forward/core-concepts/organizations) - Learn about organization management and monitoring
- [Pricing plans](../pricing) - Compare billing models and SKUs across Free, Developer, SaaS, and Enterprise plans
- [Resource Limits](/docs/forward/pricing/limits) - Learn about the storage, QPS, and other resource limits for different plans
---
URL: https://www.tinybird.co/docs/forward/monitoring/latency
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Measure API endpoint latency \| Tinybird"
theme-color: "#171612"
description: "Latency is an essential metric to monitor in real-time applications. Learn how to measure and monitor the latency of your API endpoints in Tinybird."
inkeep:version: "forward"
---
# Measure API endpoint latency (https://www.tinybird.co/docs/forward/monitoring/latency#measure-api-endpoint-latency)
Latency is the time it takes for a request to travel from the client to the server and back; the time it takes for a request to be sent and received. Latency is usually measured in seconds or milliseconds (ms). The lower the latency, the faster the response time.
Latency is an essential metric to monitor in real-time applications. Read on to learn how latency is measured in Tinybird, and how to monitor and visualize the latency of your API endpoints when data is being retrieved.
## How latency is measured (https://www.tinybird.co/docs/forward/monitoring/latency#how-latency-is-measured)
When measuring latency in an end-to-end app, consider data ingestion, data transformation, and data retrieval. In Tinybird, latency is the time it takes to send a request and return the response to the client.
When calling an API endpoint, you can check this metric defined as `elapsed` in the `statistics` object of the response:
Statistics object within an example Tinybird API endpoint call
```
{
"meta": [ ... ],
"data": [ ... ],
"rows": 10,
"statistics": {
"elapsed": 0.001706275,
"rows_read": 10,
"bytes_read": 180
}
}
```
## Monitor latency (https://www.tinybird.co/docs/forward/monitoring/latency#monitor-latency)
To monitor the latency of your API endpoints, use the `pipe_stats_rt` and `pipe_stats` [Service data sources](./service-datasources):
- `pipe_stats_rt` consists of the real-time statistics of your API endpoints, and has a `duration` field that encapsulates the latency time in seconds.
- `pipe_stats` contains the **aggregated** statistics of your API endpoints by date, and presents a `avg_duration_state` field which is the average duration of the API endpoint by day in seconds.
Because the `avg_duration_state` field is an intermediate state, you'd need to merge it when querying the data source using something like `avgMerge`.
For details on building pipes and endpoints that monitor the performance of your API endpoints using the `pipe_stats_rt` and `pipe_stats` data sources, follow the [API endpoint performance guide](./analyze-endpoints-performance#example-2-analyzing-the-performance-of-api-endpoints-over-time).
## Visualize latency (https://www.tinybird.co/docs/forward/monitoring/latency#visualize-latency)
In your workspace, go to **Time Series** and select **Endpoint performance** to visualize the latency of your API endpoints over time.
## Next steps (https://www.tinybird.co/docs/forward/monitoring/latency#next-steps)
- Read this blog on [Monitoring global API latency](https://www.tinybird.co/blog-posts/dev-qa-global-api-latency-chronark).
---
URL: https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring
Last update: 2026-07-24T12:51:58.000Z
Content:
---
title: "Monitor Kafka to ClickHouse® Connector Lag \| Tinybird"
theme-color: "#171612"
description: "Monitor Tinybird Kafka connector lag, throughput, errors, and partition behavior with Kafka meta columns and the kafka_ops_log Service Data Source."
inkeep:version: "forward"
---
# Monitor Kafka to ClickHouse® connector and consumer lag (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#monitor-kafka-to-clickhouse-connector-and-consumer-lag)
Tinybird's Kafka connector automatically consumes data from Kafka topics and ingests it into ClickHouse® Data Sources. The connector features consumer autoscaling and multi-broker connectivity. The `tinybird.kafka_ops_log` [Service Data Source](/docs/forward/monitoring/service-datasources) provides operational logs for monitoring your integration, tracking consumer lag, analyzing performance, and optimizing your connector configuration.
Use `kafka_ops_log` to monitor consumer lag, track throughput, identify errors, and analyze performance across your Data Sources. The connector's serverless architecture automatically scales consumers based on load, and `kafka_ops_log` provides visibility into this autoscaling behavior along with error handling and debugging information.
Logs are retained for 30 days and include information about each partition, topic, and Data Source in your workspace. This supports analysis of partition assignment, rebalancing, and consumer group scaling behavior.
**New to the Kafka connector?** If you're setting up the Kafka connector for the first time, see the [Kafka connector documentation](/docs/forward/ingest-data/connectors/kafka) for setup instructions, configuration options, and troubleshooting. This monitoring guide focuses on operational monitoring once your connector is running.
## Kafka meta columns in your Data Sources (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#kafka-meta-columns-in-your-data-sources)
In addition to monitoring via `kafka_ops_log`, each Data Source connected to Kafka includes [Kafka meta columns](/docs/forward/ingest-data/connectors/kafka#kafka-meta-columns) that store metadata from Kafka messages. These columns are automatically added to your Data Source schema and can be used for analysis, debugging, and monitoring.
The available meta columns include:
- **`__topic`**: The Kafka topic the message was read from
- **`__partition`**: The partition number the message was read from
- **`__offset`**: The message offset within the partition
- **`__timestamp`**: The timestamp of the Kafka message
- **`__key`**: The message key (if present)
- **`__value`**: The raw message value (if `KAFKA_STORE_RAW_VALUE` is set to `True`)
- **`__headers`**: Kafka headers as a Map (if `KAFKA_STORE_HEADERS` is set to `True`)
### Use cases for Kafka meta columns (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#use-cases-for-kafka-meta-columns)
**Track message ordering and detect gaps**: Use `__offset` to identify missing messages or gaps in your data ingestion. Compare consecutive offsets to detect when messages are skipped.
**Analyze distribution**: Use `__partition` to understand how messages are distributed across partitions. This helps verify that your partition key design is working as expected and that messages are evenly distributed.
**Correlate with `kafka_ops_log`**: Join your Data Source with `kafka_ops_log` using `__partition` and `__offset` to correlate ingested messages with connector operations, helping you understand ingestion latency and identify bottlenecks.
**Message timestamp analysis**: Compare `__timestamp` (Kafka message timestamp) with your Data Source's ingestion timestamp to understand end-to-end latency from message production to ClickHouse ingestion.
**Debug with message keys**: Use `__key` to track specific messages or understand how keys map to partitions, which is essential for partition key design optimization.
**Leverage headers for routing and tracing**: If `KAFKA_STORE_HEADERS` is set to `True`, use `__headers` to access custom metadata, trace IDs, routing information, or other context stored in Kafka headers. This is useful for distributed tracing, A/B testing, or routing logic based on header values.
| Field | Type | Description |
| --- | --- | --- |
| `timestamp` | `DateTime` | Date and time when the operation took place. |
| `datasource_id` | `String` | ID of your Data Source. The Data Source ID is consistent after renaming operations. |
| `topic` | `String` | Kafka topic. |
| `partition` | `Int16` | Partition number, or `-1` for all partitions. Use this field to analyze partition assignment and identify which partitions are assigned to your consumer. |
| `msg_type` | `String` | `'info'` for regular messages, `'warning'` for issues related to the user's Kafka cluster, deserialization or Materialized Views, and `'error'` for other issues. |
| `lag` | `Int64` | Number of messages behind for the partition. This is the difference between the high-water mark and the last commit offset. This is the primary metric for consumer lag monitoring. |
| `processed_messages` | `Int32` | Messages processed for a topic and partition. |
| `processed_bytes` | `Int32` | Amount of bytes processed. |
| `committed_messages` | `Int32` | Messages ingested for a topic and partition. |
| `msg` | `String` | Information in the case of warnings or errors. Empty otherwise. |
The `tinybird.kafka_ops_log` Service Data Source includes operational metrics from your serverless Kafka connector. Each row represents a snapshot of the connector's state for a specific partition at a point in time, providing visibility into autoscaling consumer behavior, error handling, and debugging information.
## Monitor consumer lag (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#monitor-consumer-lag)
Consumer lag is the difference between the latest offset in a partition (high-water mark) and the last committed offset by your consumer. High lag values indicate that your consumer is falling behind the producer, which can lead to delayed data ingestion. With Tinybird's serverless connector, consumer autoscaling automatically adjusts to handle increased load, but monitoring lag helps you understand when scaling occurs and verify that autoscaling is working effectively.
Monitoring consumer lag is critical for performance tuning. When lag increases, Tinybird's autoscaling infrastructure automatically scales consumers to handle the load. The `kafka_ops_log` provides visibility into this autoscaling behavior, helping you understand partition assignment patterns and verify that your integration is performing optimally. For more details on consumer configuration, see the [Kafka connector documentation](/docs/forward/ingest-data/connectors/kafka).
### Find current consumer lag by partition (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#find-current-consumer-lag-by-partition)
Get the most recent lag measurement for each partition:
```
SELECT
datasource_id,
topic,
partition,
lag,
timestamp
FROM tinybird.kafka_ops_log
WHERE timestamp > now() - INTERVAL 1 hour
AND partition >= 0
AND msg_type = 'info'
ORDER BY timestamp DESC
LIMIT 1 BY datasource_id, topic, partition
```
### Consumer lag trend over time (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#consumer-lag-trend-over-time)
Track how lag changes over time to identify patterns and trends:
```
SELECT
toStartOfHour(timestamp) as hour,
datasource_id,
topic,
partition,
max(lag) as max_lag,
avg(lag) as avg_lag
FROM tinybird.kafka_ops_log
WHERE timestamp > now() - INTERVAL 24 hour
AND partition >= 0
AND msg_type = 'info'
GROUP BY hour, datasource_id, topic, partition
ORDER BY hour DESC, max_lag DESC
```
Rising lag trends may indicate that your current consumer group configuration cannot keep up with the message rate. This is particularly useful for understanding how Tinybird's consumer autoscaling responds to varying load patterns.
### Find partitions with high lag (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#find-partitions-with-high-lag)
Identify partitions that need immediate attention:
```
SELECT
datasource_id,
topic,
partition,
max(lag) as current_lag,
timestamp
FROM tinybird.kafka_ops_log
WHERE timestamp > now() - INTERVAL 1 hour
AND partition >= 0
AND msg_type = 'info'
GROUP BY datasource_id, topic, partition, timestamp
HAVING current_lag > 10000
ORDER BY current_lag DESC
```
High lag on specific partitions may indicate uneven message distribution, where some partitions receive more messages than others. This is valuable information for partition key design and scaling decisions.
## Track throughput and performance (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#track-throughput-and-performance)
Throughput monitoring helps you understand how efficiently your connector is processing messages. This is essential for performance tuning and understanding how the autoscaling infrastructure responds to varying message rates.
### Messages processed per hour by topic (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#messages-processed-per-hour-by-topic)
Track message processing rates over time:
```
SELECT
toStartOfHour(timestamp) as hour,
datasource_id,
topic,
sum(processed_messages) as total_messages,
sum(processed_bytes) as total_bytes,
sum(committed_messages) as total_committed
FROM tinybird.kafka_ops_log
WHERE timestamp > now() - INTERVAL 24 hour
AND msg_type = 'info'
GROUP BY hour, datasource_id, topic
ORDER BY hour DESC
```
Use this to track throughput trends and identify peak processing periods. This data helps you understand how consumer autoscaling responds to load changes and supports capacity planning.
### Processing rate comparison across partitions (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#processing-rate-comparison-across-partitions)
Compare processing rates to identify bottlenecks:
```
SELECT
datasource_id,
topic,
partition,
sum(processed_messages) as total_processed,
sum(committed_messages) as total_committed,
count(*) as operation_count,
sum(processed_messages) / count(*) as avg_messages_per_operation
FROM tinybird.kafka_ops_log
WHERE timestamp > now() - INTERVAL 1 hour
AND partition >= 0
AND msg_type = 'info'
GROUP BY datasource_id, topic, partition
ORDER BY total_processed DESC
```
Uneven processing rates may indicate that you need to review your partition key design or adjust your partition assignment strategy.
### Throughput efficiency (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#throughput-efficiency)
Compare processed messages to committed messages to understand your ingestion success rate:
```
SELECT
datasource_id,
topic,
sum(processed_messages) as processed,
sum(committed_messages) as committed,
(sum(committed_messages) * 100.0 / sum(processed_messages)) as success_rate_pct
FROM tinybird.kafka_ops_log
WHERE timestamp > now() - INTERVAL 24 hour
AND msg_type = 'info'
AND processed_messages > 0
GROUP BY datasource_id, topic
ORDER BY success_rate_pct ASC
```
A low success rate indicates that messages are being processed but not successfully ingested into ClickHouse, which may point to schema issues, data quality problems, or configuration errors. The connector's quarantine feature automatically handles problematic messages, and error details in `kafka_ops_log` help you debug and resolve issues. Learn more about [Quarantine Data Sources](/docs/forward/guides/quarantine).
## Monitor errors and warnings (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#monitor-errors-and-warnings)
Errors and warnings can indicate issues with your Kafka cluster connectivity, deserialization problems, Materialized View errors, or configuration issues. The serverless connector includes comprehensive error handling with quarantine for problematic messages. Regular monitoring of `kafka_ops_log` helps you catch and resolve issues before they impact your data pipeline. For troubleshooting guidance, see the [Kafka connector troubleshooting guide](/docs/forward/ingest-data/connectors/kafka/troubleshooting).
### Recent errors and warnings (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#recent-errors-and-warnings)
View all errors and warnings from your connector operations:
```
SELECT
timestamp,
datasource_id,
topic,
partition,
msg_type,
msg,
lag
FROM tinybird.kafka_ops_log
WHERE timestamp > now() - INTERVAL 24 hour
AND msg_type IN ('warning', 'error')
ORDER BY timestamp DESC
```
Review these messages to identify patterns and root causes. Common issues include partition rebalancing events, deserialization failures, and connectivity problems.
### Error count by Data Source and topic (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#error-count-by-data-source-and-topic)
Identify which Data Sources or topics are experiencing the most issues:
```
SELECT
datasource_id,
topic,
msg_type,
count(*) as error_count
FROM tinybird.kafka_ops_log
WHERE timestamp > now() - INTERVAL 24 hour
AND msg_type IN ('warning', 'error')
GROUP BY datasource_id, topic, msg_type
ORDER BY error_count DESC
```
This helps prioritize troubleshooting efforts and identify if specific topics or partitions are problematic.
### Group errors by message type (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#group-errors-by-message-type)
Identify recurring issues and patterns:
```
SELECT
msg,
msg_type,
count(*) as occurrence_count,
count(DISTINCT datasource_id) as affected_datasources
FROM tinybird.kafka_ops_log
WHERE timestamp > now() - INTERVAL 7 day
AND msg_type IN ('warning', 'error')
AND msg != ''
GROUP BY msg, msg_type
ORDER BY occurrence_count DESC
```
This analysis helps you understand common failure modes in your connector.
## Health checks (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#health-checks)
Regular health checks help you assess the overall status of your integration. These queries provide a comprehensive view of connector health, including consumer lag, throughput, and error rates. You can also set up automated [health checks](/docs/forward/monitoring/health-checks) for continuous monitoring.
### Connector health summary (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#connector-health-summary)
Get a comprehensive health overview of all your connectors:
```
SELECT
datasource_id,
topic,
max(lag) as max_lag,
sum(processed_messages) as total_processed,
sum(committed_messages) as total_committed,
countIf(msg_type = 'error') as error_count,
countIf(msg_type = 'warning') as warning_count,
max(timestamp) as last_activity
FROM tinybird.kafka_ops_log
WHERE timestamp > now() - INTERVAL 1 hour
GROUP BY datasource_id, topic
ORDER BY max_lag DESC
```
Use this for a quick assessment of your connector performance and to identify connectors that need attention.
### Find inactive connectors (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#find-inactive-connectors)
Find Data Sources that haven't processed messages recently:
```
SELECT
datasource_id,
topic,
max(timestamp) as last_activity,
now() - max(timestamp) as time_since_last_activity
FROM tinybird.kafka_ops_log
WHERE timestamp > now() - INTERVAL 7 day
GROUP BY datasource_id, topic
HAVING time_since_last_activity > INTERVAL 1 hour
ORDER BY time_since_last_activity DESC
```
Inactive connectors may indicate that the consumer has stopped, the topic has no new messages, or there's a connectivity issue.
## Partition-level analysis (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#partition-level-analysis)
Partition analysis helps you understand how your partition assignment strategy is working and identify partitions that may need attention. This analysis helps with scaling decisions and optimizing your partition key design.
### Partition performance comparison (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#partition-performance-comparison)
Compare performance metrics across all partitions:
```
SELECT
datasource_id,
topic,
partition,
max(lag) as max_lag,
avg(lag) as avg_lag,
sum(processed_messages) as total_messages,
countIf(msg_type = 'error') as errors,
countIf(msg_type = 'warning') as warnings
FROM tinybird.kafka_ops_log
WHERE timestamp > now() - INTERVAL 24 hour
AND partition >= 0
GROUP BY datasource_id, topic, partition
ORDER BY max_lag DESC, errors DESC
```
This helps you understand if your partition assignment is balanced and whether you need to adjust your setup.
### Lag distribution analysis (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#lag-distribution-analysis)
Understand the distribution of consumer lag across partitions:
```
SELECT
datasource_id,
topic,
quantile(0.5)(lag) as median_lag,
quantile(0.95)(lag) as p95_lag,
quantile(0.99)(lag) as p99_lag,
max(lag) as max_lag,
min(lag) as min_lag
FROM tinybird.kafka_ops_log
WHERE timestamp > now() - INTERVAL 1 hour
AND partition >= 0
AND msg_type = 'info'
GROUP BY datasource_id, topic
ORDER BY max_lag DESC
```
This helps identify if lag is evenly distributed or concentrated in specific partitions. Uneven lag distribution may indicate that you need to review your partition key design or consider rebalancing.
## Alerting queries (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#alerting-queries)
Set up automated alerts based on these queries to catch issues early. These queries are designed to trigger alerts when critical conditions are met, helping you maintain reliable data pipelines. Combine these with [health checks](/docs/forward/monitoring/health-checks) for comprehensive monitoring.
**Connect to monitoring tools**: You can connect these monitoring queries to external alerting tools like Grafana, Datadog, PagerDuty, and Slack. Query the [ClickHouse HTTP interface](/docs/forward/query-data/clickhouse-interface) directly, or create API endpoints from these queries. For Prometheus-compatible tools, you can export queries in [Prometheus format](/docs/forward/guides/consume-api-endpoints-in-prometheus-format). Configure your monitoring tools to poll these queries periodically and trigger alerts when thresholds are exceeded.
### High consumer lag alert (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#high-consumer-lag-alert)
Alert when consumer lag exceeds a threshold:
```
SELECT
datasource_id,
topic,
partition,
lag,
timestamp
FROM tinybird.kafka_ops_log
WHERE timestamp > now() - INTERVAL 5 minute
AND partition >= 0
AND msg_type = 'info'
AND lag > 50000
ORDER BY timestamp DESC
LIMIT 1 BY datasource_id, topic, partition
```
High lag indicates that your consumer is falling behind and may need scaling. With Tinybird's consumer autoscaling, the infrastructure automatically handles increased load, but this alert helps you verify autoscaling effectiveness.
### Error rate alert (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#error-rate-alert)
Alert when the error rate exceeds a threshold:
```
SELECT
datasource_id,
topic,
countIf(msg_type = 'error') as error_count,
count(*) as total_operations,
(countIf(msg_type = 'error') * 100.0 / count(*)) as error_rate_pct
FROM tinybird.kafka_ops_log
WHERE timestamp > now() - INTERVAL 15 minute
GROUP BY datasource_id, topic
HAVING error_rate_pct > 5
ORDER BY error_rate_pct DESC
```
High error rates indicate problems with your connector configuration, cluster connectivity, or data quality issues that need immediate attention.
### Processing stall alert (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#processing-stall-alert)
Alert when no messages are being processed:
```
SELECT
datasource_id,
topic,
max(timestamp) as last_activity,
sum(processed_messages) as messages_in_window
FROM tinybird.kafka_ops_log
WHERE timestamp > now() - INTERVAL 10 minute
GROUP BY datasource_id, topic
HAVING messages_in_window = 0
ORDER BY last_activity DESC
```
This indicates that your consumer may have stopped, there's a connectivity issue, or the topic has no new messages.
## Combine with other Service Data Sources (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#combine-with-other-service-data-sources)
You can join `tinybird.kafka_ops_log` with other [Service Data Sources](/docs/forward/monitoring/service-datasources) for deeper analysis. This supports comprehensive analysis that correlates connector operations with Data Source ingestion metrics. Learn more about all available [Service Data Sources](/docs/forward/monitoring/service-datasources#tinybird-kafka-ops-log).
### Correlate with Data Source operations (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#correlate-with-data-source-operations)
Join Kafka operations log with [Data Source operations log](/docs/forward/monitoring/service-datasources#tinybird-datasources-ops-log) to understand the complete pipeline from message consumption to ClickHouse ingestion:
```
SELECT
k.timestamp,
k.datasource_id,
k.topic,
k.partition,
k.processed_messages,
k.committed_messages,
d.rows as rows_in_datasource,
d.elapsed_time
FROM tinybird.kafka_ops_log k
LEFT JOIN (
SELECT
datasource_id,
rows,
elapsed_time,
timestamp
FROM tinybird.datasources_ops_log
WHERE event_type = 'append-kafka'
AND timestamp > now() - INTERVAL 1 hour
) d ON k.datasource_id = d.datasource_id
AND toStartOfMinute(k.timestamp) = toStartOfMinute(d.timestamp)
WHERE k.timestamp > now() - INTERVAL 1 hour
AND k.msg_type = 'info'
ORDER BY k.timestamp DESC
```
This helps you identify bottlenecks in the ingestion process and optimize your connector performance.
### Analyze ingestion using Kafka meta columns (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#analyze-ingestion-using-kafka-meta-columns)
Use the [Kafka meta columns](/docs/forward/ingest-data/connectors/kafka#kafka-meta-columns) in your Data Sources to analyze ingestion patterns. For example, track message gaps by analyzing offset sequences:
```
SELECT
__partition,
__topic,
min(__offset) as min_offset,
max(__offset) as max_offset,
count(*) as message_count,
max(__offset) - min(__offset) + 1 as expected_count,
(max(__offset) - min(__offset) + 1) - count(*) as missing_messages
FROM your_kafka_datasource
WHERE timestamp > now() - INTERVAL 1 hour
GROUP BY __partition, __topic
HAVING missing_messages > 0
```
This helps identify if any messages were skipped during ingestion. You can also analyze distribution to verify your partition key design:
```
SELECT
__partition,
count(*) as message_count,
count(DISTINCT __key) as unique_keys,
min(__timestamp) as first_message,
max(__timestamp) as last_message
FROM your_kafka_datasource
WHERE timestamp > now() - INTERVAL 24 hour
GROUP BY __partition
ORDER BY __partition
```
This shows how messages are distributed across partitions, helping you understand if your partition assignment is balanced.
## Schema evolution monitoring (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#schema-evolution-monitoring)
Monitor schema changes and their impact on ingestion:
```
SELECT
timestamp,
datasource_id,
topic,
msg,
count(*) as occurrence_count
FROM tinybird.kafka_ops_log
WHERE msg_type = 'warning'
AND (msg LIKE '%schema%' OR msg LIKE '%deserialization%')
AND timestamp > now() - INTERVAL 24 hour
GROUP BY timestamp, datasource_id, topic, msg
ORDER BY occurrence_count DESC
```
This helps identify schema evolution issues and deserialization problems that may affect ingestion performance.
## Performance tuning best practices (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#performance-tuning-best-practices)
Based on monitoring data from `tinybird.kafka_ops_log`, follow these best practices:
- **Monitor consumer lag regularly**: Set up alerts for high lag values to catch issues early. With Tinybird's serverless architecture, consumer autoscaling automatically handles increased load, but monitoring lag helps you verify that autoscaling is working effectively.
- **Track error patterns**: Review error messages periodically to identify recurring issues. Common problems include partition assignment issues, deserialization failures, and connectivity problems.
- **Compare partition performance**: If one partition has significantly higher lag than others, investigate the cause. This may indicate uneven message distribution and the need to review your partition key design.
- **Monitor throughput trends**: Track processing rates over time to understand how the autoscaling infrastructure responds to varying message rates.
- **Use time windows**: Filter queries by recent time windows (for example, last hour, last 24 hours) to focus on current state and avoid analyzing stale data.
- **Analyze partition assignment**: Use partition-level metrics to understand if your partition assignment strategy is working effectively. Uneven partition loads may require rebalancing or adjustments to your setup.
- **Optimize partition key design**: If you see uneven lag distribution across partitions, consider reviewing your partition key design to ensure messages are distributed evenly.
For detailed performance optimization strategies, see the [performance optimization guide](/docs/forward/ingest-data/connectors/kafka/guides/performance-optimization).
## Next steps (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#next-steps)
### Kafka connector documentation (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#kafka-connector-documentation)
- **[Kafka connector guide](/docs/forward/ingest-data/connectors/kafka)** - Complete setup, configuration, and troubleshooting guide
- **[Troubleshooting guide](/docs/forward/ingest-data/connectors/kafka/troubleshooting)** - Common errors and solutions
- **[Performance optimization](/docs/forward/ingest-data/connectors/kafka/guides/performance-optimization)** - Throughput tuning and best practices
- **[Partitioning strategies](/docs/forward/ingest-data/connectors/kafka/guides/partitioning-strategies)** - Partition optimization
- **[Schema management](/docs/forward/ingest-data/connectors/kafka/guides/schema-management)** - Schema evolution and data type mapping
### Monitoring and observability (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#monitoring-and-observability)
- **[Service Data Sources](/docs/forward/monitoring/service-datasources)** - Explore other monitoring options
- **[Health checks](/docs/forward/monitoring/health-checks)** - Automated monitoring setup
- **[Data Source operations](/docs/forward/monitoring/service-datasources#tinybird-datasources-ops-log)** - Monitor ingestion performance
- **[Workspace jobs](/docs/forward/monitoring/jobs)** - Additional pipeline observability
### Integrate with monitoring tools (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#integrate-with-monitoring-tools)
Connect monitoring queries to Grafana, Datadog, PagerDuty, Slack, and other alerting systems by querying the [ClickHouse HTTP interface](/docs/forward/query-data/clickhouse-interface) directly. You can also create API endpoints from these queries, or export them in [Prometheus format](/docs/forward/guides/consume-api-endpoints-in-prometheus-format) for Prometheus-compatible tools.
---
URL: https://www.tinybird.co/docs/forward/monitoring/jobs
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Monitor jobs in your workspace \| Tinybird"
theme-color: "#171612"
description: "Many of the operations you can run in your workspace are executed using jobs. jobs_log provides you with an overview of all your jobs."
inkeep:version: "forward"
---
# Monitor jobs in your workspace (https://www.tinybird.co/docs/forward/monitoring/jobs#monitor-jobs-in-your-workspace)
Many operations in your Tinybird workspace, like imports, copy jobs, sinks and populates, run as background jobs within the platform. This approach ensures that the system can handle a large volume of requests efficiently without causing timeouts or delays in your workflow.
Monitoring and managing jobs, for example querying job statuses, types, and execution details, is essential for maintaining a healthy workspace. The two mechanisms for generic job monitoring are the [Jobs API](/docs/api-reference/jobs-api) and the `jobs_log` data source.
You can also track more specific things using dedicated Service data sources, such as `datasources_ops_log` for import, replaces, or copy, or `sinks_ops_log` for sink operations, or tracking jobs across [Organizations](/docs/forward/core-concepts/organizations) with `organization.jobs_log`. See [Service data sources docs](./service-datasources).
The Jobs API and the `jobs_log` return identical information about job execution. However, the Jobs API has some limitations: It reports only on a single workspace, returns only 100 records, from the last 48 hours. If you want to monitor jobs outside these parameters, use the `jobs_log` data source.
## Track a specific job (https://www.tinybird.co/docs/forward/monitoring/jobs#track-a-specific-job)
The most elemental use case is to track a specific job. You can do this using the Jobs API or SQL queries.
### Jobs API (https://www.tinybird.co/docs/forward/monitoring/jobs#jobs-api-track-specific-job)
The Jobs API is a convenient way to programmatically check the status of a job. By sending a GET request, you can retrieve detailed information about a specific job. This method is particularly useful for integration into scripts or applications.
```
curl \
-X GET "https://$TB_HOST/v0/jobs/{job_id}" \
-H "Authorization: Bearer $TOKEN"
```
Replace `{job_id}` with the actual job ID.
The API host in the following examples must match your Workspace's region. See the full list of [regions and hosts](/docs/api-reference#regions-and-endpoints)
### SQL queries (https://www.tinybird.co/docs/forward/monitoring/jobs#sql-queries-track-specific-job)
Alternatively, you can use SQL to query the `jobs_log` data source from directly within a Tinybird pipe. This method is ideal for users who are comfortable with SQL and prefer to run queries directly against the data, and then expose them with an endpoint or perform any other actions with it.
```
SELECT * FROM tinybird.jobs_log WHERE job_id='{job_id}'
```
Replace `{job_id}` with the desired job ID. This query retrieves all columns for the specified job, providing comprehensive details about its execution.
## Track specific job types (https://www.tinybird.co/docs/forward/monitoring/jobs#track-specific-job-types)
Track jobs by type to monitor and analyze all jobs of a certain category, such as all `copy` jobs. This can help you understand the performance and status of specific job types across your entire Workspace.
### Jobs API (https://www.tinybird.co/docs/forward/monitoring/jobs#jobs-api-track-specific-job-types)
You can fetch all jobs of a specific type by making a GET request against the Jobs API:
```
curl \
-X GET "https://$TB_HOST/v0/jobs?kind=copy" \
-H "Authorization: Bearer $TOKEN"
```
Replace `copy` with the type of job you want to track. Make sure you have set your Tinybird host (`$TB_HOST`) and authorization token (`$TOKEN`) correctly.
### SQL queries (https://www.tinybird.co/docs/forward/monitoring/jobs#sql-queries-track-specific-job-types)
Alternatively, you can run an SQL query to fetch all jobs of a specific type from the `jobs_log` data source:
```
SELECT * FROM tinybird.jobs_log WHERE job_type='copy'
```
Replace `copy` with the desired job type. This query retrieves all columns for jobs of the specified type.
## Track ongoing jobs (https://www.tinybird.co/docs/forward/monitoring/jobs#track-ongoing-jobs)
To keep track of jobs that are currently running, you can query the status of jobs in progress. This helps in monitoring the real-time workload and managing system performance.
### Jobs API (https://www.tinybird.co/docs/forward/monitoring/jobs#jobs-api-ongoing-jobs)
By making an HTTP GET request to the Jobs API, you can fetch all jobs that are currently in the `working` status:
```
curl \
-X GET "https://$TB_HOST/v0/jobs?status=working" \
-H "Authorization: Bearer $TOKEN"
```
This call retrieves jobs that are actively running. Ensure you have set your Tinybird host (`$TB_HOST`) and authorization token (`$TOKEN`) correctly.
### SQL queries (https://www.tinybird.co/docs/forward/monitoring/jobs#sql-queries-ongoing-jobs)
You can also use an SQL query to fetch currently running jobs from the `jobs_log` data source:
```
SELECT * FROM tinybird.jobs_log WHERE status='working'
```
This query retrieves all columns for jobs with the status `working`, allowing you to monitor ongoing operations.
## Track errored jobs (https://www.tinybird.co/docs/forward/monitoring/jobs#track-errored-jobs)
Tracking errored jobs is crucial for identifying and resolving issues that may arise during job execution. Jobs API or SQL queries to `jobs_log` helps you monitor jobs that errored during the execution.
### Jobs API (https://www.tinybird.co/docs/forward/monitoring/jobs#jobs-api-errored-jobs)
You can use the Jobs API to fetch details of jobs that have ended in error.
Use the following `curl` command to retrieve all jobs that have a status of `error`:
```
curl \
-X GET "https://$TB_HOST/v0/jobs?status=error" \
-H "Authorization: Bearer $TOKEN"
```
This call fetches a list of jobs that are currently in an errored state, providing details that can be used for further analysis or debugging. Make sure you've set your Tinybird host (`$TB_HOST`) and authorization token (`$TOKEN`) correctly.
### SQL queries (https://www.tinybird.co/docs/forward/monitoring/jobs#sql-queries-errored-jobs)
Alternatively, you can use SQL to query the `jobs_log` data source directly.
Use the following SQL query to fetch job IDs, job types, and error messages for jobs that have encountered errors in the past day:
```
SELECT job_id, job_type, error
FROM tinybird.jobs_log
WHERE
status='error' AND
created_at > now() - INTERVAL 1 DAY
```
### Track success rate (https://www.tinybird.co/docs/forward/monitoring/jobs#track-success-rate)
Extrapolating from errored jobs, you can also use `jobs_log` to calculate the success rate of your workspace jobs:
```
SELECT
job_type,
pipe_id,
countIf(status='done') AS job_success,
countIf(status='error') AS job_error,
job_success / (job_success + job_error) as success_rate
FROM tinybird.jobs_log
WHERE
created_at > now() - INTERVAL 1 DAY
GROUP BY job_type, pipe_id
```
## Get job execution metadata (https://www.tinybird.co/docs/forward/monitoring/jobs#get-job-execution-metadata)
In the `jobs_log` data source, there is a property called `job_metadata` that contains metadata related to job executions. This includes the execution type, manual or scheduled, for Copy and Sink jobs, or the count of quarantined rows for append operations, along with many other properties.
Use JSON functions within SQL queries to extract and analyze this metadata. You can get information about job executions directly from the `jobs_log` Data Source.
The following SQL query is an example of how to extract specific metadata fields from the `job_metadata` property, such as the import mode and counts of quarantined rows and invalid lines, and how to aggregate this data for analysis:
```
SELECT
job_type,
JSONExtractString(job_metadata, 'mode') AS import_mode,
sum(simpleJSONExtractUInt(job_metadata, 'quarantine_rows')) AS quarantine_rows,
sum(simpleJSONExtractUInt(job_metadata, 'invalid_lines')) AS invalid_lines
FROM tinybird.jobs_log
WHERE
job_type='import' AND
created_at >= toStartOfDay(now())
GROUP BY job_type, import_mode
```
You can build many other use cases with the properties in `job_metadata`.
## Advanced use cases (https://www.tinybird.co/docs/forward/monitoring/jobs#advanced-use-cases)
Beyond basic tracking, you can leverage the `jobs_log` data source for more advanced use cases, such as gathering statistics and performance metrics. This can help you optimize job scheduling and resource allocation.
### Get queue status (https://www.tinybird.co/docs/forward/monitoring/jobs#get-queue-status)
The following SQL query returns the number of jobs that are waiting to be executed, the number of jobs that are in progress, and how many of them are done already:
```
SELECT
job_type,
countIf(status='waiting') AS jobs_in_queue,
countIf(status='working') AS jobs_in_progress,
countIf(status='done') AS jobs_succeeded,
countIf(status='error') AS jobs_errored
FROM tinybird.jobs_log
WHERE
created_at > now() - INTERVAL 1 DAY
GROUP BY job_type
```
### Run time statistics grouped by type of job (https://www.tinybird.co/docs/forward/monitoring/jobs#run-time-statistics-grouped-by-type-of-job)
The following SQL query calculates the maximum, minimum, median, and p95 running time (in seconds) grouped by type of job over the past day. This helps in understanding the efficiency of different job types:
```
SELECT
job_type,
max(date_diff('s', started_at, updated_at)) as max_run_time_in_secs,
min(date_diff('s', started_at, updated_at)) as min_run_time_in_secs,
median(date_diff('s', started_at, updated_at)) as median_run_time_in_secs,
quantile(0.95)(date_diff('s', started_at, updated_at)) as p95_run_time_in_secs
FROM tinybird.jobs_log
WHERE
created_at > now() - INTERVAL 1 DAY
GROUP BY job_type
```
### Statistics on queue time by type of job (https://www.tinybird.co/docs/forward/monitoring/jobs#statistics-on-queue-time-by-type-of-job)
The following SQL query calculates the average queue time, in seconds, for a specific type of job over the past day. This can help in identifying bottlenecks in job scheduling:
```
SELECT
job_type,
max(date_diff('s', created_at, started_at)) as max_run_time_in_secs,
min(date_diff('s', created_at, started_at)) as min_run_time_in_secs,
median(date_diff('s', created_at, started_at)) as median_run_time_in_secs,
quantile(0.95)(date_diff('s', created_at, started_at)) as p95_run_time_in_secs
FROM tinybird.jobs_log
WHERE
created_at > now() - INTERVAL 1 DAY
GROUP BY job_type
```
### Get statistics on job completion rate (https://www.tinybird.co/docs/forward/monitoring/jobs#get-statistics-on-job-completion-rate)
The following SQL query calculates the success rate by job type, for example `copy`, and Pipe over the past day. This can help you assess workflow reliability by measuring job completion rates and finding potential issues:
```
SELECT
job_type,
pipe_id,
countIf(status='done') AS job_success,
countIf(status='error') AS job_error,
job_success / (job_success + job_error) as success_rate
FROM tinybird.jobs_log
WHERE
created_at > now() - INTERVAL 1 DAY
GROUP BY job_type, pipe_id
```
### Statistics on the amount of manual versus scheduled run jobs (https://www.tinybird.co/docs/forward/monitoring/jobs#statistics-on-the-amount-of-manual-versus-scheduled-run-jobs)
The following SQL query calculates the percentage of manual and scheduled jobs. Use the distribution of manually executed and scheduled jobs to identify unexpected on-demand jobs:
```
SELECT
job_type,
countIf(JSONExtractString(job_metadata, 'execution_type')='manual') AS job_manual,
countIf(JSONExtractString(job_metadata, 'execution_type')='scheduled') AS job_scheduled
FROM tinybird.jobs_log
WHERE
job_type='copy' AND
created_at > now() - INTERVAL 1 DAY
GROUP BY job_type
```
## Next steps (https://www.tinybird.co/docs/forward/monitoring/jobs#next-steps)
- Read up on the `jobs_log` Service data source specification.
---
URL: https://www.tinybird.co/docs/forward/monitoring/health-checks
Last update: 2026-05-29T14:27:44.000Z
Content:
---
title: "Health checks \| Tinybird"
theme-color: "#171612"
description: "Use the built-in Tinybird tools to monitor your data ingestion and API endpoint processes."
inkeep:version: "forward"
---
# Check the health of your data sources (https://www.tinybird.co/docs/forward/monitoring/health-checks#check-the-health-of-your-data-sources)
After you have fixed all the possible errors in your source files, matched the data source schema to your needs, and done on-the-fly transformations, you can start ingesting data periodically. Knowing the status of your ingestion processes helps you to keep your data clean and consistent.
## Data source summary (https://www.tinybird.co/docs/forward/monitoring/health-checks#data-source-summary)
Select a data source to see the size of the data source, the number of rows, the number of rows in the quarantine data source, and when it was last updated. The log contains details of the events for the data source, which appears as the results of the query.
## Service data sources for continuous monitoring (https://www.tinybird.co/docs/forward/monitoring/health-checks#service-data-sources-for-continuous-monitoring)
[Service data sources](./service-datasources) can help you with ingestion health checks. You can use them like any other data source in your Workspace, which means you can create API Endpoints to monitor your ingestion processes.
Querying the `tinybird.datasources_ops_log` directly, you can, for example, lists your ingest processes during the last week:
LISTING INGESTIONS IN THE LAST 7 DAYS
```
SELECT *
FROM tinybird.datasources_ops_log
WHERE toDate(timestamp) > now() - INTERVAL 7 DAY
ORDER BY timestamp DESC
```
This query calculates the percentage of quarantined rows for a given period of time:
CALCULATE % OF ROWS THAT WENT TO QUARANTINE
```
SELECT
countIf(result != 'ok') / countIf(result == 'ok') * 100 percentage_failed,
sum(rows_quarantine) / sum(rows) * 100 quarantined_rows
FROM tinybird.datasources_ops_log
```
The following query monitors the average duration of your periodic ingestion processes for a given data source:
CALCULATING AVERAGE INGEST DURATION
```
SELECT avg(elapsed_time) avg_duration
FROM tinybird.datasources_ops_log
WHERE datasource_id = 't_8417d5126ed84802aa0addce7d1664f2'
```
If you want to configure or build an external service that monitors these metrics, you need to create an Endpoint and raise an alert when passing a threshold. When you receive an alert, you can check the quarantine data source or the Operations log to see what's going on and fix your source files or ingestion processes.
## Monitoring API endpoints (https://www.tinybird.co/docs/forward/monitoring/health-checks#monitoring-api-endpoints)
You can use the `pipe_stats` and `pipe_stats_rt` [Service data sources](./service-datasources) to monitor the performance of your Endpoints.
Every request to a pipe is logged to `tinybird.pipe_stats_rt` and kept in this data source for the last week.
The following example Endpoint aggregates the statistics for each hour for the selected pipe.
PIPE_STATS_RT_BY_HR
```
SELECT
toStartOfHour(start_datetime) as hour,
count() as view_count,
round(avg(duration), 2) as avg_time,
arrayElement(quantiles(0.50)(duration),1) as quantile_50,
arrayElement(quantiles(0.95)(duration),1) as quantile_95,
arrayElement(quantiles(0.99)(duration),1) as quantile_99
FROM tinybird.pipe_stats_rt
WHERE pipe_id = 'PIPE_ID'
GROUP BY hour
ORDER BY hour
```
`pipe_stats` contains statistics about your pipe Endpoints' API calls aggregated per day using intermediate states.
PIPE_STATS_BY_DATE
```
SELECT
date,
sum(view_count) view_count,
sum(error_count) error_count,
avgMerge(avg_duration_state) avg_time,
quantilesTimingMerge(0.9, 0.95, 0.99)(quantile_timing_state) quantiles_timing_in_millis_array
FROM tinybird.pipe_stats
WHERE pipe_id = 'PIPE_ID'
GROUP BY date
ORDER BY date
```
You can use these API endpoints to trigger alerts whenever statistics pass predefined thresholds. [Export API endpoint statistics in Prometheus format](/docs/forward/guides/consume-api-endpoints-in-prometheus-format) to integrate with your monitoring and alerting tools.
---
URL: https://www.tinybird.co/docs/forward/monitoring/analyze-endpoints-performance
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Analyze Endpoint Performance \| Tinybird"
theme-color: "#171612"
description: "Learn more about how to measure the performance of your API Endpoints."
inkeep:version: "forward"
---
# Analyze the performance of your API Endpoints (https://www.tinybird.co/docs/forward/monitoring/analyze-endpoints-performance#analyze-the-performance-of-your-api-endpoints)
You can use the `pipe_stats` and `pipe_stats_rt` Service Data Sources to analyze the performance of your Endpoints. Read on to see several practical examples that show what you can do with these Data Sources.
## Knowing what to optimize (https://www.tinybird.co/docs/forward/monitoring/analyze-endpoints-performance#knowing-what-to-optimize)
Before you optimize, you need to know what to optimize. The `pipe_stats` and `pipe_stats_rt` Service Data Sources let you see how your Endpoints are performing, so you can find causes of overhead and improve performance.
These Service Data Sources provide performance data and consumption data for every single request. You can also filter and sort results by Tokens to see who is accessing your Endpoints and how often.
The difference between `pipe_stats_rt` and `pipe_stats` is that `pipe_stats` provides aggregate stats, like average request duration and total read bytes, per day, whereas `pipe_stats_rt` offers the same information but without aggregation. Every single request is stored in `pipe_stats_rt`. The examples in this guide use `pipe_stats_rt`, but you can use the same logic with `pipe_stats` if you need more than 7 days of lookback.
## Before you start (https://www.tinybird.co/docs/forward/monitoring/analyze-endpoints-performance#before-you-start)
You need a high-level understanding of Tinybird's [Service Data Sources](./service-datasources).
### Understand the core stats (https://www.tinybird.co/docs/forward/monitoring/analyze-endpoints-performance#understand-the-core-stats)
This guide focuses on the following fields in the `pipe_stats_rt` Service Data Source:
- `pipe_name` (String): Pipe name as returned in Pipes API.
- `duration` (Float): the duration in seconds of each specific request.
- `read_bytes` (UInt64): How much data was scanned for this particular request.
- `read_rows` (UInt64): How many rows were scanned.
- `token_name` (String): The name of the Token used in a particular request.
- `status_code` (Int32): The HTTP status code returned for this particular request.
- `memory_usage` (UInt64): Memory consumption by the query, in bytes. High memory usage indicates the query may not be optimized efficiently.
You can find the full schema for `pipe_stats_rt` in the [API docs](./service-datasources#tinybird-pipe-stats-rt).
The value of `pipe_name` is "query_api" in the event as it's a Query API request. The following section covers how to monitor query performance when using the Query API.
### Use the Query API with metadata parameters (https://www.tinybird.co/docs/forward/monitoring/analyze-endpoints-performance#use-the-query-api-with-metadata-parameters)
If you are using the [Query API](/docs/api-reference/query-api) to run queries in Tinybird you can still track query performance using the `pipe_stats_rt` Service Data Source. Add metadata related to the query as request parameters, as well as any existing parameters used in your query.
For example, when running a query against the Query API, use a parameter called `app_name` to track all queries from the "explorer" app. Here's an example using `curl`:
Using the metadata parameters with the Query API
```
curl -X POST \
-H "Authorization: Bearer " \
--data "% SELECT * FROM events LIMIT {{Int8(my_limit, 10)}}" \
"https:///v0/sql?my_limit=10&app_name=explorer"
```
When you run the following queries, use the `parameters` attribute to access those queries where `app_name` equals "explorer":
Simple Parameterized Query
```
SELECT *
FROM tinybird.pipe_stats_rt
WHERE parameters['app_name'] = 'explorer'
```
## Detect errors in your Endpoints (https://www.tinybird.co/docs/forward/monitoring/analyze-endpoints-performance#detect-errors-in-your-endpoints)
If you want to monitor the number of errors per Endpoint over the last hour, you can run the following query:
Errors in the last hour
```
SELECT
pipe_name,
status_code,
count() as error_count
FROM tinybird.pipe_stats_rt
WHERE status_code >= 400
AND start_datetime > now() - INTERVAL 1 HOUR
GROUP BY pipe_name, status_code
ORDER BY status_code desc
```
If you have errors, the query would return something like:
```
Pipe_a | 404 | 127
Pipe_b | 403 | 32
```
With one query, you can see in real time if your Endpoints are experiencing errors, and investigate further if so.
## Analyze the performance of Endpoints over time (https://www.tinybird.co/docs/forward/monitoring/analyze-endpoints-performance#analyze-the-performance-of-endpoints-over-time)
You can also use `pipe_stats_rt` to track how long API calls take using the `duration` field, and seeing how that changes over time. API performance is directly related to how much data you are reading per request, so if your Endpoint is dynamic, request duration varies.
For instance, it might be receiving start and end date parameters that alter how long a period is being read.
API Endpoint performance over time
```
SELECT
toStartOfMinute(start_datetime) t,
pipe_name,
avg(duration) avg_duration,
quantile(.95)(duration) p95_duration,
count() requests
FROM tinybird.pipe_stats_rt
WHERE
start_datetime >= {{DateTime(start_date_time, '2022-05-01 00:00:00', description="Start date time")}} AND
start_datetime < {{DateTime(end_date_time, '2022-05-25 00:00:00', description="End date time")}}
GROUP BY t, pipe_name
ORDER BY t desc, pipe_name
```
## Monitor memory usage of Endpoints (https://www.tinybird.co/docs/forward/monitoring/analyze-endpoints-performance#monitor-memory-usage-of-endpoints)
Memory usage is an important metric for identifying inefficient queries. High memory consumption can indicate complex operations, large result sets, or unoptimized queries that need attention.
Here's an example of using `pipe_stats_rt` to find the Endpoints with the highest memory usage in the last 24 hours:
High memory usage endpoints last 24 hours
```
SELECT
pipe_name,
formatReadableSize(avg(memory_usage) as avg_memory) as avg_memory_usage,
formatReadableSize(max(memory_usage)) as max_memory_usage,
formatReadableSize(quantile(0.9)(memory_usage)) as p90_memory_usage,
count() as request_count
FROM tinybird.pipe_stats_rt
WHERE
start_datetime >= now() - INTERVAL 24 HOUR
GROUP BY pipe_name
ORDER BY avg_memory DESC
```
You can also identify endpoints that show high memory usage relative to the amount of data they process:
Memory efficiency analysis
```
SELECT
pipe_name,
formatReadableSize(avg(memory_usage)) as avg_memory_usage,
formatReadableSize(avg(read_bytes)) as avg_read_bytes,
avg(memory_usage) / avg(read_bytes) as memory_per_byte_ratio,
count() as request_count
FROM tinybird.pipe_stats_rt
WHERE
start_datetime >= now() - INTERVAL 24 HOUR
AND read_bytes > 0
GROUP BY pipe_name
ORDER BY memory_per_byte_ratio DESC
```
## Find the endpoints that process the most data (https://www.tinybird.co/docs/forward/monitoring/analyze-endpoints-performance#find-the-endpoints-that-process-the-most-data)
You might want to find Endpoints that repeatedly scan large amounts of data. They are your best candidates for optimization to reduce time and spend.
Here's an example of using `pipe_stats_rt` to find the Endpoints that have processed the most data as a percentage of all processed data in the last 24 hours:
Most processed data last 24 hours
```
WITH (
SELECT sum(read_bytes)
FROM tinybird.pipe_stats_rt
WHERE
start_datetime >= now() - INTERVAL 24 HOUR
) as total,
sum(read_bytes) as processed_byte
SELECT
pipe_id,
quantile(0.9)(duration) as p90,
formatReadableSize(processed_byte) AS processed_formatted,
processed_byte*100/total as percentage
FROM tinybird.pipe_stats_rt
WHERE
start_datetime >= now() - INTERVAL 24 HOUR
GROUP BY pipe_id
ORDER BY percentage DESC
```
### Include consumption of the Query API (https://www.tinybird.co/docs/forward/monitoring/analyze-endpoints-performance#include-consumption-of-the-query-api)
If you use Tinybird's Query API to query your Data Sources directly, you probably want to include in your analysis which queries are consuming more.
Whenever you use the Query API, the field `pipe_name` contain the value `query_api`. The actual query is included as part of the `q` parameter in the `url` field. You can modify the query in the previous section to extract the SQL query that's processing the data.
Using the Query API
```
WITH (
SELECT sum(read_bytes)
FROM tinybird.pipe_stats_rt
WHERE
start_datetime >= now() - INTERVAL 24 HOUR
) as total,
sum(read_bytes) as processed_byte
SELECT
if(pipe_name = 'query_api', normalizeQuery(extractURLParameter(decodeURLComponent(url), 'q')),pipe_name) as pipe_name,
quantile(0.9)(duration) as p90,
formatReadableSize(processed_byte) AS processed_formatted,
processed_byte*100/total as percentage
FROM tinybird.pipe_stats_rt
WHERE
start_datetime >= now() - INTE
RVAL 24 HOUR
GROUP BY pipe_name
ORDER BY percentage DESC
```
## Monitor usage of Tokens (https://www.tinybird.co/docs/forward/monitoring/analyze-endpoints-performance#monitor-usage-of-tokens)
If you use your Endpoint with different Tokens, for example if allowing different customers to check their own data, you can track and control which Tokens are being used to access these endpoints.
The following example shows, for the last 24 hours, the number and size of requests per Token:
Token usage last 24 hours
```
SELECT
count() requests,
formatReadableSize(sum(read_bytes)) as total_read_bytes,
token_name
FROM tinybird.pipe_stats_rt
WHERE
start_datetime >= now() - INTERVAL 24 HOUR
GROUP BY token_name
ORDER BY requests DESC
```
To get this information, request the Token name (`token_name` column) or id (`token` column).
## Next steps (https://www.tinybird.co/docs/forward/monitoring/analyze-endpoints-performance#next-steps)
- Learn how to [monitor jobs in your Workspace](./jobs).
- Monitor the [latency of your Endpoints](./latency).
---
URL: https://www.tinybird.co/docs/forward/ingest-data/table-functions
Last update: 2026-07-31T11:10:11.000Z
Content:
---
title: "Table functions \| Tinybird"
theme-color: "#171612"
description: "Ingest data from remote databases into Tinybird."
inkeep:version: "forward"
---
# Table functions (https://www.tinybird.co/docs/forward/ingest-data/table-functions#table-functions)
Tinybird table functions allow you to read data from an existing database and schedule a regular copy pipe to orchestrate synchronization. You can load full tables or incrementally sync your data.
Tinybird supports the following table functions:
- [Apache Iceberg](/docs/forward/ingest-data/table-functions/iceberg)
- [MySQL](/docs/forward/ingest-data/table-functions/mysql)
- [PostgreSQL](/docs/forward/ingest-data/table-functions/postgresql)
- [URL](/docs/forward/ingest-data/table-functions/url)
## Prerequisites (https://www.tinybird.co/docs/forward/ingest-data/table-functions#prerequisites)
Your database needs to be open and public, exposed to the internet with publicly signed certs, so you can connect to it by passing the hostname, port, username, and password.
Store credentials as Tinybird secrets. Do not paste usernames, passwords, or access keys as plain text in pipe SQL or datafiles.
```
tb secret set DB_USERNAME
tb secret set DB_PASSWORD
```
Set the secret in each environment where the copy pipe runs. For example, use `tb --cloud secret set` for Cloud and `tb --branch= secret set` for a branch.
Reference those secrets in the table function with `tb_secret()`:
```
{{ tb_secret("DB_USERNAME") }}
{{ tb_secret("DB_PASSWORD") }}
```
In Tinybird Classic, use the [Environment Variables API](/docs/api-reference/environment-variables-api#post-v0secrets) to create the same secret values.
## Load an external table (https://www.tinybird.co/docs/forward/ingest-data/table-functions#load-an-external-table)
Create a new pipe node. Call the table function in the `FROM` and pass the connection details:
PostgreSQL table function example
```
SELECT *
FROM postgresql(
'postgresql.example.com:5432',
'postgres',
'orders',
{{ tb_secret("DB_USERNAME") }},
{{ tb_secret("DB_PASSWORD") }}
)
```
Publish this node as a Copy Pipe. You can append only new data or replace all data.
### Using datafiles (https://www.tinybird.co/docs/forward/ingest-data/table-functions#using-datafiles)
You can also define node logic in [Pipe files](/docs/forward/dev-reference/datafiles/pipe-files). For example, this `orders_backfill` pipe reads a PostgreSQL `orders` table and replaces the target Data Source on demand:
```
NODE all_orders
SQL >
%
SELECT *
FROM postgresql(
'postgresql.example.com:5432',
'postgres',
'orders',
{{ tb_secret("DB_USERNAME") }},
{{ tb_secret("DB_PASSWORD") }}
)
TYPE COPY
TARGET_DATASOURCE orders
COPY_SCHEDULE @on-demand
COPY_MODE replace
```
## Include filters (https://www.tinybird.co/docs/forward/ingest-data/table-functions#include-filters)
You can use a source column to filter by a value in Tinybird, for example:
Example copy pipe with PostgreSQL table function and filters
```
SELECT *
FROM postgresql(
'postgresql.example.com:5432',
'postgres',
'orders',
{{ tb_secret("DB_USERNAME") }},
{{ tb_secret("DB_PASSWORD") }}
)
WHERE orderDate > (SELECT max(orderDate) FROM orders)
```
## Schedule runs (https://www.tinybird.co/docs/forward/ingest-data/table-functions#schedule-runs)
When publishing as a copy pipe, most users set it to run at a frequent interval using a cron expression.
You can also trigger the copy pipe on demand:
```
curl -H "Authorization: Bearer " \
-X POST "https:///v0/pipes//run"
```
Having on-demand pipes in your workspace is helpful, as you can run a full sync manually any time you need it. You might also use them for weekly full syncs.
## Synchronization strategies (https://www.tinybird.co/docs/forward/ingest-data/table-functions#synchronization-strategies)
When copying data from your database to Tinybird, you can use one of the following strategies:
- Use `COPY_MODE replace` to synchronize small dimensions tables, up to a few million rows, in a frequent schedule (1 to 5 minutes).
- Use `COPY_MODE append` to do incremental appends. For example, you can append events data tagged with a timestamp. Combine it with `COPY_SCHEDULE` and filters in the copy pipe SQL to sync the new events.
### Timeouts (https://www.tinybird.co/docs/forward/ingest-data/table-functions#timeouts)
When synchronizing dimensions tables with `COPY_MODE replace` and 1 minute schedule, the copy job might time out because it can't ingest the whole table in the defined schedule.
Timeouts depend on several factors:
- The timeout configured in your external database.
- The external database load.
- Network connectivity, for example when copying data from different cloud regions.
Follow these tips to avoid timeouts using incremental appends:
- Make sure to tag your data with an updated timestamp and use the column to filter the copy pipe SQL.
- Configure the Copy Pipe with an incremental append strategy and 1 minute schedule. This configuration ingests only new records from the last minute, optimizing the copy job duration.
- Create an index in the external table to speed up filtering.
- Create the target data source as a ReplacingMergeTree using a unique or primary key as the `ENGINE_SORTING_KEY`. Rows with the same `ENGINE_SORTING_KEY` are deduplicated. Remember to use the `FINAL` keyword when querying the data source to force deduplication at query time.
- Combine this approach with an hourly or daily replacement to get rid of deleted rows.
## Observability (https://www.tinybird.co/docs/forward/ingest-data/table-functions#observability)
Job executions are logged in the `datasources_ops_log` [Service Data Source](/docs/forward/monitoring/service-datasources). You can check this log directly in the Data Source view page in the UI. Filter by `datasource_id` to monitor ingestion through the table functions from the `datasources_ops_log`:
Example query to the datasources_ops_log Service data source
```
SELECT
timestamp,
event_type,
result,
error,
job_id
FROM
tinybird.datasources_ops_log
WHERE
datasource_id = 't_1234'
AND
event_type = 'copy'
ORDER BY timestamp DESC
```
## Limits (https://www.tinybird.co/docs/forward/ingest-data/table-functions#limits)
The table functions inherit all the [limits of copy pipes](/docs/forward/pricing/limits#copy-pipe-limits).
Check the [limits page](/docs/forward/pricing/limits) for limits on ingestion, queries, API Endpoints, and more.
## Billing (https://www.tinybird.co/docs/forward/ingest-data/table-functions#billing)
There are no additional or specific costs for the table function itself. Only the costs associated with Copy Pipes apply. For more information on data operations and their charges, see [Developer](/docs/forward/pricing/developer), [SaaS](/docs/forward/pricing/saas), and [Enterprise](/docs/forward/pricing/enterprise).
---
URL: https://www.tinybird.co/docs/forward/ingest-data/local-file
Last update: 2026-06-02T11:48:14.000Z
Content:
---
title: "Ingest data from files \| Tinybird"
theme-color: "#171612"
description: "Learn how to ingest data from files to Tinybird."
inkeep:version: "forward"
---
# Ingest data from files (https://www.tinybird.co/docs/forward/ingest-data/local-file#ingest-data-from-files)
You can ingest data from files to Tinybird using the [Data sources API](/docs/api-reference/datasource-api), the [tb datasource](/docs/forward/dev-reference/commands/tb-datasource) CLI command, the TypeScript SDK, or the Python SDK. [Ingestion limits](/docs/forward/pricing/limits#ingestion-limits) apply.
## Supported file types (https://www.tinybird.co/docs/forward/ingest-data/local-file#supported-file-types)
Tinybird supports these file types and compression formats at ingest time:
| File type | Method | Accepted extensions | Compression formats supported |
| --- | --- | --- | --- |
| CSV | File upload, URL | `.csv`, `.csv.gz` | `gzip` |
| NDJSON | File upload, URL, Events API | `.ndjson`, `.ndjson.gz` | `gzip` |
| Parquet | File upload, URL | `.parquet`, `.parquet.gz` | `gzip` |
| Avro | Kafka | | `gzip` |
## Analyze the schema of a file (https://www.tinybird.co/docs/forward/ingest-data/local-file#analyze-the-schema-of-a-file)
Before you upload data from a file or create a data source, you can analyze the schema of the file. Tinybird infers column names, types, and JSONPaths. This is helpful to identify the most appropriate data types for your columns. See [Data types](/docs/sql-reference/data-types).
The following examples show how to analyze a local NDJSON file.
- Tinybird CLI
- Analyze API
```
tb datasource analyze local_file.ndjson
```
## Append data from a file (https://www.tinybird.co/docs/forward/ingest-data/local-file#append-data-from-a-file)
You can append data from a local or remote file to a data source in Tinybird Local or Tinybird Cloud.
Use `tb datasource append` in the CLI, `mode=append` with the Data Sources API, `append` in the TypeScript SDK, or `append` in the Python SDK.
### Append from a local file (https://www.tinybird.co/docs/forward/ingest-data/local-file#append-from-a-local-file)
- Tinybird CLI
- cURL
- TypeScript SDK
- Python SDK
```
tb --cloud datasource append local_file.csv
```
### Append from a remote file (https://www.tinybird.co/docs/forward/ingest-data/local-file#append-from-a-remote-file)
- Tinybird CLI
- cURL
- TypeScript SDK
- Python SDK
```
tb --cloud datasource append http://example_url/file.csv
```
When appending CSV files, you can improve performance by excluding the CSV Header line. However, in this case, make sure the CSV columns are ordered. If you can't guarantee the order of columns in your CSV, include the CSV header.
## Replace data from a file (https://www.tinybird.co/docs/forward/ingest-data/local-file#replace-data-from-a-file)
You can replace existing all data or a selection of data in a data source with the contents of a file. You can replace with data from local or remote files.
When using `mode=replace` with an S3 URL via the API, you must use a pre-signed URL. Unlike `mode=append`, `mode=replace` is a multi-step process that passes the URL to a background worker with no access to your S3 Connector credentials. Generate a pre-signed URL programmatically using the AWS SDK or CLI before passing it to the API.
Use `tb datasource replace` in the CLI, `mode=replace` with the Data Sources API, `replace` in the TypeScript SDK, or `replace` in the Python SDK.
### Replace from a local file (https://www.tinybird.co/docs/forward/ingest-data/local-file#replace-from-a-local-file)
- Tinybird CLI
- cURL
- TypeScript SDK
- Python SDK
```
tb --cloud datasource replace local_file.csv
```
### Replace from a remote file (https://www.tinybird.co/docs/forward/ingest-data/local-file#replace-from-a-remote-file)
- Tinybird CLI
- cURL
- TypeScript SDK
- Python SDK
```
tb --cloud datasource replace http://example_url/file.csv
```
## Replace data based on conditions (https://www.tinybird.co/docs/forward/ingest-data/local-file#replace-data-based-on-conditions)
Instead of replacing all data, you can also replace specific partitions of data. To do this, you define an SQL condition that describes the filter that's applied. All matching rows are deleted before finally ingesting the new file. Only the rows matching the condition are ingested.
Replacements are made by partition, so make sure that the condition filters on the partition key of the data source. If the source file contains rows that don't match the filter, the rows are ignored.
Conditional replace is supported in the CLI and the Data Sources API. The TypeScript SDK and Python SDK `replace` methods don't currently support replace conditions.
### Replace from a local file with a condition (https://www.tinybird.co/docs/forward/ingest-data/local-file#replace-from-a-local-file-with-a-condition)
- Tinybird CLI
- cURL
```
tb --cloud datasource replace local_file.csv --sql-condition "my_partition_key > 123"
```
### Replace from a remote file with a condition (https://www.tinybird.co/docs/forward/ingest-data/local-file#replace-from-a-remote-file-with-a-condition)
- Tinybird CLI
- cURL
```
tb --cloud datasource replace http://example_url/file.csv --sql-condition "my_partition_key > 123"
```
All the dependencies of the data source are recalculated so that your data is consistent after the replacement. If you have n-level dependencies, they're also updated by this operation.
Although replacements are atomic, Tinybird can't assure data consistency if you continue appending data to any related data source at the same time the replacement takes place. The new incoming data is discarded.
---
URL: https://www.tinybird.co/docs/forward/ingest-data/ingestion-protection
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Ingestion protection \| Tinybird"
theme-color: "#171612"
description: "How Tinybird prevents ingestion overload, ensures fair resource usage, and protects your data pipelines."
inkeep:version: "forward"
---
# Ingestion protection (https://www.tinybird.co/docs/forward/ingest-data/ingestion-protection#ingestion-protection)
Tinybird provides **ingestion flow control mechanisms** to prevent resource saturation and ensure fair usage across shared infrastructure. These mechanisms protect your ingestion pipelines from causing general service degradation or even data loss, maintaining overall system health and reliability.
When needed, these mechanisms may **temporarily delay ingestion** (reducing data freshness) or **temporarily reject ingestion requests**. In all cases, Tinybird monitors these events closely and sends you **notifications** so that you can take timely corrective action.
## Understanding the challenges of data ingestion (https://www.tinybird.co/docs/forward/ingest-data/ingestion-protection#understanding-the-challenges-of-data-ingestion)
[ClickHouse®](https://clickhouse.com/docs/en) is not optimized for high-frequency writes. Frequent inserts can saturate system resources, but many real-time use cases benefit from near-continuous data ingestion. [Asynchronous Inserts](https://clickhouse.com/docs/optimize/asynchronous-inserts) can help in some cases, but they come with important limitations, including potential issues with Materialized Views and the risk of data duplication.
Tinybird goes beyond the base capabilities of ClickHouse® to offer a **complete data ingestion solution**. You can send high-frequency data through:
- **The Events API** [Reference](events-api)
- **Connectors** [Kafka ingestion](connectors/kafka)
To make the most of these services, it’s important to:
- Understand your ingestion patterns.
- Plan your throughput and resource requirements.
- Monitor and act when ingestion begins to saturate your current [plan’s limits](/docs/forward/pricing/limits).
Use Tinybird's **High-Frequency Ingestion (HFI)** platform and **backpressure mechanisms** to preserve system stability in dedicated and shared environments.
## How frequent ingestion works (https://www.tinybird.co/docs/forward/ingest-data/ingestion-protection#how-frequent-ingestion-works)
To avoid overwhelming the ClickHouse® clusters, Tinybird gathers incoming insertions over a short period known as the **flush interval** before writing (flushing) them into the ClickHouse® tables that back your landing Data Sources.
- The **flush interval** duration depends on your workspace’s billing plan.
- A **longer flush interval** reduces cluster load but increases data freshness latency.
When ingestion flushes occur, data is written **separately for each partition** of the destination table. Therefore:
- A high number of partitions per insertion can still cause ingestion bottlenecks.
- It’s important to **define partitioning and insertion strategies carefully** to avoid excessive pressure on the ingestion system.
## When writes fail (https://www.tinybird.co/docs/forward/ingest-data/ingestion-protection#when-writes-fail)
Ingestion into a landing Data Source may fail for multiple reasons, including:
- Cluster saturation
- Network connectivity problems
- Timeout errors
- SQL execution errors
Tinybird classifies errors as either:
- **Retriable** — temporary issues that can be retried later.
- **Non-retriable** — permanent issues such as invalid data or SQL errors.
### Retriable failures (https://www.tinybird.co/docs/forward/ingest-data/ingestion-protection#retriable-failures)
Data affected by retriable errors is stored in a **staging area** and automatically retried later. During this process:
- You may experience ingestion delays.
- Tinybird’s ingestion logic avoids duplicates caused for partially inserted data for up to **5 hours**. After this 5-hour window, de-duplication is not guaranteed.
### Non-retriable failures (https://www.tinybird.co/docs/forward/ingest-data/ingestion-protection#non-retriable-failures)
Data affected by non-retriable errors is kept in the **quarantine table** associated with your Data Source. You can inspect these records and take corrective action manually.
**Be aware of failures**
- When using the **Events API**, make use of the `wait` parameter to [obtain synchronous confirmation of every write](events-api#write-acknowledgements)
- Access the [Services data sources](../monitoring/service-datasources) to detect problems
## Backpressure and flow control (https://www.tinybird.co/docs/forward/ingest-data/ingestion-protection#backpressure-and-flow-control)
Tinybird’s **backpressure mechanisms** preserve the health of your ingestion infrastructure by isolating problematic Data Sources before they affect others.
When a particular Data Source causes excessive resource consumption, for example, due to:
- Inefficient Materialized Views
- Poor partitioning configuration
- Long-running queries
— the system automatically applies backpressure in two escalating phases.
### 1. Delayed ingestion (https://www.tinybird.co/docs/forward/ingest-data/ingestion-protection#1-delayed-ingestion)
The first line of defense is **delayed ingestion**. When backpressure is activated, insertions are temporarily delayed and resumed when resource consumption returns to safe levels.
Tinybird continuously monitors:
- The number of delayed insertions
- The age of pending insertions
If these metrics increase beyond safe limits, the system escalates protection to the next phase.
#### Typical triggers for delayed ingestion: (https://www.tinybird.co/docs/forward/ingest-data/ingestion-protection#typical-triggers-for-delayed-ingestion)
- High **memory usage**
- Excessive **throughput**, exceeding cluster capacity
Throughput overload can result not only from ingestion frequency but also from:
> - Large payload sizes
> - Long insert durations, for example due to complex Materialized Views
> - High partition counts
> - Your cluster’s capacity, defined by your billing plan, which determines when these limits are reached.
Whenever possible, Tinybird applies the delay **only to the affected Data Source**, minimizing the impact on your workspace or cluster.
During delayed ingestion:
- Data is accepted, but its insertion into landing tables is **postponed**.
- You’ll see **reduced data freshness** (similar to a longer flush interval).
- If the issue is transient, ingestion recovers automatically.
You receive an **email notification** whenever delayed ingestion activates. Always review these notifications and take corrective action if necessary.
### 2. Temporary ingestion restriction (https://www.tinybird.co/docs/forward/ingest-data/ingestion-protection#2-temporary-ingestion-restriction)
If ingestion delays persist and accumulated data grows beyond safe thresholds, Tinybird temporarily **rate limits ingestion** for the affected Data Source (by applying a temporary rate limit of 1 request/message per second for one minute).
These restrictions can also apply when **CPU usage** exceeds plan limits.
When this occurs:
- **Events API** calls return HTTP **429** responses with details about the cause.
- Tinybird pauses **Connectors** temporarily and automatically attempts recovery.
You’ll receive an **email alert** immediately when ingestion is denied.
This temporary service denial prevents cascading issues that could otherwise impact system stability. In most cases, ingestion resumes automatically once the issue is resolved.
If the problem persists:
- Review the metrics and logs associated with your ingestion workload.
- Check for inefficient queries, heavy partitioning, or high system load.
- Contact **Tinybird Support** if you cannot identify the root cause.
In rare cases, ingestion protection may be triggered by issues unrelated to your ingestion, such as cloud provider disruptions. Tinybird monitoring systems detect and remediate such situations as quickly as possible.
## Best practices (https://www.tinybird.co/docs/forward/ingest-data/ingestion-protection#best-practices)
To reduce the likelihood of backpressure activation:
- Design **efficient partitioning strategies**.
- Optimize **Materialized Views** and avoid long-running transformations during ingestion.
- Batch inserts where possible rather than sending individual events. Use the appropriate ingestion mechanism for your scenario. For large, infrequent writes, you can ingest [data from files](local-file) or [from S3](connectors/s3).
- Monitor ingestion metrics regularly.
- Pay attention to **flush intervals**, **notifications**, and **system alerts**.
- Choose the right **billing plan** for your ingestion needs, and consider upgrading when ingestion volume or concurrency approaches the limits of your current plan.
## Summary (https://www.tinybird.co/docs/forward/ingest-data/ingestion-protection#summary)
Tinybird’s ingestion protection mechanisms ensure:
- **Fair usage** across users and workspaces
- **Reliability** of ingestion pipelines
- **Automatic recovery** from transient issues
- **Transparent communication** through alerts and monitoring
These mechanisms combine **delayed ingestion**, **automatic retries**, and **temporary rate limiting** to keep your ingestion stable, efficient, and predictable.
**See also:**
- [Events API Reference](events-api)
- [Kafka Connector Guide](connectors/kafka)
- [Quarantine Data Sources](../core-concepts/data-sources#quarantine-data-sources)
---
URL: https://www.tinybird.co/docs/forward/ingest-data/events-api
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Send events \| Tinybird"
theme-color: "#171612"
description: "Send JSON and NDJSON data to Tinybird by calling the Events API."
inkeep:version: "forward"
---
# Send events (https://www.tinybird.co/docs/forward/ingest-data/events-api#send-events)
You send events to a [data source](/docs/forward/core-concepts/data-sources) using the Events API or the [tb datasource append](/docs/forward/dev-reference/commands/tb-datasource#tb-datasource-append) CLI command with the `--events` flag. [Ingestion limits](/docs/forward/pricing/limits#ingestion-limits) apply.
## Send JSON events (https://www.tinybird.co/docs/forward/ingest-data/events-api#send-json-events)
You can send individual events, but microbatching several events usually gives you much higher throughput. Use NDJSON with cURL or the CLI, `ingestBatch` in the TypeScript SDK, or `ingest_batch` in the Python SDK.
- cURL
- Tinybird CLI
- TypeScript SDK
- Python SDK
Send NDJSON events
```
curl \
-H "Authorization: Bearer " \
-d $'{"date": "2020-04-05", "city": "Chicago"}\n{"date": "2020-04-05", "city": "Madrid"}\n' \
'https://api.tinybird.co/v0/events?name='
```
NDJSON means each JSON event is separated by a newline `\n` character. The SDK batch methods accept arrays of JSON objects and handle the Events API request for you.
## Token (https://www.tinybird.co/docs/forward/ingest-data/events-api#token)
The token you use to send events must have the `DATASOURCE:APPEND` scope. This should be defined in your data source file using `TOKEN {token_name} append`. For more details, see [resource-scoped tokens](/docs/forward/core-concepts/static-tokens#resource-scoped-tokens).
## Limits (https://www.tinybird.co/docs/forward/ingest-data/events-api#limits)
The Events API delivers a default capacity of:
- Up to 100 requests per second per data source
- Up to 10 MB per request per Data Source on Free plans and up to 100 MB on Developer, SaaS, and Enterprise plans
If you reach this limit, you receive an `HTTP 413 - Request Entity Too Large` error. Split larger requests into multiple smaller ones to ensure smooth data ingestion.
### Rate limit headers (https://www.tinybird.co/docs/forward/ingest-data/events-api#rate-limit-headers)
The Events API returns the following headers with the response:
| Header Name | Description |
| --- | --- |
| `X-RateLimit-Limit` | The maximum number of requests you're permitted to make in the current limit window. |
| `X-RateLimit-Remaining` | The number of requests remaining in the current rate limit window. |
| `X-RateLimit-Reset` | The time in seconds after the current rate limit window resets. |
| `Retry-After` | The time to wait before making a another request. Only present on 429 responses. |
### Check the payload size (https://www.tinybird.co/docs/forward/ingest-data/events-api#check-the-payload-size)
To avoid hitting the request limit size, you can check your payload size before sending. For example:
- shell
- python
- js
```
echo '{"date": "2020-04-05", "city": "Chicago"}' | wc -c | awk '{print $1/1024/1024 " MB"}'
```
## Compress the data you send (https://www.tinybird.co/docs/forward/ingest-data/events-api#compress-the-data-you-send)
You can compress the data you send to the Events API using Gzip. Compressing events adds overhead to the ingestion process, which can introduce latency, although it's typically minimal.
Here is an example of sending a JSON event compressed with Gzip from the command line:
```
echo '{"timestamp":"2022-10-27T11:43:02.099Z","transaction_id":"8d1e1533-6071-4b10-9cda-b8429c1c7a67","name":"Bobby Drake","email":"bobby.drake@pressure.io","age":42,"passport_number":3847665,"flight_from":"Barcelona","flight_to":"London","extra_bags":1,"flight_class":"economy","priority_boarding":false,"meal_choice":"vegetarian","seat_number":"15D","airline":"Red Balloon"}' | gzip > body.gz
curl \
-X POST 'https:///v0/events?name=gzip_events_example' \
-H "Authorization: Bearer " \
-H "Content-Encoding: gzip" \
--data-binary @body.gz
```
## Write operation acknowledgements (https://www.tinybird.co/docs/forward/ingest-data/events-api#write-acknowledgements)
When you send data to the Events API, you usually receive an `HTTP202` response, which indicates that the request was successful, although it doesn't confirm that the data has been committed into the underlying database. This is useful when guarantees on writes aren't strictly necessary.
Typically, it takes under two seconds to receive a response from the Events API. For example:
```
curl \
-X POST 'https:///v0/events?name=events_example' \
-H "Authorization: Bearer " \
-d $'{"timestamp":"2022-10-27T11:43:02.099Z"}'
< HTTP/2 202
< content-type: application/json
< content-length: 42
<
{"successful_rows":2,"quarantined_rows":0}
```
If your use case requires absolute guarantees that data is committed, use the `wait` parameter. The `wait` parameter is a boolean that accepts a value of `true` or `false`. A value of `false` is the default behavior, equivalent to omitting the parameter entirely.
Using `wait=true` with your request asks the Events API to wait for acknowledgement that the data you sent has been committed into the underlying database. You then receive an `HTTP200` response that confirms data has been committed.
Adding `wait=true` to your request can result in a slower response time. Use a timeout of at least 10 seconds when waiting for the response. For example:
```
curl \
-X POST 'https:///v0/events?name=events_example&wait=true' \
-H "Authorization: Bearer " \
-d $'{"timestamp":"2022-10-27T11:43:02.099Z"}'
< HTTP/2 200
< content-type: application/json
< content-length: 42
<
{"successful_rows":2,"quarantined_rows":0}
```
Log your requests and responses from and to the Events API. This helps you get visibility into any failures.
---
URL: https://www.tinybird.co/docs/forward/development-workflow/using-cloud-branches
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Using Cloud Branches \| Tinybird"
theme-color: "#171612"
description: "Use Tinybird Cloud Branches in your development workflow."
inkeep:version: "forward"
---
# Using Cloud Branches (https://www.tinybird.co/docs/forward/development-workflow/using-cloud-branches#using-cloud-branches)
Use Cloud Branches to build and test changes in Tinybird Cloud before deploying them to your main Workspace. The workflow is the same one used in the quickstarts: configure the project, build in a branch, test with branch data, then deploy.
Use Cloud Branches when you need production-like data, connector validation, shared preview URLs, or pull request review. Use [Tinybird Local](/docs/forward/development-workflow/local-development) when fixture data is enough and you want a fully local loop.
For the concept model, see [Branches](/docs/forward/core-concepts/branches). Branches are not promoted to production. You validate changes in a branch, then deploy the project to your main Tinybird Cloud Workspace with `tb deploy`, an SDK deploy command, or CI/CD.
## Check your project config (https://www.tinybird.co/docs/forward/development-workflow/using-cloud-branches#check-your-project-config)
Your project config defines which files Tinybird builds and which development environment commands use by default.
Use `include` to point at your datafiles or SDK resource definitions. Set `dev_mode` or `devMode` to `branch` when you want `build` to target Cloud Branches by default. Set `branch_data_mode` to control the production data attached when `build` auto-creates a Cloud Branch. You can still override the target with `--branch` or `--local` on the command line.
- Tinybird CLI
- TypeScript SDK
- Python SDK
tinybird.config.json
```
{
"dev_mode": "branch",
"branch_data_mode": "last_partition",
"include": [
"tinybird"
]
}
```
## Typical branch workflow (https://www.tinybird.co/docs/forward/development-workflow/using-cloud-branches#typical-branch-workflow)
1. Set the development environment to Cloud Branches in `tinybird.config.json`.
2. Create a Git branch.
3. Build the project in a matching Tinybird Cloud Branch.
4. Edit Data Sources, Pipes, Endpoints, Tokens, or Connections.
5. Rebuild and test the branch with fixture data, production-like data, or connector data.
6. Run a deployment check against the main Cloud Workspace.
7. Merge and deploy.
```
git checkout -b add-plan-to-events
tb build
```
With `dev_mode` or `devMode` set to `branch`, `tb build` uses your Git branch name as the Tinybird Cloud Branch name. If the Cloud Branch does not exist, Tinybird creates it using `branch_data_mode` from `tinybird.config.json`. By default, that attaches the latest production partition. Run `tb build` again after editing project files.
For commands other than `build`, pass the branch name explicitly with `--branch=`.
```
Using dev_mode=branch. Running build against Tinybird branch 'add-plan-to-events'.
Running against Tinybird Cloud: Workspace production | branch add-plan-to-events
Creating branch 'add-plan-to-events' with data_mode=last_partition...
» Git branch: add-plan-to-events
» Tinybird Cloud branch: add-plan-to-events ✓ created
» Building project...
✓ datasources/events.datasource created
✓ Build completed in 6.2s
```
## Build in a branch (https://www.tinybird.co/docs/forward/development-workflow/using-cloud-branches#build-in-a-branch)
Run a build before testing changes. If your project is in Git, Tinybird creates or reuses a Cloud Branch with the same name as your Git branch.
- Tinybird CLI
- TypeScript SDK
- Python SDK
```
git checkout -b add-plan-to-events
tb build
```
When you edit project files, run build again to apply the changes to the Cloud Branch.
## Bring production-like data into a branch (https://www.tinybird.co/docs/forward/development-workflow/using-cloud-branches#bring-production-like-data-into-a-branch)
When `dev_mode=branch`, `tb build` auto-creates Cloud Branches with the latest production partition by default. Configure this behavior in `tinybird.config.json` with `branch_data_mode`:
tinybird.config.json
```
{
"dev_mode": "branch",
"branch_data_mode": "last_partition"
}
```
This attaches the latest production partition to the branch without copying the underlying data. Writes, deletes, and schema changes stay isolated in the branch.
Use production-like branch data for:
- Schema changes that need real values.
- Endpoint changes where fixture data is not enough.
- Query performance checks on realistic partition sizes.
- Review apps that need production-like responses.
Do not use branch data as a backup or long-lived replica. Branches are short-lived validation environments.
When you create a branch manually with `tb branch create`, pass `--last-partition` to attach production data. Branches created without that flag copy Workspace resources, not data:
```
tb branch create add-plan-to-events --last-partition
tb --branch=add-plan-to-events build
```
## Query and call branch resources (https://www.tinybird.co/docs/forward/development-workflow/using-cloud-branches#query-and-call-branch-resources)
Run SQL against the branch:
```
tb --branch=add-plan-to-events sql "SELECT plan, count() FROM events GROUP BY plan"
```
Open Tinybird UI for the branch:
```
tb --branch=add-plan-to-events open
```
The URL is `https://cloud.tinybird.co///~`.
Call a branch Endpoint with a branch Token:
```
curl -H "Authorization: Bearer " \
"https://api.tinybird.co/v0/pipes/events_by_plan.json"
```
## Test connector data (https://www.tinybird.co/docs/forward/development-workflow/using-cloud-branches#test-connector-data)
Use `--with-connections` when your branch needs Kafka, S3, or GCS connector Data Sources.
```
tb build --with-connections
```
### Kafka (https://www.tinybird.co/docs/forward/development-workflow/using-cloud-branches#kafka)
Kafka ingestion starts stopped in branches. Start it only when you are ready to test:
```
tb --branch=add-plan-to-events datasource start my_kafka_datasource
```
Stop it when you finish:
```
tb --branch=add-plan-to-events datasource stop my_kafka_datasource
```
Each start creates a new consumer group for the branch and starts from the latest offset. This avoids collisions with production and other branches.
### S3 and GCS (https://www.tinybird.co/docs/forward/development-workflow/using-cloud-branches#s3-and-gcs)
Import a sample instead of syncing the whole bucket:
```
tb --branch=add-plan-to-events datasource sample my_s3_datasource --wait
```
Sample imports are separate jobs and do not affect production sync state.
## Validate before deploy (https://www.tinybird.co/docs/forward/development-workflow/using-cloud-branches#validate-before-deploy)
Before merging, run the checks that match the change. For schema or connector changes, include a Cloud deployment check.
```
tb build
tb test run
tb --cloud deploy --check
```
For SDK projects, use the matching SDK commands:
- TypeScript SDK
- Python SDK
```
npx tinybird build
npx tinybird deploy --check
```
## Preview branches in CI (https://www.tinybird.co/docs/forward/development-workflow/using-cloud-branches#preview-branches-in-ci)
Use preview branches for pull requests. They let reviewers and preview deployments call the changed Tinybird project before it is merged.
For TypeScript SDK projects:
```
npx tinybird preview --check
```
For Python SDK projects:
```
uv run tinybird preview --check
```
For datafile projects, create and build a branch explicitly:
```
tb branch create tmp_ci_my_feature --last-partition
tb --branch=tmp_ci_my_feature build
```
See [CI/CD](/docs/forward/development-workflow/cicd#preview-deployments).
## Work without Git (https://www.tinybird.co/docs/forward/development-workflow/using-cloud-branches#work-without-git)
If your project is not in a Git repository, pass the branch name explicitly to `build` too:
```
tb --branch=add-plan-to-events build
```
## Clean up branches (https://www.tinybird.co/docs/forward/development-workflow/using-cloud-branches#clean-up-branches)
Branches are meant to be temporary. Delete them after the pull request is merged or closed.
- Tinybird CLI
- TypeScript SDK
- Python SDK
```
tb branch rm preview_1
```
## When to use Local instead (https://www.tinybird.co/docs/forward/development-workflow/using-cloud-branches#when-to-use-local-instead)
Use [Tinybird Local](/docs/forward/core-concepts/tinybird-local) instead of Cloud Branches when fixture data is enough, you need a fully local loop, or you want CI checks that don't modify Cloud resources.
---
URL: https://www.tinybird.co/docs/forward/development-workflow/manual-deployment
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Manual deployment \| Tinybird"
theme-color: "#171612"
description: "Deploy your Tinybird project to Tinybird Cloud using the CLI."
inkeep:version: "forward"
---
# Manual deployment (https://www.tinybird.co/docs/forward/development-workflow/manual-deployment#manual-deployment)
You can deploy your data projects to Tinybird Cloud directly from the command line using the [Tinybird CLI](/docs/forward/dev-reference/commands).
If your project was initialized with `tb init --dev-mode local` or `tb init --dev-mode branch`, running `tb deploy` without explicit environment flags targets your cloud main workspace automatically.
## What is a deployment? (https://www.tinybird.co/docs/forward/development-workflow/manual-deployment#what-is-a-deployment)
Deployments are versions of your project resources and data running on local or cloud infrastructure.
There are two types of deployments:
- **Staging deployments**: Deployments you can use to validate your changes. You access them using the `--staging` flag.
- **Live deployments**: Deployments that make your changes available to your users.
Each type can be deployed to Tinybird Local (`--local`) or Tinybird Cloud (`--cloud`).
### Deployment status (https://www.tinybird.co/docs/forward/development-workflow/manual-deployment#deployment-status)
Deployments have the following statuses:
- `In progress`: The deployment is in progress. Use `--wait` to wait for it to finish.
- `Live`: The deployment is active and has been promoted from staging.
- `Staging`: The deployment is active in staging. Use `--staging` to access it.
- `Failed`: The deployment failed. Try `tb deploy --check` to debug the issue.
- `Deleted`: The deployment was deleted as a result of creating new deployments.
## Deploy from the CLI (https://www.tinybird.co/docs/forward/development-workflow/manual-deployment#deploy-from-the-cli)
1
### Check the deployment (https://www.tinybird.co/docs/forward/development-workflow/manual-deployment#check-the-deployment)
Before creating the deployment, you can check the deployment with the `--check` flag. This runs a series of checks to ensure the deployment is ready. This is similar to a dry run.
```
tb --cloud deployment create --check
```
The `--check` flag validates external connections to S3, Kafka, GCS, and databases referenced via table functions. For local success, set connection secrets with `tb secret set` and use `tb local start --use-aws-creds` for S3 connections.
2
### Create a staging deployment (https://www.tinybird.co/docs/forward/development-workflow/manual-deployment#create-a-staging-deployment)
Create a new staging deployment in Tinybird Cloud. Pass the `--wait` flag to wait for the deployment to finish:
```
tb --cloud deployment create --wait
```
To run commands against the staging deployment, use the `--staging` flag. For example:
```
tb --staging --cloud endpoint ls
```
3
### Promote to live (https://www.tinybird.co/docs/forward/development-workflow/manual-deployment#promote-to-live)
When the staging deployment is ready, promote it to a live deployment in Tinybird Cloud:
```
tb --cloud deployment promote
```
To deploy and promote in one step, use `tb deploy`. If you prefer explicit flags, run `tb --cloud deploy`.
## Staging deployments (https://www.tinybird.co/docs/forward/development-workflow/manual-deployment#staging-deployments)
You can write data to, and read data from, a staging deployment before promoting it to live. This is useful when you've made schema changes that might be incompatible with the current live deployment, like adding new columns.
Automatic data source changes that Tinybird applies with `ALTER`, such as adding a column or changing a data source TTL, are applied only when the deployment is promoted to live. Those changes aren't available in staging deployments.
### Writing to staging deployments (https://www.tinybird.co/docs/forward/development-workflow/manual-deployment#writing-to-staging-deployments)
You can use the [Events API](/docs/forward/ingest-data/events-api) to write directly to staging deployments through the `__tb__min_deployment` parameter, which indicates the target deployment ID. For example:
```
curl \
-H "Authorization: Bearer " \
-d '{"date": "2020-04-05 00:05:38", "city": "Chicago", "new_column": "value"}' \
'https:///v0/events?name=events_test&__tb__min_deployment=5'
```
In the example, if the ID of your current live deployment is 4 and you're creating a deployment with an ID of 5, Tinybird ingests the data into staging deployment 5 only. Follow these steps:
1. Make schema changes in a staging deployment.
2. Ingest data compatible with the new schema.
3. Validate the changes work as expected.
4. Promote the deployment to live when ready.
Without the parameter, data would be rejected if it doesn't match the schema of the current live deployment.
To get the deployment ID, run `tb deployment ls`.
### Reading from staging deployments (https://www.tinybird.co/docs/forward/development-workflow/manual-deployment#reading-from-staging-deployments)
You can query data from a staging deployment using [pipe endpoints](/docs/forward/core-concepts/api-endpoints). To access a staging endpoint, add the `__tb__deployment` parameter to your API request:
```
curl \
-H "Authorization: Bearer " \
'https:///v0/pipes/my_endpoint?__tb__deployment=5'
```
Follow these steps:
1. Test your endpoints with the new schema changes.
2. Validate query results before promoting to live.
3. Ensure your app works correctly with the updated data structure.
To get the deployment ID, run `tb deployment ls`.
### Continuous operation (https://www.tinybird.co/docs/forward/development-workflow/manual-deployment#continuous-operation)
Once the deployment is promoted to live, you can continue using the same API calls. The `__tb__min_deployment` or `__tb__deployment` parameters ensure compatibility both before and after promotion.
For more details on the Events API parameters, see the [Events API Reference](/docs/api-reference/events-api).
## On-demand compute for deployment populates (https://www.tinybird.co/docs/forward/development-workflow/manual-deployment#on-demand-compute-for-deployment-populates)
When a deployment needs to populate Data Sources, such as when creating Materialized Views or evolving schemas, Tinybird can run the populate on dedicated on-demand compute instead of your main workspace infrastructure. This keeps the populate isolated from production query and ingestion workloads.
Tinybird automatically uses on-demand compute for large deployment populates. For pricing, regional rates, and example calculations, see [On-demand CPUs](/docs/forward/pricing#on-demand-cpus).
---
URL: https://www.tinybird.co/docs/forward/development-workflow/local-development
Last update: 2026-05-29T14:27:44.000Z
Content:
---
title: "Local development \| Tinybird"
theme-color: "#171612"
description: "Develop your Tinybird project locally. Make changes and test them before deploying to Tinybird Cloud."
inkeep:version: "forward"
---
# Local development (https://www.tinybird.co/docs/forward/development-workflow/local-development#local-development)
Use Tinybird Local to build and test your project on your machine before deploying to Tinybird Cloud. The workflow is the same one used in the quickstarts: configure the project, build, test the Endpoint, then deploy.
Use Tinybird Local when fixture data is enough and you want a fast loop on your machine. Use [Cloud Branches](/docs/forward/development-workflow/using-cloud-branches) when you need production-like data, connector validation, shared preview URLs, or pull request review.
For installation, Docker settings, local services, API URLs, persistence, and local Tokens, see [Tinybird Local](/docs/forward/core-concepts/tinybird-local).
## Check your project config (https://www.tinybird.co/docs/forward/development-workflow/local-development#check-your-project-config)
Your project config defines which files Tinybird builds and which development environment commands use by default.
Use `include` to point at your datafiles or SDK resource definitions. Set `dev_mode` or `devMode` to `local` when you want `build` to target Tinybird Local by default. You can still override the target with `--local` or `--branch` on the command line.
- Tinybird CLI
- TypeScript SDK
- Python SDK
tinybird.config.json
```
{
"dev_mode": "local",
"include": [
"tinybird"
]
}
```
## Typical local workflow (https://www.tinybird.co/docs/forward/development-workflow/local-development#typical-local-workflow)
1. Set the development environment to Local in `tinybird.config.json`.
2. Start Tinybird Local.
3. Check out a git branch.
4. Build the project with fixture data.
5. Edit Data Sources, Pipes, Endpoints, Tokens, or Connections.
6. Rebuild and query local resources or call local Endpoints.
7. Run a deployment check against Tinybird Cloud.
8. Merge and deploy ideally via CI/CD.
## Start Tinybird Local (https://www.tinybird.co/docs/forward/development-workflow/local-development#start-tinybird-local)
Start Tinybird Local before you build or test local resources:
```
tb local start
```
The CLI logs the status of the container and services. Do not build the project until you see the `Tinybird Local is ready!` message.
```
» Starting Tinybird Local...
* Tinybird Local container status: healthy
» Checking services...
✓ Tinybird Local container
✓ Clickhouse
✓ Redis
✓ Server
✓ Events
✓ Tinybird Local authentication
✓ Tinybird Local is ready!
```
Run Tinybird Local in the background with `--daemon`:
```
tb local start --daemon
```
If Tinybird Local becomes unhealthy, restart it:
```
tb local restart
```
## Build the project (https://www.tinybird.co/docs/forward/development-workflow/local-development#build-the-project)
Run a build before testing changes. It validates resource definitions, resolves dependencies, and loads matching fixture files.
- Tinybird CLI
- TypeScript SDK
- Python SDK
```
tb build
```
If your config does not default to Local, pass the flag explicitly:
```
tb --local build
```
For SDK projects, use the matching SDK command, such as `npx tinybird build --local`.
```
» Building project...
✓ datasources/user_actions.datasource created
✓ endpoints/user_actions_line_chart.pipe created
✓ Build completed in 0.2s
```
In the `tb local start` logs, you can see whether the request is successful.
```
[API] POST /v1/build 200 67.1ms
```
When you edit project files, run build again to apply the changes to Tinybird Local.
If you want automatic rebuilds while editing local files, run `tb dev`. The examples use explicit `tb build` because it is the same command you run in CI and branch workflows.
## Test local resources (https://www.tinybird.co/docs/forward/development-workflow/local-development#test-local-resources)
Call local Endpoints with the local API host and a local Token:
```
curl -H "Authorization: Bearer " \
"http://localhost:7181/v0/pipes/user_actions_line_chart.json"
```
You can also run SQL against Tinybird Local:
```
tb --local sql "SELECT count() FROM user_actions"
```
For test files and fixtures, see [Test your project](/docs/forward/guides/test-your-project).
## Open Tinybird UI (https://www.tinybird.co/docs/forward/development-workflow/local-development#open-tinybird-ui)
You can also use Tinybird UI to inspect your project. Run the following command to open the Tinybird UI pointing to your local environment.
```
tb open
```
The URL is `https://cloud.tinybird.co///~local~`
## Deploy after validation (https://www.tinybird.co/docs/forward/development-workflow/local-development#deploy-after-validation)
When the local checks pass, validate the deployment against Tinybird Cloud before merging or releasing:
```
tb --cloud deploy --check
```
Then deploy manually or let CI/CD deploy from the main branch:
```
tb --cloud deploy
```
See [Manual deployment](/docs/forward/development-workflow/manual-deployment) and [CI/CD](/docs/forward/development-workflow/cicd).
---
URL: https://www.tinybird.co/docs/forward/development-workflow/examples
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Development workflow examples \| Tinybird"
theme-color: "#171612"
description: "Examples for iterating on Tinybird projects with the Tinybird CLI, the TypeScript SDK, the Python SDK, branches, CI, and deployments."
inkeep:version: "forward"
---
# Development workflow examples (https://www.tinybird.co/docs/forward/development-workflow/examples#development-workflow-examples)
These examples follow the order you usually use in a project: build first, test locally or in a branch, validate in CI, then deploy.
Use the command set that matches your project format.
| Task | Tinybird CLI | TypeScript SDK | Python SDK |
| --- | --- | --- | --- |
| Build or validate | `tb build` | `npx tinybird build` | `uv run tinybird build` |
| Preview a pull request | `git checkout -b ... && tb build` | `npx tinybird preview --check` | `uv run tinybird preview --check` |
| Check production deployment | `tb --cloud deploy --check` | `npx tinybird deploy --check` | `uv run tinybird deploy --check` |
| Deploy to production | `tb --cloud deploy` | `npx tinybird deploy` | `uv run tinybird deploy` |
## Local workflow (https://www.tinybird.co/docs/forward/development-workflow/examples#local-workflow)
Use Local for fast checks with fixture data. See [Tinybird Local](/docs/forward/core-concepts/tinybird-local) for installation, Docker settings, persistence, local APIs, and local Tokens. Use a [Cloud Branch](/docs/forward/development-workflow/using-cloud-branches) when you need production-like data or connector behavior.
### Build the project (https://www.tinybird.co/docs/forward/development-workflow/examples#build-the-project)
Run a build before testing changes. It catches invalid resource definitions and dependency issues.
- Tinybird CLI
- TypeScript SDK
- Python SDK
```
tb build
```
For Tinybird CLI projects, `tb build` loads matching fixtures automatically. For example, `fixtures/events.ndjson` is loaded into `datasources/events.datasource`. See [Test your project](/docs/forward/guides/test-your-project#fixture-files).
### Change an Endpoint and rebuild (https://www.tinybird.co/docs/forward/development-workflow/examples#change-an-endpoint-and-rebuild)
Edit the resource, then rebuild the project.
- Tinybird CLI
- TypeScript SDK
- Python SDK
```
tb local start --daemon
tb build
```
datasources/events.datasource
```
SCHEMA >
`timestamp` DateTime `json:$.timestamp`,
`user_id` String `json:$.user_id`,
`event` LowCardinality(String) `json:$.event`,
`plan` Nullable(String) `json:$.plan`
ENGINE "MergeTree"
ENGINE_SORTING_KEY "timestamp"
```
endpoints/events_by_plan.pipe
```
NODE endpoint
SQL >
SELECT
plan,
count() AS events
FROM events
WHERE timestamp >= now() - INTERVAL 7 DAY
GROUP BY plan
ORDER BY events DESC
TYPE ENDPOINT
```
Call the local Endpoint after the build finishes:
```
curl -H "Authorization: Bearer " \
"http://localhost:7181/v0/pipes/events_by_plan.json"
```
## Branch workflow (https://www.tinybird.co/docs/forward/development-workflow/examples#branch-workflow)
Branches are short-lived Cloud environments. Use them to test changes without changing production.
### Develop in a Cloud Branch (https://www.tinybird.co/docs/forward/development-workflow/examples#develop-in-a-cloud-branch)
For everyday branch development, create a Git branch and build. Tinybird creates or reuses a Cloud Branch with the same name as the Git branch.
```
git checkout -b add-plan-to-events
tb build
```
Rebuild after editing resources:
- Tinybird CLI
- TypeScript SDK
- Python SDK
```
tb build
```
Query the branch after the build finishes:
- Tinybird CLI
- TypeScript SDK
- Python SDK
```
tb --branch=add-plan-to-events sql "SELECT plan, count() FROM events GROUP BY plan"
```
When you need production-like data, `tb build` attaches the latest production partition automatically when `dev_mode=branch`. You can also create the branch explicitly with `--last-partition` before building:
```
tb branch create add-plan-to-events --last-partition
tb --branch=add-plan-to-events build
```
For connector Data Sources, create the branch with `--with-connections` and start or sample the connector when you are ready to test. See [Cloud Branches](/docs/forward/development-workflow/using-cloud-branches#test-connector-data).
## Pull requests and CI (https://www.tinybird.co/docs/forward/development-workflow/examples#pull-requests-and-ci)
Run the same checks in CI that you run locally. For pull requests, create a preview branch so reviewers and preview apps can query the changed Tinybird project.
### Preview a pull request (https://www.tinybird.co/docs/forward/development-workflow/examples#preview-a-pull-request)
- Tinybird CLI
- TypeScript SDK
- Python SDK
```
tb branch create tmp_ci_my_feature --last-partition
tb --branch=tmp_ci_my_feature build
```
Use `preview` in GitHub Actions, GitLab CI, Vercel, or another CI system to create an ephemeral branch for each pull request. See [Preview deployments](/docs/forward/development-workflow/cicd#preview-deployments).
### Validate before deploying (https://www.tinybird.co/docs/forward/development-workflow/examples#validate-before-deploying)
Use deployment checks for changes that affect Data Source schemas, Materialized Views, engine settings, or connector configuration.
- Tinybird CLI
- TypeScript SDK
- Python SDK
```
tb build
tb test run
tb --cloud deploy --check
```
## Data Source evolution (https://www.tinybird.co/docs/forward/development-workflow/examples#data-source-evolution)
For the migration rules behind Data Source changes, read [Evolve Data Sources](/docs/forward/guides/evolve-data-source). The following examples show the most common cases.
### Rename a column with FORWARD_QUERY (https://www.tinybird.co/docs/forward/development-workflow/examples#rename-a-column-with-forward-query)
Column renames need an explicit mapping. Tinybird sees a removed column and a new column. Without a mapping, the new column gets a default value.
Update the Data Source definition and run a deployment check before merging. With the Tinybird CLI, add `FORWARD_QUERY` to the `.datasource` file. In SDK projects, update the schema in code and review the generated deployment plan. If the plan requires a forward query, use the expression from the Data Source evolution guide as the source of truth.
- Tinybird CLI
- TypeScript SDK
- Python SDK
datasources/events.datasource
```
SCHEMA >
`timestamp` DateTime `json:$.timestamp`,
`account_id` String `json:$.account_id`,
`event` LowCardinality(String) `json:$.event`
ENGINE "MergeTree"
ENGINE_SORTING_KEY "timestamp"
FORWARD_QUERY >
SELECT timestamp, user_id AS account_id, event
```
The `FORWARD_QUERY` runs against the live Data Source during deployment. It must include the `SELECT` list only. Do not include `FROM` or `WHERE`.
### Add a nullable column with ALTER (https://www.tinybird.co/docs/forward/development-workflow/examples#add-a-nullable-column-with-alter)
Adding a nullable column is usually an automatic `ALTER` during deployment. The change is applied when the deployment is promoted to live, not while it is in staging. See [Automatic ALTER operations](/docs/forward/guides/evolve-data-source#automatic-alter-operations).
- Tinybird CLI
- TypeScript SDK
- Python SDK
datasources/events.datasource
```
SCHEMA >
`timestamp` DateTime `json:$.timestamp`,
`user_id` String `json:$.user_id`,
`event` LowCardinality(String) `json:$.event`,
`plan` Nullable(String) `json:$.plan`
ENGINE "MergeTree"
ENGINE_SORTING_KEY "timestamp"
```
Run a deployment check before merging the change:
- Tinybird CLI
- TypeScript SDK
- Python SDK
```
tb --cloud deploy --check
```
### Change a type with safe conversion (https://www.tinybird.co/docs/forward/development-workflow/examples#change-a-type-with-safe-conversion)
If existing values might not cast cleanly, make the conversion explicit. This example changes `session_id` from `String` to `UUID` and uses a default UUID when the old value is invalid.
- Tinybird CLI
- TypeScript SDK
- Python SDK
datasources/sessions.datasource
```
SCHEMA >
`timestamp` DateTime `json:$.timestamp`,
`session_id` UUID `json:$.session_id`,
`user_id` String `json:$.user_id`
ENGINE "MergeTree"
ENGINE_SORTING_KEY "timestamp"
FORWARD_QUERY >
SELECT
timestamp,
accurateCastOrDefault(session_id, 'UUID') AS session_id,
user_id
```
After the deployment is live and you no longer need the migration expression, remove the `FORWARD_QUERY` in a follow-up change. Leaving an old `FORWARD_QUERY` can overwrite values in a later backfill.
## Deployments (https://www.tinybird.co/docs/forward/development-workflow/examples#deployments)
After the checks pass, deploy the project or create a staging deployment for manual promotion.
### Deploy to production (https://www.tinybird.co/docs/forward/development-workflow/examples#deploy-to-production)
- Tinybird CLI
- TypeScript SDK
- Python SDK
```
tb --cloud deploy
```
### Promote a staging deployment (https://www.tinybird.co/docs/forward/development-workflow/examples#promote-a-staging-deployment)
If you prefer explicit staging and promotion, create a staging deployment, inspect it, then promote it:
```
tb --cloud deployment create --wait
tb --staging --cloud endpoint ls
tb --cloud deployment promote
```
For regular production changes, put the same checks in CI/CD. See [CI/CD](/docs/forward/development-workflow/cicd).
---
URL: https://www.tinybird.co/docs/forward/development-workflow/cicd
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "CI/CD \| Tinybird"
theme-color: "#171612"
description: "Deploy your Tinybird project through CI/CD workflows."
inkeep:version: "forward"
---
# Deploying to Tinybird through CI/CD (https://www.tinybird.co/docs/forward/development-workflow/cicd#deploying-to-tinybird-through-cicd)
After you create your data project in Git, you can implement continuous integration (CI) and continuous deployment (CD) workflows to automate interaction with Tinybird.
When you initialize a project using `tb init`, Tinybird can generate CI/CD templates that you can use in GitHub and GitLab to automate testing and deployment.
The Tinybird Local container is a key part of the CI workflow. See [Tinybird Local](/docs/forward/core-concepts/tinybird-local) for more information.
## CI workflow (https://www.tinybird.co/docs/forward/development-workflow/cicd#ci-workflow)
As you expand and iterate on your data projects, you can continuously validate your changes. In the same way that you write integration and acceptance tests for source code in a software project, you can write automated tests for your API endpoints to run on each pull or merge request.
A potential CI workflow could run the following steps when you open a pull request:
1. Install Tinybird CLI and Tinybird Local: Sets up dependencies and installs the Tinybird CLI to run the required commands.
2. Build project: Checks the syntax and correctness of your datafiles.
3. Test project: Runs fixture tests, data quality tests, or both to validate changes.
4. Deployment check: Validates the deployment before creating it, similar to a dry run.
The following templates are available for GitHub and GitLab:
- GitHub
- GitLab
```
name: Tinybird - CI Workflow
on:
workflow_dispatch:
pull_request:
branches:
- main
- master
types: [opened, reopened, labeled, unlabeled, synchronize]
concurrency: ${{ github.workflow }}-${{ github.event.pull_request.number }}
env:
TINYBIRD_HOST: ${{ secrets.TINYBIRD_HOST }}
TINYBIRD_TOKEN: ${{ secrets.TINYBIRD_TOKEN }}
jobs:
ci:
runs-on: ubuntu-latest
defaults:
run:
working-directory: '.'
services:
tinybird:
image: tinybirdco/tinybird-local:latest
ports:
- 7181:7181
steps:
- uses: actions/checkout@v4
- name: Install Tinybird CLI
run: curl https://tinybird.co | sh
- name: Build project
run: tb --local build
- name: Test project
run: tb --local test run
- name: Deployment check
run: tb --cloud --host ${{ env.TINYBIRD_HOST }} --token ${{ env.TINYBIRD_TOKEN }} deploy --check
```
## Preview deployments (https://www.tinybird.co/docs/forward/development-workflow/cicd#preview-deployments)
You can create an ephemeral [branch](/docs/forward/core-concepts/branches) for each pull request so you can test your changes with production data before deploying. There are two ways to set up preview deployments: using the TypeScript SDK or the Tinybird CLI.
### Using the TypeScript SDK (https://www.tinybird.co/docs/forward/development-workflow/cicd#using-the-typescript-sdk)
Requires `@tinybirdco/sdk` version `0.0.53` or later.
The `tinybird preview` command creates a branch named `tmp_ci_`, builds your resources, and deploys them. If a branch with the same name already exists, it is deleted and recreated.
The TypeScript SDK auto-detects CI environments (Vercel, GitHub Actions, GitLab CI, CircleCI, Azure Pipelines, Bitbucket Pipelines) and resolves the correct branch token at runtime. The host is inferred from the token, so no `TINYBIRD_HOST` or additional configuration is needed.
- GitHub
- GitLab
```
name: Tinybird - Preview
on:
pull_request:
branches:
- main
- master
types: [opened, reopened, synchronize]
concurrency: ${{ github.workflow }}-${{ github.event.pull_request.number }}
jobs:
preview:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: npx tinybird preview
env:
TINYBIRD_TOKEN: ${{ secrets.TINYBIRD_TOKEN }}
```
### Using the Tinybird CLI (https://www.tinybird.co/docs/forward/development-workflow/cicd#using-the-tinybird-cli)
You can also create preview branches using the `tb` CLI. This approach gives you more control over branch creation and supports connectors.
- GitHub
- GitLab
```
name: Tinybird - Preview
on:
pull_request:
branches:
- main
- master
types: [opened, reopened, synchronize]
concurrency: ${{ github.workflow }}-${{ github.event.pull_request.number }}
env:
TINYBIRD_HOST: ${{ secrets.TINYBIRD_HOST }}
TINYBIRD_TOKEN: ${{ secrets.TINYBIRD_TOKEN }}
jobs:
preview:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install Tinybird CLI
run: curl https://tinybird.co | sh
- name: Create branch
run: tb --host ${{ env.TINYBIRD_HOST }} --token ${{ env.TINYBIRD_TOKEN }} branch create tmp_ci_${{ github.head_ref }} --last-partition
- name: Build and deploy
run: tb --host ${{ env.TINYBIRD_HOST }} --token ${{ env.TINYBIRD_TOKEN }} --branch=tmp_ci_${{ github.head_ref }} build
```
### Preview with connectors (https://www.tinybird.co/docs/forward/development-workflow/cicd#preview-with-connectors)
When your project uses Kafka, S3, or GCS connectors, preview branches can include them. The `tinybird preview` command doesn't create preview branches that ingest data from connectors. To test with connectors, create the branch manually with the `tb` CLI. See [Test with connector data](/docs/forward/core-concepts/branches#test-with-connector-data).
```
tb branch create tmp_ci_my_feature --last-partition --with-connections
```
Kafka connections are stopped by default in preview branches. Start them when you're ready to test:
```
tb --branch=tmp_ci_my_feature datasource start my_kafka_datasource
tb --branch=tmp_ci_my_feature datasource stop my_kafka_datasource
```
For S3/GCS, import sample data:
```
tb --branch=tmp_ci_my_feature datasource sample my_s3_datasource --wait
```
### Cleanup (https://www.tinybird.co/docs/forward/development-workflow/cicd#cleanup)
Preview branches are ephemeral. Delete them when the pull request is closed or merged. You can automate this with a CI workflow:
- GitHub
- GitLab
```
name: Tinybird - Cleanup Preview
on:
pull_request:
branches:
- main
- master
types: [closed]
concurrency: ${{ github.workflow }}-${{ github.event.pull_request.number }}
jobs:
cleanup:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: npx tinybird branch delete tmp_ci_${{ github.head_ref }}
env:
TINYBIRD_TOKEN: ${{ secrets.TINYBIRD_TOKEN }}
```
Or delete branches manually using the TypeScript SDK or the Tinybird CLI:
```
tinybird branch delete tmp_ci_my_feature
tb branch rm tmp_ci_my_feature
```
### Frontend preview integration (https://www.tinybird.co/docs/forward/development-workflow/cicd#frontend-preview-integration)
The TypeScript SDK runtime client (`@tinybirdco/sdk` version `0.0.53` or later) detects preview environments automatically. When your frontend is deployed to a preview environment, such as a Vercel Preview Deployment, it automatically queries the Tinybird preview branch that matches the frontend's git branch, with no code changes or extra environment variables required.
```
import { Tinybird } from '@tinybirdco/sdk'
const tb = new Tinybird({ token: process.env.TINYBIRD_TOKEN })
// In preview environments (e.g., a Vercel Preview Deployment), the SDK
// automatically resolves the matching branch token.
// No additional configuration is needed.
const result = await tb.query('SELECT * FROM my_endpoint')
```
Auto-detection works with Vercel, GitHub Actions, GitLab CI, CircleCI, Azure Pipelines, and Bitbucket Pipelines.
## CD workflow (https://www.tinybird.co/docs/forward/development-workflow/cicd#cd-workflow)
Once your changes are validated by the CI pipeline, you can automate the deployment process and let Tinybird handle the migration for you with no downtime.
A potential CD workflow could run the following steps when you merge a pull request:
1. Install Tinybird CLI: Sets up dependencies and installs the Tinybird CLI to run the required commands.
2. Deploy project: Creates a staging deployment in Tinybird Cloud, migrates data, promotes to live, and removes previous deployment.
The following templates are available for GitHub and GitLab:
- GitHub
- GitLab
```
name: Tinybird - CD Workflow
on:
push:
branches:
- main
- master
concurrency: ${{ github.workflow }}-${{ github.event.ref }}
env:
TINYBIRD_HOST: ${{ secrets.TINYBIRD_HOST }}
TINYBIRD_TOKEN: ${{ secrets.TINYBIRD_TOKEN }}
jobs:
cd:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install Tinybird CLI
run: curl https://tinybird.co | sh
- name: Deploy project
run: tb --cloud --host ${{ env.TINYBIRD_HOST }} --token ${{ env.TINYBIRD_TOKEN }} deploy
```
## Secrets (https://www.tinybird.co/docs/forward/development-workflow/cicd#secrets)
Make sure to provide the values for the following secrets in your CI/CD settings:
- `TINYBIRD_HOST`
- `TINYBIRD_TOKEN`
Run `tb info` to get the value for the secret `TINYBIRD_HOST`. It is the `api` URL in Tinybird Cloud. For example:
```
tb info
» Tinybird Cloud:
--------------------------------------------------------------------------------------------
user: tinybird@domain.co
workspace_name: forward
workspace_id: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXX
token: YOUR-ADMIN-TOKEN
user_token: YOUR-USER-TOKEN
api: https://api.tinybird.co
ui: https://cloud.tinybird.co/gcp/europe-west2/forward
--------------------------------------------------------------------------------------------
» Tinybird Local:
--------------------------------------------------------------------------------------------
user: tinybird@domain.co
workspace_name: forward
workspace_id: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXX
token: YOUR-LOCAL-ADMIN-TOKEN
user_token: YOUR-LOCAL-USER-TOKEN
api: http://localhost:7181
ui: http://cloud.tinybird.co/local/7181/forward
--------------------------------------------------------------------------------------------
» Project:
---------------------------------------------------
current: /path/to/your/project
.tinyb: /path/to/your/project/.tinyb
project: /path/to/your/project
---------------------------------------------------
```
And run `tb --cloud token copy "admin "` to get the value of `TINYBIRD_TOKEN`.
```
tb --cloud token copy "admin your_user@email.com"
Running against Tinybird Cloud: Workspace forward
** Token 'admin your_user@email.com' copied to clipboard
```
When running `tb test run`, Tinybird creates a fresh Workspace for each test run. Secrets don't persist between test runs. To avoid test failures, add a default value to your secrets. For example:
```
GCS_SERVICE_ACCOUNT_CREDENTIALS_JSON {{ tb_secret("secret_name", "default_value") }}
```
`tb_secret` replacements happen at parser time in the server. If a secret changes after a deployment, Tinybird doesn't detect it automatically and requires a new deployment.
---
URL: https://www.tinybird.co/docs/forward/development-workflow/agent-skills
Last update: 2026-07-29T12:18:04.000Z
Content:
---
title: "Tinybird Agent Skills \| Tinybird"
theme-color: "#171612"
description: "Help coding agents build, test, and deploy Tinybird projects."
inkeep:version: "forward"
---
# Tinybird Agent Skills (https://www.tinybird.co/docs/forward/development-workflow/agent-skills#tinybird-agent-skills)
Tinybird Agent Skills give compatible AI coding agents instructions for working with Tinybird projects. They provide context for creating and reviewing project files, SQL, tests, and deployment workflows. The agent uses the skills through the tools available in your coding environment, such as the Tinybird CLI or an SDK.
The skills follow the [Agent Skills format](https://agentskills.io/) and work with compatible coding agents such as Amp, Claude Code, Codex, Cursor, Gemini CLI, and OpenCode. Their source is available in the [Tinybird Agent Skills GitHub repository](https://github.com/tinybirdco/tinybird-agent-skills).
## Install Tinybird Agent Skills (https://www.tinybird.co/docs/forward/development-workflow/agent-skills#install-tinybird-agent-skills)
Run the following command in your project:
Install Tinybird Agent Skills
```
npx skills add tinybirdco/tinybird-agent-skills
```
Update installed skills to their latest versions:
Update installed Agent Skills
```
npx skills update
```
Once installed, a compatible coding agent can select a skill when it detects a relevant task. You can also ask the agent which skills are available.
## Available skills (https://www.tinybird.co/docs/forward/development-workflow/agent-skills#available-skills)
### `tinybird-best-practices` (https://www.tinybird.co/docs/forward/development-workflow/agent-skills#tinybird-best-practices)
Use this skill for general Tinybird project work. It covers project structure, Data Sources, Pipes, Endpoints, SQL and query optimization, deployments, testing, and secrets management.
### `tinybird-cli-guidelines` (https://www.tinybird.co/docs/forward/development-workflow/agent-skills#tinybird-cli-guidelines)
Use this skill when working with datafiles or running `tb` commands. It covers Tinybird Local, builds, deployments, data operations, Tokens, secrets, mock data, and tests.
### `tinybird-typescript-sdk-guidelines` (https://www.tinybird.co/docs/forward/development-workflow/agent-skills#tinybird-typescript-sdk-guidelines)
Use this skill with `@tinybirdco/sdk` and TypeScript projects. It covers typed Data Sources and Pipes, Tinybird clients, ingestion and queries, SDK CLI workflows, and migration from datafiles.
### `tinybird-python-sdk-guidelines` (https://www.tinybird.co/docs/forward/development-workflow/agent-skills#tinybird-python-sdk-guidelines)
Use this skill with `tinybird-sdk` and Python projects. It covers Data Sources, Pipes, Endpoints, Connections, clients, ingestion, queries, deployment commands, and migration from datafiles.
## Example prompts (https://www.tinybird.co/docs/forward/development-workflow/agent-skills#example-prompts)
After installing the skills, give your coding agent a concrete project task:
- "Create a Data Source for user events and add fixture data for local testing."
- "Review this Pipe and optimize its Endpoint for low latency."
- "Add tests for the Endpoints and run them with the Tinybird CLI."
- "Define a Kafka Connection and Materialized View with the Python SDK."
- "Validate this project and prepare it for deployment to Tinybird Cloud."
The coding agent applies the relevant skill guidance, then uses the tools and credentials available in your project to perform the task.
## Agent Skills and MCP (https://www.tinybird.co/docs/forward/development-workflow/agent-skills#agent-skills-and-mcp)
Agent Skills provide packaged instructions and Tinybird-specific context inside a coding workflow. Use them when an agent needs to create or review project files, write SQL, add tests, or work through build and deployment commands.
The [Tinybird MCP server](/docs/forward/query-data/mcp) connects an agent to a live Workspace. The configured Token limits its access, and the agent can inspect or query data and call published Endpoints. Use MCP when the task depends on live Workspace resources or data.
You can combine both. Agent Skills guide how the coding agent changes the project, while MCP lets the agent inspect the live Workspace or verify results through its permitted queries and Endpoints.
---
URL: https://www.tinybird.co/docs/forward/dev-reference/typescript-sdk-resources
Last update: 2026-06-18T10:43:45.000Z
Content:
---
title: "TypeScript SDK resources \| Tinybird"
theme-color: "#171612"
description: "Define Tinybird resources in TypeScript with the Tinybird SDK."
inkeep:version: "forward"
---
# TypeScript SDK resources (https://www.tinybird.co/docs/forward/dev-reference/typescript-sdk-resources#typescript-sdk-resources)
Use the TypeScript SDK to define Tinybird resources as code with full type inference. You can define datasources, pipes, endpoints, materialized views, copy pipes, sink pipes, and connections in TypeScript and sync them to Tinybird.
## Project layout (https://www.tinybird.co/docs/forward/dev-reference/typescript-sdk-resources#project-layout)
The SDK scaffolds the following files by default:
- `lib/tinybird.ts`
- `tinybird.config.mjs`
## Configure credentials (https://www.tinybird.co/docs/forward/dev-reference/typescript-sdk-resources#configure-credentials)
Create a `.env.local` file in your project root with the Tinybird token and host:
```
TINYBIRD_TOKEN=p.your_token_here
TINYBIRD_URL=https://api.tinybird.co
```
## Tokens (https://www.tinybird.co/docs/forward/dev-reference/typescript-sdk-resources#tokens)
Use `defineToken` to declare reusable token names, then attach them to datasources or pipes with each resource's `tokens` option.
```
import {
defineDatasource,
defineEndpoint,
defineToken,
engine,
node,
t,
} from "@tinybirdco/sdk";
export const eventsAppend = defineToken("events_append");
export const eventsRead = defineToken("events_read");
export const events = defineDatasource("events", {
schema: {
timestamp: t.dateTime(),
payload: t.string(),
},
engine: engine.mergeTree({
sortingKey: ["timestamp"],
}),
tokens: [
{ token: eventsAppend, scope: "APPEND" },
{ token: eventsRead, scope: "READ" },
],
});
export const eventsApi = defineEndpoint("events_api", {
nodes: [
node({
name: "latest",
sql: `SELECT timestamp, payload FROM events ORDER BY timestamp DESC LIMIT 100`,
}),
],
output: {
timestamp: t.dateTime(),
payload: t.string(),
},
tokens: [{ token: eventsRead, scope: "READ" }],
});
```
Token names must start with a letter or underscore and then contain only alphanumeric characters or underscores. Use `READ` or `APPEND` for datasource token bindings, and `READ` for pipe or endpoint token bindings.
## Datasources (https://www.tinybird.co/docs/forward/dev-reference/typescript-sdk-resources#datasources)
Define datasources with schemas and engines:
```
import { defineDatasource, engine, t, type InferRow } from "@tinybirdco/sdk";
export const pageViews = defineDatasource("page_views", {
description: "Page view tracking data",
schema: {
timestamp: t.dateTime(),
pathname: t.string(),
session_id: t.string(),
country: t.string().lowCardinality().nullable(),
},
engine: engine.mergeTree({
sortingKey: ["pathname", "timestamp"],
}),
});
export type PageViewsRow = InferRow;
```
## Endpoints (https://www.tinybird.co/docs/forward/dev-reference/typescript-sdk-resources#endpoints)
Define API endpoints:
```
import {
defineEndpoint,
node,
p,
t,
type InferParams,
type InferOutputRow,
} from "@tinybirdco/sdk";
export const topPages = defineEndpoint("top_pages", {
description: "Get the most visited pages",
params: {
start_date: p.dateTime(),
end_date: p.dateTime(),
limit: p.int32().optional(10),
},
nodes: [
node({
name: "aggregated",
sql: `
SELECT pathname, count() AS views
FROM page_views
WHERE timestamp >= {{DateTime(start_date)}}
AND timestamp <= {{DateTime(end_date)}}
GROUP BY pathname
ORDER BY views DESC
LIMIT {{Int32(limit, 10)}}
`,
}),
],
output: {
pathname: t.string(),
views: t.uint64(),
},
});
export type TopPagesParams = InferParams;
export type TopPagesOutput = InferOutputRow;
```
## Pipes (https://www.tinybird.co/docs/forward/dev-reference/typescript-sdk-resources#pipes)
Define internal pipes that are not exposed as API endpoints:
```
import { definePipe, node, p } from "@tinybirdco/sdk";
export const filteredEvents = definePipe("filtered_events", {
description: "Filter events by date range",
params: {
start_date: p.dateTime(),
end_date: p.dateTime(),
},
nodes: [
node({
name: "filtered",
sql: `
SELECT * FROM events
WHERE timestamp >= {{DateTime(start_date)}}
AND timestamp <= {{DateTime(end_date)}}
`,
}),
],
});
```
## Materialized views (https://www.tinybird.co/docs/forward/dev-reference/typescript-sdk-resources#materialized-views)
Use materialized views to populate derived datasources:
```
import {
defineDatasource,
defineMaterializedView,
node,
t,
engine,
} from "@tinybirdco/sdk";
export const dailyStats = defineDatasource("daily_stats", {
schema: {
date: t.date(),
pathname: t.string(),
views: t.simpleAggregateFunction("sum", t.uint64()),
},
engine: engine.aggregatingMergeTree({
sortingKey: ["date", "pathname"],
}),
jsonPaths: false,
});
export const dailyStatsMv = defineMaterializedView("daily_stats_mv", {
datasource: dailyStats,
nodes: [
node({
name: "aggregate",
sql: `
SELECT toDate(timestamp) AS date, pathname, count() AS views
FROM page_views
GROUP BY date, pathname
`,
}),
],
});
```
## Copy pipes (https://www.tinybird.co/docs/forward/dev-reference/typescript-sdk-resources#copy-pipes)
Use copy pipes for snapshots or scheduled exports:
```
import { defineCopyPipe, node } from "@tinybirdco/sdk";
export const dailySnapshot = defineCopyPipe("daily_snapshot", {
datasource: dailyStats,
schedule: "0 0 * * *",
mode: "append",
nodes: [
node({
name: "snapshot",
sql: `SELECT * FROM daily_stats WHERE date = today() - 1`,
}),
],
});
```
## Sink pipes (https://www.tinybird.co/docs/forward/dev-reference/typescript-sdk-resources#sink-pipes)
Use sink pipes to publish query results to Kafka or S3:
```
import { defineSinkPipe, node } from "@tinybirdco/sdk";
export const kafkaEventsSink = defineSinkPipe("kafka_events_sink", {
sink: {
connection: eventsKafka,
topic: "events_export",
schedule: "@on-demand",
},
nodes: [
node({
name: "publish",
sql: `SELECT timestamp, payload FROM kafka_events`,
}),
],
});
export const s3EventsSink = defineSinkPipe("s3_events_sink", {
sink: {
connection: landingS3,
bucketUri: "s3://my-bucket/exports/",
fileTemplate: "events_{date}",
format: "csv",
schedule: "@once",
},
nodes: [
node({
name: "export",
sql: `SELECT timestamp, session_id FROM s3_landing`,
}),
],
});
```
## Connections (https://www.tinybird.co/docs/forward/dev-reference/typescript-sdk-resources#connections)
Define connector connections and reference them in datasources:
```
import {
defineGCSConnection,
defineKafkaConnection,
defineDynamoDBConnection,
defineDatasource,
engine,
secret,
t,
} from "@tinybirdco/sdk";
export const eventsKafka = defineKafkaConnection("events_kafka", {
bootstrapServers: "kafka.example.com:9092",
securityProtocol: "SASL_SSL",
saslMechanism: "PLAIN",
key: secret("KAFKA_KEY"),
secret: secret("KAFKA_SECRET"),
});
export const landingGCS = defineGCSConnection("landing_gcs", {
serviceAccountCredentialsJson: secret("GCS_SERVICE_ACCOUNT_CREDENTIALS_JSON"),
});
export const kafkaEvents = defineDatasource("kafka_events", {
schema: {
timestamp: t.dateTime(),
payload: t.string(),
},
engine: engine.mergeTree({
sortingKey: ["timestamp"],
}),
kafka: {
connection: eventsKafka,
topic: "events",
groupId: "events-consumer",
},
});
export const gcsLanding = defineDatasource("gcs_landing", {
schema: {
timestamp: t.dateTime(),
session_id: t.string(),
},
engine: engine.mergeTree({
sortingKey: ["timestamp"],
}),
gcs: {
connection: landingGCS,
bucketUri: "gs://my-gcs-bucket/events/*.csv",
schedule: "@auto",
},
});
```
## DynamoDB (https://www.tinybird.co/docs/forward/dev-reference/typescript-sdk-resources#dynamodb)
The DynamoDB connector mirrors an Amazon DynamoDB table into Tinybird: an initial backfill via a Point-in-Time Recovery export to S3, followed by continuous Change Data Capture from DynamoDB Streams. See the [DynamoDB connector](/docs/forward/ingest-data/connectors/dynamodb) page for the table requirements and IAM setup.
Defining the connection in code isn't enough. Tinybird assumes an IAM role in your own AWS account, and the role's trust policy must include a Workspace-specific external ID that only Tinybird generates. Run `tb connection create dynamodb` to get the external ID, the `Principal` account, and the policy JSON, then create the role in AWS and store its ARN as the `DYNAMODB_ROLE_ARN` secret (set it from the dashboard or with `tb secret set`). Without the correct external ID, `deploy` fails with a `403 ... external ID` error. See the [DynamoDB connector](/docs/forward/ingest-data/connectors/dynamodb) page for the full setup.
Define the connection with `defineDynamoDBConnection`. It takes a `region` and an `arn` (the IAM role Tinybird assumes to read your table, its stream, and the S3 export bucket); both are required:
```
import { defineDynamoDBConnection, secret } from "@tinybirdco/sdk";
export const eventsDynamo = defineDynamoDBConnection("events_dynamo", {
region: "us-east-1",
arn: secret("DYNAMODB_ROLE_ARN"),
});
```
The `arn` is read from a workspace secret. The SDK has no command for creating secrets, so set `DYNAMODB_ROLE_ARN` from the Tinybird dashboard or with `tb secret set DYNAMODB_ROLE_ARN ` before you deploy. See the [DynamoDB connector](/docs/forward/ingest-data/connectors/dynamodb) page for the IAM role and trust policy the ARN must point to.
Each row represents a *change* to the table, so map the change record's fields with `jsonPath()` and use a `replacingMergeTree` engine with `ver` and `isDeleted` to query the current state. Attach the connection through the `dynamodb` key with the table ARN and the name of an S3 `exportBucket` you own (the bucket name only, with no `s3://` prefix):
```
import { defineDatasource, t, engine } from "@tinybirdco/sdk";
import { eventsDynamo } from "./connections";
export const dynamoEvents = defineDatasource("dynamo_events", {
schema: {
id: t.string().jsonPath("$.Id"),
_record: t.string().jsonPath("$.NewImage"),
_timestamp: t.dateTime64(3).jsonPath("$.ApproximateCreationDateTime"),
_event_name: t.string().lowCardinality().jsonPath("$.eventName"),
_is_deleted: t.uint8().jsonPath("$._is_deleted"),
},
engine: engine.replacingMergeTree({
sortingKey: ["id"],
ver: "_timestamp",
isDeleted: "_is_deleted",
}),
dynamodb: {
connection: eventsDynamo,
tableArn: "arn:aws:dynamodb:us-east-1:123456789012:table/events",
exportBucket: "my-export-bucket",
},
});
```
## Next steps (https://www.tinybird.co/docs/forward/dev-reference/typescript-sdk-resources#next-steps)
- Review the SDK CLI: [TypeScript SDK CLI commands](/docs/forward/dev-reference/commands/typescript-sdk-cli)
- Follow the SDK quickstart: [Quick start: TypeScript SDK](/docs/forward/quickstarts/typescript-sdk)
---
URL: https://www.tinybird.co/docs/forward/dev-reference/template-functions
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Template functions \| Tinybird"
theme-color: "#171612"
description: "Template functions available in Tinybird datafiles."
inkeep:version: "forward"
---
# Template functions (https://www.tinybird.co/docs/forward/dev-reference/template-functions#template-functions)
The following template functions are available. You can use them in [datafiles](/docs/forward/dev-reference/datafiles) to accomplish different tasks. For the conceptual model of parameters, settings, secrets, and control flow, see [Templating language](/docs/forward/core-concepts/templating-language).
## defined (https://www.tinybird.co/docs/forward/dev-reference/template-functions#defined)
Checks whether a variable is defined.
defined function
```
%
SELECT
date
FROM my_table
{% if defined(param) %}
WHERE ...
{% end %}
```
## column (https://www.tinybird.co/docs/forward/dev-reference/template-functions#column)
Retrieves the column by its name from a variable.
column function
```
%
{% set var_1 = 'name' %}
SELECT
{{column(var_1)}}
FROM my_table
```
## columns (https://www.tinybird.co/docs/forward/dev-reference/template-functions#columns)
Retrieves columns by their name from a variable.
columns function
```
%
{% set var_1 = 'name,age,address' %}
SELECT
{{columns(var_1)}}
FROM my_table
```
## date_diff_in_seconds (https://www.tinybird.co/docs/forward/dev-reference/template-functions#date-diff-in-seconds)
Returns the absolute value of the difference in seconds between two `DateTime`. See [DateTime](/docs/sql-reference/data-types/datetime).
The function accepts the following parameters:
- `date_1`: the first date or DateTime.
- `date_2`: the second date or DateTime.
- `date_format`: (optional) the format of the dates. Defaults to `'%Y-%m-%d %H:%M:%S'`, so you can pass `DateTime` as `YYYY-MM-DD hh:mm:ss` when calling the function.
- `backup_date_format`: (optional) the format of the dates if the first format doesn't match. Use it when your default input format is a DateTime (`2022-12-19 18:42:22`) but you receive a date instead (`2022-12-19`).
- `none_if_error`: (optional) whether to return `None` if the dates don't match the provided formats. Defaults to `False`. Use it to provide an alternate logic in case any of the dates are specified in a different format.
An example of how to use the function:
```
date_diff_in_seconds('2022-12-19T18:42:23.521Z', '2022-12-19T18:42:23.531Z', date_format='%Y-%m-%dT%H:%M:%S.%fz')
```
The following example shows how to use the function in a datafile:
date_diff_in_seconds function
```
%
SELECT
date, events
{% if date_diff_in_seconds(date_end, date_start, date_format="%Y-%m-%dT%H:%M:%Sz") < 3600 %}
FROM my_table_raw
{% else %}
FROM my_table_hourly_agg
{% end %}
WHERE date BETWEEN
parseDateTimeBestEffort({{String(date_start,'2023-01-11T12:24:04Z')}})
AND
parseDateTimeBestEffort({{String(date_end,'2023-01-11T12:24:05Z')}})
```
## date_diff_in_minutes (https://www.tinybird.co/docs/forward/dev-reference/template-functions#date-diff-in-minutes)
Same behavior as [date_diff_in_seconds](https://www.tinybird.co/docs/forward/dev-reference/template-functions#date_diff_in_seconds), but returns the difference in minutes.
## date_diff_in_hours (https://www.tinybird.co/docs/forward/dev-reference/template-functions#date-diff-in-hours)
Same behavior as [date_diff_in_seconds](https://www.tinybird.co/docs/forward/dev-reference/template-functions#date_diff_in_seconds), but returns the difference in hours.
## date_diff_in_days (https://www.tinybird.co/docs/forward/dev-reference/template-functions#date-diff-in-days)
Returns the absolute value of the difference in days between two dates or DateTime.
date_diff_in_days function
```
%
SELECT
date
FROM my_table
{% if date_diff_in_days(date_end, date_start) < 7 %}
WHERE ...
{% end %}
```
`date_format` is optional and defaults to `'%Y-%m-%d`, so you can pass DateTime as `YYYY-MM-DD` when calling the function.
As with `date_diff_in_seconds`, `date_diff_in_minutes`, and `date_diff_in_hours`, other [date_formats](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes) are supported.
## max_threads (https://www.tinybird.co/docs/forward/dev-reference/template-functions#max-threads)
Sets the maximum number of threads a query can use.
This is useful for local development and Copy Pipes when the local ClickHouse® instance has limited threads available.
max_threads function
```
%
{{max_threads(1)}}
SELECT *
FROM events
```
See [Run Copy Pipes locally](/docs/forward/core-concepts/copy-pipes#run-copy-pipes-in-tinybird-local) for a Copy Pipe example.
## split_to_array (https://www.tinybird.co/docs/forward/dev-reference/template-functions#split-to-array)
Splits comma separated values into an array. The function accepts the following parameters:
`split_to_array(arr, default, separator=',')`
- `arr`: the value to split.
- `default`: the default value to use if `arr` is empty.
- `separator`: the separator to use. Defaults to `,`.
The following example splits `code` into an array of integers:
split_to_array function
```
%
SELECT
arrayJoin(arrayMap(x -> toInt32(x), {{split_to_array(code, '')}})) as codes
FROM my_table
```
The following example splits `param` into an array of strings using `|` as the custom separator:
split_to_array with a custom separator function
```
%
SELECT
{{split_to_array(String(param, 'hi, how are you|fine thanks'), separator='|')}}
```
## enumerate_with_last (https://www.tinybird.co/docs/forward/dev-reference/template-functions#enumerate-with-last)
Creates an iterable array, returning a boolean value that allows to check if the current element is the last element in the array. You can use it alongside the [split_to_array function](https://www.tinybird.co/docs/forward/dev-reference/template-functions#split_to_array).
## symbol (https://www.tinybird.co/docs/forward/dev-reference/template-functions#symbol)
Retrieves the value of a variable. The function accepts the following parameters:
`symbol(x, quote)`
For example:
enumerate_with_last function
```
%
SELECT
{% for _last, _x in enumerate_with_last(split_to_array(attr, 'amount')) %}
sum({{symbol(_x)}}) as {{symbol(_x)}}
{% if not _last %}, {% end %}
{% end %}
FROM my_table
```
## sql_and (https://www.tinybird.co/docs/forward/dev-reference/template-functions#sql-and)
Creates a list of "WHERE" clauses, along with "AND" separated filters, that checks if a field () is or isn't () in a list/tuple ().
The function accepts the following parameters:
`sql_and(__= [, ...] )`
- ``: any column in the table.
- ``: one of: `in`, `not_in`, `gt` (>), `lt` (<), `gte` (>=), `lte` (<=)
- ``: any of the transform type functions (`Array(param, 'Int8')`, `String(param)`, etc.). If one parameter isn't specified, then the filter is ignored.
For example:
sql_and function
```
%
SELECT *
FROM my_table
WHERE 1
{% if defined(param) or defined(param2_not_in) %}
AND {{sql_and(
param__in=Array(param, 'Int32', defined=False),
param2__not_in=Array(param2_not_in, 'String', defined=False))}}
{% end %}
```
If this is queried with `param=1,2` and `param2_not_in=ab,bc,cd`, it translates to:
sql_and function - generated sql
```
SELECT *
FROM my_table
WHERE 1
AND param IN [1,2]
AND param2 NOT IN ['ab','bc','cd']
```
If this is queried with `param=1,2` only, but `param2_not_in` isn't specified, it translates to:
sql_and function - generated sql param missing
```
SELECT *
FROM my_table
WHERE 1
AND param IN [1,2]
```
## Transform types functions (https://www.tinybird.co/docs/forward/dev-reference/template-functions#transform-types-functions)
The following functions validate the type of a template variable and cast it to the desired data type. They also provide a default value if no value is passed.
- `Boolean(x)`
- `DateTime64(x)`
- `DateTime(x)`
- `Date(x)`
- `Float32(x)`
- `Float64(x)`
- `Int8(x)`
- `Int16(x)`
- `Int32(x)`
- `Int64(x)`
- `Int128(x)`
- `Int256(x)`
- `UInt8(x)`
- `UInt16(x)`
- `UInt32(x)`
- `UInt64(x)`
- `UInt128(x)`
- `UInt256(x)`
- `String(x)`
- `Array(x)`
Each function accepts the following parameters:
`type(x, default, description=, required=)`
- `x`: the parameter or value.
- `default`: (optional) the default value to use if `x` is empty.
- `description`: (optional) the description of the value.
- `required`: (optional) whether the value is required.
For example, `Int32` in the following query, `lim` is the parameter to be cast to an `Int32`, `10` is the default value, and so on:
transform_type_functions
```
%
SELECT * FROM TR LIMIT {{Int32(lim, 10, description="Limit the number of rows in the response", required=False)}}
```
---
URL: https://www.tinybird.co/docs/forward/dev-reference/python-sdk-resources
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Python SDK resources \| Tinybird"
theme-color: "#171612"
description: "Define Tinybird resources in Python with the Tinybird SDK."
inkeep:version: "forward"
---
# Python SDK resources (https://www.tinybird.co/docs/forward/dev-reference/python-sdk-resources#python-sdk-resources)
Use the Python SDK to define Tinybird resources as code. You can define Data Sources, Pipes, Endpoints, Materialized Views, Copy Pipes, Sink Pipes, and Connections in Python and sync them to Tinybird.
Supported Python versions for Python SDK workflows are 3.11, 3.12, and 3.13. Python 3.14 is not supported yet because these workflows rely on the Tinybird CLI.
## Project layout (https://www.tinybird.co/docs/forward/dev-reference/python-sdk-resources#project-layout)
A Python SDK project usually includes:
- `tinybird.config.json`
- `tb_project/resources.py`
- `tb_project/client.py`
`tinybird.config.json` points `include` at the folder or files where you define resources.
## Configure credentials (https://www.tinybird.co/docs/forward/dev-reference/python-sdk-resources#configure-credentials)
Create a `.env.local` file in your project root:
```
TINYBIRD_TOKEN=p.your_token_here
TINYBIRD_URL=https://api.tinybird.co
```
## Tokens (https://www.tinybird.co/docs/forward/dev-reference/python-sdk-resources#tokens)
Use `define_token` to declare reusable token names, then attach them to Data Sources or Pipes through each resource's `tokens` option.
```
from tinybird_sdk import define_datasource, define_endpoint, define_token, engine, node, t
events_append = define_token("events_append")
events_read = define_token("events_read")
events = define_datasource("events", {
"schema": {
"timestamp": t.date_time(),
"payload": t.string(),
},
"engine": engine.merge_tree({"sorting_key": ["timestamp"]}),
"tokens": [
{"token": events_append, "scope": "APPEND"},
{"token": events_read, "scope": "READ"},
],
})
events_api = define_endpoint("events_api", {
"nodes": [
node({
"name": "latest",
"sql": "SELECT timestamp, payload FROM events ORDER BY timestamp DESC LIMIT 100",
}),
],
"output": {
"timestamp": t.date_time(),
"payload": t.string(),
},
"tokens": [
{"token": events_read, "scope": "READ"},
],
})
```
Token names must start with a letter or underscore and then use only alphanumeric characters or underscores. In Data Source token bindings, use `READ` or `APPEND` scopes. In Pipe or Endpoint token bindings, use `READ`.
## Datasources (https://www.tinybird.co/docs/forward/dev-reference/python-sdk-resources#datasources)
Define Data Sources with `define_datasource`:
```
from tinybird_sdk import define_datasource, engine, t
page_views = define_datasource("page_views", {
"description": "Page view tracking data",
"schema": {
"timestamp": t.date_time(),
"pathname": t.string(),
"session_id": t.string(),
"country": t.string().low_cardinality().nullable(),
},
"engine": engine.merge_tree({
"sorting_key": ["pathname", "timestamp"],
}),
})
```
### Column JSON paths (https://www.tinybird.co/docs/forward/dev-reference/python-sdk-resources#column-json-paths)
Use `column()` when you need custom JSONPath mappings:
```
from tinybird_sdk import column, define_datasource, engine, t
events = define_datasource("events", {
"schema": {
"event_id": column(t.string(), {"json_path": "$.id"}),
"timestamp": column(t.date_time(), {"json_path": "$.ts"}),
},
"engine": engine.merge_tree({"sorting_key": ["event_id"]}),
})
```
## Endpoints (https://www.tinybird.co/docs/forward/dev-reference/python-sdk-resources#endpoints)
Define API endpoints with `define_endpoint`:
```
from tinybird_sdk import define_endpoint, node, p, t
top_pages = define_endpoint("top_pages", {
"description": "Get the most visited pages",
"params": {
"start_date": p.date_time(),
"end_date": p.date_time(),
"limit": p.int32().optional(10),
},
"nodes": [
node({
"name": "aggregated",
"sql": """
SELECT pathname, count() AS views
FROM page_views
WHERE timestamp >= {{DateTime(start_date)}}
AND timestamp <= {{DateTime(end_date)}}
GROUP BY pathname
ORDER BY views DESC
LIMIT {{Int32(limit, 10)}}
""",
}),
],
"output": {
"pathname": t.string(),
"views": t.uint64(),
},
})
```
## Pipes (https://www.tinybird.co/docs/forward/dev-reference/python-sdk-resources#pipes)
Define internal pipes with `define_pipe`:
```
from tinybird_sdk import define_pipe, node, p
filtered_events = define_pipe("filtered_events", {
"params": {
"start_date": p.date_time(),
"end_date": p.date_time(),
},
"nodes": [
node({
"name": "filtered",
"sql": """
SELECT *
FROM events
WHERE timestamp >= {{DateTime(start_date)}}
AND timestamp <= {{DateTime(end_date)}}
""",
}),
],
})
```
## Materialized views (https://www.tinybird.co/docs/forward/dev-reference/python-sdk-resources#materialized-views)
Use `define_materialized_view` to populate derived Data Sources:
```
from tinybird_sdk import define_datasource, define_materialized_view, engine, node, t
daily_stats = define_datasource("daily_stats", {
"schema": {
"date": t.date(),
"pathname": t.string(),
"views": t.simple_aggregate_function("sum", t.uint64()),
},
"engine": engine.aggregating_merge_tree({
"sorting_key": ["date", "pathname"],
}),
})
daily_stats_mv = define_materialized_view("daily_stats_mv", {
"datasource": daily_stats,
"nodes": [
node({
"name": "aggregate",
"sql": """
SELECT toDate(timestamp) AS date, pathname, count() AS views
FROM page_views
GROUP BY date, pathname
""",
}),
],
})
```
## Copy pipes (https://www.tinybird.co/docs/forward/dev-reference/python-sdk-resources#copy-pipes)
Use `define_copy_pipe` for scheduled snapshots:
```
from tinybird_sdk import define_copy_pipe, node
daily_snapshot = define_copy_pipe("daily_snapshot", {
"datasource": daily_stats,
"copy_schedule": "0 0 * * *",
"copy_mode": "append",
"nodes": [
node({
"name": "snapshot",
"sql": "SELECT * FROM daily_stats WHERE date = today() - 1",
}),
],
})
```
## Sink pipes (https://www.tinybird.co/docs/forward/dev-reference/python-sdk-resources#sink-pipes)
Use `define_sink_pipe` to publish query results to Kafka or S3:
```
from tinybird_sdk import define_sink_pipe, node
kafka_events_sink = define_sink_pipe("kafka_events_sink", {
"sink": {
"connection": events_kafka,
"topic": "events_export",
"schedule": "@on-demand",
},
"nodes": [
node({
"name": "publish",
"sql": "SELECT timestamp, payload FROM kafka_events",
}),
],
})
s3_events_sink = define_sink_pipe("s3_events_sink", {
"sink": {
"connection": landing_s3,
"bucket_uri": "s3://my-bucket/exports/",
"file_template": "events_{date}",
"format": "csv",
"schedule": "@once",
},
"nodes": [
node({
"name": "export",
"sql": "SELECT timestamp, session_id FROM events",
}),
],
})
```
## Connections (https://www.tinybird.co/docs/forward/dev-reference/python-sdk-resources#connections)
Define connector connections with `define_kafka_connection`, `define_s3_connection`, `define_gcs_connection`, and `define_dynamodb_connection`:
```
from tinybird_sdk import (
define_dynamodb_connection,
define_gcs_connection,
define_kafka_connection,
define_s3_connection,
secret,
)
events_kafka = define_kafka_connection("events_kafka", {
"bootstrap_servers": "kafka.example.com:9092",
"security_protocol": "SASL_SSL",
"sasl_mechanism": "PLAIN",
"key": secret("KAFKA_KEY"),
"secret": secret("KAFKA_SECRET"),
})
landing_s3 = define_s3_connection("landing_s3", {
"region": "eu-west-1",
"arn": secret("AWS_ROLE_ARN"),
})
landing_gcs = define_gcs_connection("landing_gcs", {
"service_account_credentials_json": secret("GCS_SERVICE_ACCOUNT_CREDENTIALS_JSON"),
})
events_dynamodb = define_dynamodb_connection("events_dynamodb", {
"region": "us-east-1",
"arn": secret("DYNAMODB_ROLE_ARN"),
})
```
A DynamoDB connection takes a `region` and an `arn`. The `arn` is the IAM role Tinybird assumes to read your table, its stream, and the S3 export bucket. Both fields are required.
## Datasource ingestion config (https://www.tinybird.co/docs/forward/dev-reference/python-sdk-resources#datasource-ingestion-config)
You can attach Kafka, S3, GCS, or DynamoDB ingestion directly to a Data Source:
```
from tinybird_sdk import define_datasource, engine, t
kafka_events = define_datasource("kafka_events", {
"schema": {
"timestamp": t.date_time(),
"payload": t.string(),
},
"engine": engine.merge_tree({"sorting_key": ["timestamp"]}),
"kafka": {
"connection": events_kafka,
"topic": "events",
"group_id": "events-consumer",
},
})
```
### DynamoDB (https://www.tinybird.co/docs/forward/dev-reference/python-sdk-resources#dynamodb)
The DynamoDB connector mirrors an Amazon DynamoDB table into Tinybird: an initial backfill via a Point-in-Time Recovery export to S3, followed by continuous Change Data Capture from DynamoDB Streams. See the [DynamoDB connector](/docs/forward/ingest-data/connectors/dynamodb) page for the table requirements and IAM setup.
Defining the connection in code isn't enough. Tinybird assumes an IAM role in your own AWS account, and the role's trust policy must include a Workspace-specific external ID that only Tinybird generates. Run `tinybird connection create dynamodb` to get the external ID, the `Principal` account, and the policy JSON. Then create the role in AWS and store its ARN as the `DYNAMODB_ROLE_ARN` secret. Without the correct external ID, `deploy` fails with a `403 ... external ID` error. See the [DynamoDB connector](/docs/forward/ingest-data/connectors/dynamodb) page for the full setup.
Each row in the Data Source represents a *change* to your table, not the current state. To keep its size under control, DynamoDB Data Sources use the `ReplacingMergeTree` engine. See [Querying the data](https://www.tinybird.co/docs/forward/dev-reference/python-sdk-resources#querying-the-data) for considerations.
```
from tinybird_sdk import column, define_datasource, engine, t
orders = define_datasource("orders", {
"schema": {
"id": column(t.string(), {"json_path": "$.Item.id"}),
"_record": column(t.string(), {"json_path": "$.NewImage"}),
"_timestamp": column(t.date_time64(3), {"json_path": "$.ApproximateCreationDateTime"}),
"_event_name": column(t.string().low_cardinality(), {"json_path": "$.eventName"}),
"_is_deleted": column(t.uint8(), {"json_path": "$._is_deleted"}),
},
"engine": engine.replacing_merge_tree({
"sorting_key": ["id"],
"ver": "_timestamp",
"is_deleted": "_is_deleted",
}),
"dynamodb": {
"connection": events_dynamodb,
"table_arn": "arn:aws:dynamodb:us-east-1:123456789012:table/orders",
"export_bucket": "s3://my-tinybird-dynamodb-exports",
},
})
```
#### Set the role ARN secret before deploying (https://www.tinybird.co/docs/forward/dev-reference/python-sdk-resources#set-the-role-arn-secret-before-deploying)
`secret("DYNAMODB_ROLE_ARN")` resolves to a [secret](/docs/forward/dev-reference/commands/tb-secret) that must already exist in the target Workspace. Set it before you deploy:
```
tinybird --cloud secret set DYNAMODB_ROLE_ARN "arn:aws:iam::123456789012:role/TinybirdRole"
tinybird --cloud secret ls
```
CLI commands such as `secret set` and `deploy` authenticate separately from the runtime client. They use the Workspace admin Token from `tinybird login` (or an explicit `--token` / `--host`), not the `TINYBIRD_TOKEN` in `.env.local`. `.env.local` feeds only the `Tinybird(...)` runtime client.
If `deploy` or `secret` reports `Forbidden: invalid authentication token`, or runs against an unexpected Workspace, the CLI is using a stale or foreign Token rather than your `.env.local`. Re-run `tinybird login`, or pass the Workspace admin Token explicitly:
```
tinybird --cloud --host https://api.tinybird.co \
--token "" deploy --check
```
`secret set` and `deploy` are Workspace operations, so they need the Workspace admin Token, not an Organization or user Token.
## Runtime client (https://www.tinybird.co/docs/forward/dev-reference/python-sdk-resources#runtime-client)
Create a typed runtime client with `Tinybird`:
```
from tinybird_sdk import Tinybird
from .resources import page_views, top_pages
tinybird = Tinybird({
"datasources": {"page_views": page_views},
"pipes": {"top_pages": top_pages},
})
```
Then ingest and query:
```
tinybird.page_views.ingest({
"timestamp": "2026-01-01 10:00:00",
"pathname": "/home",
"session_id": "abc123",
})
result = tinybird.top_pages.query({
"start_date": "2026-01-01 00:00:00",
"end_date": "2026-01-02 00:00:00",
"limit": 5,
})
```
## Next steps (https://www.tinybird.co/docs/forward/dev-reference/python-sdk-resources#next-steps)
- Review CLI commands: [Python SDK CLI commands](/docs/forward/dev-reference/commands/python-sdk-cli)
- Follow the setup and deployment flow: [Quick start: Python SDK](/docs/forward/quickstarts/python-sdk)
---
URL: https://www.tinybird.co/docs/forward/dev-reference/list-of-errors
Last update: 2025-05-07T10:44:34.000Z
Content:
---
title: "List of API endpoint database errors \| Tinybird"
theme-color: "#171612"
description: "The following list contains all internal database errors that an API endpoint might return, and their numbers."
inkeep:version: "forward"
---
# List of internal database errors (https://www.tinybird.co/docs/forward/dev-reference/list-of-errors#list-of-internal-database-errors)
API endpoint responses have an additional HTTP header, `X-DB-Exception-Code`, where you can check the internal database error, reported as a stringified number.
The following list contains all internal database errors and their numbers:
- `UNSUPPORTED_METHOD = "1"`
- `UNSUPPORTED_PARAMETER = "2"`
- `UNEXPECTED_END_OF_FILE = "3"`
- `EXPECTED_END_OF_FILE = "4"`
- `CANNOT_PARSE_TEXT = "6"`
- `INCORRECT_NUMBER_OF_COLUMNS = "7"`
- `THERE_IS_NO_COLUMN = "8"`
- `SIZES_OF_COLUMNS_DOESNT_MATCH = "9"`
- `NOT_FOUND_COLUMN_IN_BLOCK = "10"`
- `POSITION_OUT_OF_BOUND = "11"`
- `PARAMETER_OUT_OF_BOUND = "12"`
- `SIZES_OF_COLUMNS_IN_TUPLE_DOESNT_MATCH = "13"`
- `DUPLICATE_COLUMN = "15"`
- `NO_SUCH_COLUMN_IN_TABLE = "16"`
- `DELIMITER_IN_STRING_LITERAL_DOESNT_MATCH = "17"`
- `CANNOT_INSERT_ELEMENT_INTO_CONSTANT_COLUMN = "18"`
- `SIZE_OF_FIXED_STRING_DOESNT_MATCH = "19"`
- `NUMBER_OF_COLUMNS_DOESNT_MATCH = "20"`
- `CANNOT_READ_ALL_DATA_FROM_TAB_SEPARATED_INPUT = "21"`
- `CANNOT_PARSE_ALL_VALUE_FROM_TAB_SEPARATED_INPUT = "22"`
- `CANNOT_READ_FROM_ISTREAM = "23"`
- `CANNOT_WRITE_TO_OSTREAM = "24"`
- `CANNOT_PARSE_ESCAPE_SEQUENCE = "25"`
- `CANNOT_PARSE_QUOTED_STRING = "26"`
- `CANNOT_PARSE_INPUT_ASSERTION_FAILED = "27"`
- `CANNOT_PRINT_FLOAT_OR_DOUBLE_NUMBER = "28"`
- `CANNOT_PRINT_INTEGER = "29"`
- `CANNOT_READ_SIZE_OF_COMPRESSED_CHUNK = "30"`
- `CANNOT_READ_COMPRESSED_CHUNK = "31"`
- `ATTEMPT_TO_READ_AFTER_EOF = "32"`
- `CANNOT_READ_ALL_DATA = "33"`
- `TOO_MANY_ARGUMENTS_FOR_FUNCTION = "34"`
- `TOO_FEW_ARGUMENTS_FOR_FUNCTION = "35"`
- `BAD_ARGUMENTS = "36"`
- `UNKNOWN_ELEMENT_IN_AST = "37"`
- `CANNOT_PARSE_DATE = "38"`
- `TOO_LARGE_SIZE_COMPRESSED = "39"`
- `CHECKSUM_DOESNT_MATCH = "40"`
- `CANNOT_PARSE_DATETIME = "41"`
- `NUMBER_OF_ARGUMENTS_DOESNT_MATCH = "42"`
- `ILLEGAL_TYPE_OF_ARGUMENT = "43"`
- `ILLEGAL_COLUMN = "44"`
- `ILLEGAL_NUMBER_OF_RESULT_COLUMNS = "45"`
- `UNKNOWN_FUNCTION = "46"`
- `UNKNOWN_IDENTIFIER = "47"`
- `NOT_IMPLEMENTED = "48"`
- `LOGICAL_ERROR = "49"`
- `UNKNOWN_TYPE = "50"`
- `EMPTY_LIST_OF_COLUMNS_QUERIED = "51"`
- `COLUMN_QUERIED_MORE_THAN_ONCE = "52"`
- `TYPE_MISMATCH = "53"`
- `STORAGE_DOESNT_ALLOW_PARAMETERS = "54"`
- `STORAGE_REQUIRES_PARAMETER = "55"`
- `UNKNOWN_STORAGE = "56"`
- `TABLE_ALREADY_EXISTS = "57"`
- `TABLE_METADATA_ALREADY_EXISTS = "58"`
- `ILLEGAL_TYPE_OF_COLUMN_FOR_FILTER = "59"`
- `UNKNOWN_TABLE = "60"`
- `ONLY_FILTER_COLUMN_IN_BLOCK = "61"`
- `SYNTAX_ERROR = "62"`
- `UNKNOWN_AGGREGATE_FUNCTION = "63"`
- `CANNOT_READ_AGGREGATE_FUNCTION_FROM_TEXT = "64"`
- `CANNOT_WRITE_AGGREGATE_FUNCTION_AS_TEXT = "65"`
- `NOT_A_COLUMN = "66"`
- `ILLEGAL_KEY_OF_AGGREGATION = "67"`
- `CANNOT_GET_SIZE_OF_FIELD = "68"`
- `ARGUMENT_OUT_OF_BOUND = "69"`
- `CANNOT_CONVERT_TYPE = "70"`
- `CANNOT_WRITE_AFTER_END_OF_BUFFER = "71"`
- `CANNOT_PARSE_NUMBER = "72"`
- `UNKNOWN_FORMAT = "73"`
- `CANNOT_READ_FROM_FILE_DESCRIPTOR = "74"`
- `CANNOT_WRITE_TO_FILE_DESCRIPTOR = "75"`
- `CANNOT_OPEN_FILE = "76"`
- `CANNOT_CLOSE_FILE = "77"`
- `UNKNOWN_TYPE_OF_QUERY = "78"`
- `INCORRECT_FILE_NAME = "79"`
- `INCORRECT_QUERY = "80"`
- `UNKNOWN_DATABASE = "81"`
- `DATABASE_ALREADY_EXISTS = "82"`
- `DIRECTORY_DOESNT_EXIST = "83"`
- `DIRECTORY_ALREADY_EXISTS = "84"`
- `FORMAT_IS_NOT_SUITABLE_FOR_INPUT = "85"`
- `RECEIVED_ERROR_FROM_REMOTE_IO_SERVER = "86"`
- `CANNOT_SEEK_THROUGH_FILE = "87"`
- `CANNOT_TRUNCATE_FILE = "88"`
- `UNKNOWN_COMPRESSION_METHOD = "89"`
- `EMPTY_LIST_OF_COLUMNS_PASSED = "90"`
- `SIZES_OF_MARKS_FILES_ARE_INCONSISTENT = "91"`
- `EMPTY_DATA_PASSED = "92"`
- `UNKNOWN_AGGREGATED_DATA_VARIANT = "93"`
- `CANNOT_MERGE_DIFFERENT_AGGREGATED_DATA_VARIANTS = "94"`
- `CANNOT_READ_FROM_SOCKET = "95"`
- `CANNOT_WRITE_TO_SOCKET = "96"`
- `CANNOT_READ_ALL_DATA_FROM_CHUNKED_INPUT = "97"`
- `CANNOT_WRITE_TO_EMPTY_BLOCK_OUTPUT_STREAM = "98"`
- `UNKNOWN_PACKET_FROM_CLIENT = "99"`
- `UNKNOWN_PACKET_FROM_SERVER = "100"`
- `UNEXPECTED_PACKET_FROM_CLIENT = "101"`
- `UNEXPECTED_PACKET_FROM_SERVER = "102"`
- `RECEIVED_DATA_FOR_WRONG_QUERY_ID = "103"`
- `TOO_SMALL_BUFFER_SIZE = "104"`
- `CANNOT_READ_HISTORY = "105"`
- `CANNOT_APPEND_HISTORY = "106"`
- `FILE_DOESNT_EXIST = "107"`
- `NO_DATA_TO_INSERT = "108"`
- `CANNOT_BLOCK_SIGNAL = "109"`
- `CANNOT_UNBLOCK_SIGNAL = "110"`
- `CANNOT_MANIPULATE_SIGSET = "111"`
- `CANNOT_WAIT_FOR_SIGNAL = "112"`
- `THERE_IS_NO_SESSION = "113"`
- `CANNOT_CLOCK_GETTIME = "114"`
- `UNKNOWN_SETTING = "115"`
- `THERE_IS_NO_DEFAULT_VALUE = "116"`
- `INCORRECT_DATA = "117"`
- `ENGINE_REQUIRED = "119"`
- `CANNOT_INSERT_VALUE_OF_DIFFERENT_SIZE_INTO_TUPLE = "120"`
- `UNSUPPORTED_JOIN_KEYS = "121"`
- `INCOMPATIBLE_COLUMNS = "122"`
- `UNKNOWN_TYPE_OF_AST_NODE = "123"`
- `INCORRECT_ELEMENT_OF_SET = "124"`
- `INCORRECT_RESULT_OF_SCALAR_SUBQUERY = "125"`
- `CANNOT_GET_RETURN_TYPE = "126"`
- `ILLEGAL_INDEX = "127"`
- `TOO_LARGE_ARRAY_SIZE = "128"`
- `FUNCTION_IS_SPECIAL = "129"`
- `CANNOT_READ_ARRAY_FROM_TEXT = "130"`
- `TOO_LARGE_STRING_SIZE = "131"`
- `AGGREGATE_FUNCTION_DOESNT_ALLOW_PARAMETERS = "133"`
- `PARAMETERS_TO_AGGREGATE_FUNCTIONS_MUST_BE_LITERALS = "134"`
- `ZERO_ARRAY_OR_TUPLE_INDEX = "135"`
- `UNKNOWN_ELEMENT_IN_CONFIG = "137"`
- `EXCESSIVE_ELEMENT_IN_CONFIG = "138"`
- `NO_ELEMENTS_IN_CONFIG = "139"`
- `ALL_REQUESTED_COLUMNS_ARE_MISSING = "140"`
- `SAMPLING_NOT_SUPPORTED = "141"`
- `NOT_FOUND_NODE = "142"`
- `FOUND_MORE_THAN_ONE_NODE = "143"`
- `FIRST_DATE_IS_BIGGER_THAN_LAST_DATE = "144"`
- `UNKNOWN_OVERFLOW_MODE = "145"`
- `QUERY_SECTION_DOESNT_MAKE_SENSE = "146"`
- `NOT_FOUND_FUNCTION_ELEMENT_FOR_AGGREGATE = "147"`
- `NOT_FOUND_RELATION_ELEMENT_FOR_CONDITION = "148"`
- `NOT_FOUND_RHS_ELEMENT_FOR_CONDITION = "149"`
- `EMPTY_LIST_OF_ATTRIBUTES_PASSED = "150"`
- `INDEX_OF_COLUMN_IN_SORT_CLAUSE_IS_OUT_OF_RANGE = "151"`
- `UNKNOWN_DIRECTION_OF_SORTING = "152"`
- `ILLEGAL_DIVISION = "153"`
- `AGGREGATE_FUNCTION_NOT_APPLICABLE = "154"`
- `UNKNOWN_RELATION = "155"`
- `DICTIONARIES_WAS_NOT_LOADED = "156"`
- `ILLEGAL_OVERFLOW_MODE = "157"`
- `TOO_MANY_ROWS = "158"`
- `TIMEOUT_EXCEEDED = "159"`
- `TOO_SLOW = "160"`
- `TOO_MANY_COLUMNS = "161"`
- `TOO_DEEP_SUBQUERIES = "162"`
- `TOO_DEEP_PIPELINE = "163"`
- `READONLY = "164"`
- `TOO_MANY_TEMPORARY_COLUMNS = "165"`
- `TOO_MANY_TEMPORARY_NON_CONST_COLUMNS = "166"`
- `TOO_DEEP_AST = "167"`
- `TOO_BIG_AST = "168"`
- `BAD_TYPE_OF_FIELD = "169"`
- `BAD_GET = "170"`
- `CANNOT_CREATE_DIRECTORY = "172"`
- `CANNOT_ALLOCATE_MEMORY = "173"`
- `CYCLIC_ALIASES = "174"`
- `CHUNK_NOT_FOUND = "176"`
- `DUPLICATE_CHUNK_NAME = "177"`
- `MULTIPLE_ALIASES_FOR_EXPRESSION = "178"`
- `MULTIPLE_EXPRESSIONS_FOR_ALIAS = "179"`
- `THERE_IS_NO_PROFILE = "180"`
- `ILLEGAL_FINAL = "181"`
- `ILLEGAL_PREWHERE = "182"`
- `UNEXPECTED_EXPRESSION = "183"`
- `ILLEGAL_AGGREGATION = "184"`
- `UNSUPPORTED_MYISAM_BLOCK_TYPE = "185"`
- `UNSUPPORTED_COLLATION_LOCALE = "186"`
- `COLLATION_COMPARISON_FAILED = "187"`
- `UNKNOWN_ACTION = "188"`
- `TABLE_MUST_NOT_BE_CREATED_MANUALLY = "189"`
- `SIZES_OF_ARRAYS_DOESNT_MATCH = "190"`
- `SET_SIZE_LIMIT_EXCEEDED = "191"`
- `UNKNOWN_USER = "192"`
- `WRONG_PASSWORD = "193"`
- `REQUIRED_PASSWORD = "194"`
- `IP_ADDRESS_NOT_ALLOWED = "195"`
- `UNKNOWN_ADDRESS_PATTERN_TYPE = "196"`
- `SERVER_REVISION_IS_TOO_OLD = "197"`
- `DNS_ERROR = "198"`
- `UNKNOWN_QUOTA = "199"`
- `QUOTA_DOESNT_ALLOW_KEYS = "200"`
- `QUOTA_EXCEEDED = "201"`
- `TOO_MANY_SIMULTANEOUS_QUERIES = "202"`
- `NO_FREE_CONNECTION = "203"`
- `CANNOT_FSYNC = "204"`
- `NESTED_TYPE_TOO_DEEP = "205"`
- `ALIAS_REQUIRED = "206"`
- `AMBIGUOUS_IDENTIFIER = "207"`
- `EMPTY_NESTED_TABLE = "208"`
- `SOCKET_TIMEOUT = "209"`
- `NETWORK_ERROR = "210"`
- `EMPTY_QUERY = "211"`
- `UNKNOWN_LOAD_BALANCING = "212"`
- `UNKNOWN_TOTALS_MODE = "213"`
- `CANNOT_STATVFS = "214"`
- `NOT_AN_AGGREGATE = "215"`
- `QUERY_WITH_SAME_ID_IS_ALREADY_RUNNING = "216"`
- `CLIENT_HAS_CONNECTED_TO_WRONG_PORT = "217"`
- `TABLE_IS_DROPPED = "218"`
- `DATABASE_NOT_EMPTY = "219"`
- `DUPLICATE_INTERSERVER_IO_ENDPOINT = "220"`
- `NO_SUCH_INTERSERVER_IO_ENDPOINT = "221"`
- `ADDING_REPLICA_TO_NON_EMPTY_TABLE = "222"`
- `UNEXPECTED_AST_STRUCTURE = "223"`
- `REPLICA_IS_ALREADY_ACTIVE = "224"`
- `NO_ZOOKEEPER = "225"`
- `NO_FILE_IN_DATA_PART = "226"`
- `UNEXPECTED_FILE_IN_DATA_PART = "227"`
- `BAD_SIZE_OF_FILE_IN_DATA_PART = "228"`
- `QUERY_IS_TOO_LARGE = "229"`
- `NOT_FOUND_EXPECTED_DATA_PART = "230"`
- `TOO_MANY_UNEXPECTED_DATA_PARTS = "231"`
- `NO_SUCH_DATA_PART = "232"`
- `BAD_DATA_PART_NAME = "233"`
- `NO_REPLICA_HAS_PART = "234"`
- `DUPLICATE_DATA_PART = "235"`
- `ABORTED = "236"`
- `NO_REPLICA_NAME_GIVEN = "237"`
- `FORMAT_VERSION_TOO_OLD = "238"`
- `CANNOT_MUNMAP = "239"`
- `CANNOT_MREMAP = "240"`
- `MEMORY_LIMIT_EXCEEDED = "241"`
- `TABLE_IS_READ_ONLY = "242"`
- `NOT_ENOUGH_SPACE = "243"`
- `UNEXPECTED_ZOOKEEPER_ERROR = "244"`
- `CORRUPTED_DATA = "246"`
- `INCORRECT_MARK = "247"`
- `INVALID_PARTITION_VALUE = "248"`
- `NOT_ENOUGH_BLOCK_NUMBERS = "250"`
- `NO_SUCH_REPLICA = "251"`
- `TOO_MANY_PARTS = "252"`
- `REPLICA_IS_ALREADY_EXIST = "253"`
- `NO_ACTIVE_REPLICAS = "254"`
- `TOO_MANY_RETRIES_TO_FETCH_PARTS = "255"`
- `PARTITION_ALREADY_EXISTS = "256"`
- `PARTITION_DOESNT_EXIST = "257"`
- `UNION_ALL_RESULT_STRUCTURES_MISMATCH = "258"`
- `CLIENT_OUTPUT_FORMAT_SPECIFIED = "260"`
- `UNKNOWN_BLOCK_INFO_FIELD = "261"`
- `BAD_COLLATION = "262"`
- `CANNOT_COMPILE_CODE = "263"`
- `INCOMPATIBLE_TYPE_OF_JOIN = "264"`
- `NO_AVAILABLE_REPLICA = "265"`
- `MISMATCH_REPLICAS_DATA_SOURCES = "266"`
- `STORAGE_DOESNT_SUPPORT_PARALLEL_REPLICAS = "267"`
- `CPUID_ERROR = "268"`
- `INFINITE_LOOP = "269"`
- `CANNOT_COMPRESS = "270"`
- `CANNOT_DECOMPRESS = "271"`
- `CANNOT_IO_SUBMIT = "272"`
- `CANNOT_IO_GETEVENTS = "273"`
- `AIO_READ_ERROR = "274"`
- `AIO_WRITE_ERROR = "275"`
- `INDEX_NOT_USED = "277"`
- `ALL_CONNECTION_TRIES_FAILED = "279"`
- `NO_AVAILABLE_DATA = "280"`
- `DICTIONARY_IS_EMPTY = "281"`
- `INCORRECT_INDEX = "282"`
- `UNKNOWN_DISTRIBUTED_PRODUCT_MODE = "283"`
- `WRONG_GLOBAL_SUBQUERY = "284"`
- `TOO_FEW_LIVE_REPLICAS = "285"`
- `UNSATISFIED_QUORUM_FOR_PREVIOUS_WRITE = "286"`
- `UNKNOWN_FORMAT_VERSION = "287"`
- `DISTRIBUTED_IN_JOIN_SUBQUERY_DENIED = "288"`
- `REPLICA_IS_NOT_IN_QUORUM = "289"`
- `LIMIT_EXCEEDED = "290"`
- `DATABASE_ACCESS_DENIED = "291"`
- `MONGODB_CANNOT_AUTHENTICATE = "293"`
- `INVALID_BLOCK_EXTRA_INFO = "294"`
- `RECEIVED_EMPTY_DATA = "295"`
- `NO_REMOTE_SHARD_FOUND = "296"`
- `SHARD_HAS_NO_CONNECTIONS = "297"`
- `CANNOT_PIPE = "298"`
- `CANNOT_FORK = "299"`
- `CANNOT_DLSYM = "300"`
- `CANNOT_CREATE_CHILD_PROCESS = "301"`
- `CHILD_WAS_NOT_EXITED_NORMALLY = "302"`
- `CANNOT_SELECT = "303"`
- `CANNOT_WAITPID = "304"`
- `TABLE_WAS_NOT_DROPPED = "305"`
- `TOO_DEEP_RECURSION = "306"`
- `TOO_MANY_BYTES = "307"`
- `UNEXPECTED_NODE_IN_ZOOKEEPER = "308"`
- `FUNCTION_CANNOT_HAVE_PARAMETERS = "309"`
- `INVALID_SHARD_WEIGHT = "317"`
- `INVALID_CONFIG_PARAMETER = "318"`
- `UNKNOWN_STATUS_OF_INSERT = "319"`
- `VALUE_IS_OUT_OF_RANGE_OF_DATA_TYPE = "321"`
- `BARRIER_TIMEOUT = "335"`
- `UNKNOWN_DATABASE_ENGINE = "336"`
- `DDL_GUARD_IS_ACTIVE = "337"`
- `UNFINISHED = "341"`
- `METADATA_MISMATCH = "342"`
- `SUPPORT_IS_DISABLED = "344"`
- `TABLE_DIFFERS_TOO_MUCH = "345"`
- `CANNOT_CONVERT_CHARSET = "346"`
- `CANNOT_LOAD_CONFIG = "347"`
- `CANNOT_INSERT_NULL_IN_ORDINARY_COLUMN = "349"`
- `INCOMPATIBLE_SOURCE_TABLES = "350"`
- `AMBIGUOUS_TABLE_NAME = "351"`
- `AMBIGUOUS_COLUMN_NAME = "352"`
- `INDEX_OF_POSITIONAL_ARGUMENT_IS_OUT_OF_RANGE = "353"`
- `ZLIB_INFLATE_FAILED = "354"`
- `ZLIB_DEFLATE_FAILED = "355"`
- `BAD_LAMBDA = "356"`
- `RESERVED_IDENTIFIER_NAME = "357"`
- `INTO_OUTFILE_NOT_ALLOWED = "358"`
- `TABLE_SIZE_EXCEEDS_MAX_DROP_SIZE_LIMIT = "359"`
- `CANNOT_CREATE_CHARSET_CONVERTER = "360"`
- `SEEK_POSITION_OUT_OF_BOUND = "361"`
- `CURRENT_WRITE_BUFFER_IS_EXHAUSTED = "362"`
- `CANNOT_CREATE_IO_BUFFER = "363"`
- `RECEIVED_ERROR_TOO_MANY_REQUESTS = "364"`
- `SIZES_OF_NESTED_COLUMNS_ARE_INCONSISTENT = "366"`
- `TOO_MANY_FETCHES = "367"`
- `ALL_REPLICAS_ARE_STALE = "369"`
- `DATA_TYPE_CANNOT_BE_USED_IN_TABLES = "370"`
- `INCONSISTENT_CLUSTER_DEFINITION = "371"`
- `SESSION_NOT_FOUND = "372"`
- `SESSION_IS_LOCKED = "373"`
- `INVALID_SESSION_TIMEOUT = "374"`
- `CANNOT_DLOPEN = "375"`
- `CANNOT_PARSE_UUID = "376"`
- `ILLEGAL_SYNTAX_FOR_DATA_TYPE = "377"`
- `DATA_TYPE_CANNOT_HAVE_ARGUMENTS = "378"`
- `UNKNOWN_STATUS_OF_DISTRIBUTED_DDL_TASK = "379"`
- `CANNOT_KILL = "380"`
- `HTTP_LENGTH_REQUIRED = "381"`
- `CANNOT_LOAD_CATBOOST_MODEL = "382"`
- `CANNOT_APPLY_CATBOOST_MODEL = "383"`
- `PART_IS_TEMPORARILY_LOCKED = "384"`
- `MULTIPLE_STREAMS_REQUIRED = "385"`
- `NO_COMMON_TYPE = "386"`
- `DICTIONARY_ALREADY_EXISTS = "387"`
- `CANNOT_ASSIGN_OPTIMIZE = "388"`
- `INSERT_WAS_DEDUPLICATED = "389"`
- `CANNOT_GET_CREATE_TABLE_QUERY = "390"`
- `EXTERNAL_LIBRARY_ERROR = "391"`
- `QUERY_IS_PROHIBITED = "392"`
- `THERE_IS_NO_QUERY = "393"`
- `QUERY_WAS_CANCELLED = "394"`
- `FUNCTION_THROW_IF_VALUE_IS_NON_ZERO = "395"`
- `TOO_MANY_ROWS_OR_BYTES = "396"`
- `QUERY_IS_NOT_SUPPORTED_IN_MATERIALIZED_VIEW = "397"`
- `UNKNOWN_MUTATION_COMMAND = "398"`
- `FORMAT_IS_NOT_SUITABLE_FOR_OUTPUT = "399"`
- `CANNOT_STAT = "400"`
- `FEATURE_IS_NOT_ENABLED_AT_BUILD_TIME = "401"`
- `CANNOT_IOSETUP = "402"`
- `INVALID_JOIN_ON_EXPRESSION = "403"`
- `BAD_ODBC_CONNECTION_STRING = "404"`
- `PARTITION_SIZE_EXCEEDS_MAX_DROP_SIZE_LIMIT = "405"`
- `TOP_AND_LIMIT_TOGETHER = "406"`
- `DECIMAL_OVERFLOW = "407"`
- `BAD_REQUEST_PARAMETER = "408"`
- `EXTERNAL_EXECUTABLE_NOT_FOUND = "409"`
- `EXTERNAL_SERVER_IS_NOT_RESPONDING = "410"`
- `PTHREAD_ERROR = "411"`
- `NETLINK_ERROR = "412"`
- `CANNOT_SET_SIGNAL_HANDLER = "413"`
- `ALL_REPLICAS_LOST = "415"`
- `REPLICA_STATUS_CHANGED = "416"`
- `EXPECTED_ALL_OR_ANY = "417"`
- `UNKNOWN_JOIN = "418"`
- `MULTIPLE_ASSIGNMENTS_TO_COLUMN = "419"`
- `CANNOT_UPDATE_COLUMN = "420"`
- `CANNOT_ADD_DIFFERENT_AGGREGATE_STATES = "421"`
- `UNSUPPORTED_URI_SCHEME = "422"`
- `CANNOT_GETTIMEOFDAY = "423"`
- `CANNOT_LINK = "424"`
- `SYSTEM_ERROR = "425"`
- `CANNOT_COMPILE_REGEXP = "427"`
- `UNKNOWN_LOG_LEVEL = "428"`
- `FAILED_TO_GETPWUID = "429"`
- `MISMATCHING_USERS_FOR_PROCESS_AND_DATA = "430"`
- `ILLEGAL_SYNTAX_FOR_CODEC_TYPE = "431"`
- `UNKNOWN_CODEC = "432"`
- `ILLEGAL_CODEC_PARAMETER = "433"`
- `CANNOT_PARSE_PROTOBUF_SCHEMA = "434"`
- `NO_COLUMN_SERIALIZED_TO_REQUIRED_PROTOBUF_FIELD = "435"`
- `PROTOBUF_BAD_CAST = "436"`
- `PROTOBUF_FIELD_NOT_REPEATED = "437"`
- `DATA_TYPE_CANNOT_BE_PROMOTED = "438"`
- `CANNOT_SCHEDULE_TASK = "439"`
- `INVALID_LIMIT_EXPRESSION = "440"`
- `CANNOT_PARSE_DOMAIN_VALUE_FROM_STRING = "441"`
- `BAD_DATABASE_FOR_TEMPORARY_TABLE = "442"`
- `NO_COLUMNS_SERIALIZED_TO_PROTOBUF_FIELDS = "443"`
- `UNKNOWN_PROTOBUF_FORMAT = "444"`
- `CANNOT_MPROTECT = "445"`
- `FUNCTION_NOT_ALLOWED = "446"`
- `HYPERSCAN_CANNOT_SCAN_TEXT = "447"`
- `BROTLI_READ_FAILED = "448"`
- `BROTLI_WRITE_FAILED = "449"`
- `BAD_TTL_EXPRESSION = "450"`
- `BAD_TTL_FILE = "451"`
- `SETTING_CONSTRAINT_VIOLATION = "452"`
- `MYSQL_CLIENT_INSUFFICIENT_CAPABILITIES = "453"`
- `OPENSSL_ERROR = "454"`
- `SUSPICIOUS_TYPE_FOR_LOW_CARDINALITY = "455"`
- `UNKNOWN_QUERY_PARAMETER = "456"`
- `BAD_QUERY_PARAMETER = "457"`
- `CANNOT_UNLINK = "458"`
- `CANNOT_SET_THREAD_PRIORITY = "459"`
- `CANNOT_CREATE_TIMER = "460"`
- `CANNOT_SET_TIMER_PERIOD = "461"`
- `CANNOT_DELETE_TIMER = "462"`
- `CANNOT_FCNTL = "463"`
- `CANNOT_PARSE_ELF = "464"`
- `CANNOT_PARSE_DWARF = "465"`
- `INSECURE_PATH = "466"`
- `CANNOT_PARSE_BOOL = "467"`
- `CANNOT_PTHREAD_ATTR = "468"`
- `VIOLATED_CONSTRAINT = "469"`
- `QUERY_IS_NOT_SUPPORTED_IN_LIVE_VIEW = "470"`
- `INVALID_SETTING_VALUE = "471"`
- `READONLY_SETTING = "472"`
- `DEADLOCK_AVOIDED = "473"`
- `INVALID_TEMPLATE_FORMAT = "474"`
- `INVALID_WITH_FILL_EXPRESSION = "475"`
- `WITH_TIES_WITHOUT_ORDER_BY = "476"`
- `INVALID_USAGE_OF_INPUT = "477"`
- `UNKNOWN_POLICY = "478"`
- `UNKNOWN_DISK = "479"`
- `UNKNOWN_PROTOCOL = "480"`
- `PATH_ACCESS_DENIED = "481"`
- `DICTIONARY_ACCESS_DENIED = "482"`
- `TOO_MANY_REDIRECTS = "483"`
- `INTERNAL_REDIS_ERROR = "484"`
- `SCALAR_ALREADY_EXISTS = "485"`
- `CANNOT_GET_CREATE_DICTIONARY_QUERY = "487"`
- `UNKNOWN_DICTIONARY = "488"`
- `INCORRECT_DICTIONARY_DEFINITION = "489"`
- `CANNOT_FORMAT_DATETIME = "490"`
- `UNACCEPTABLE_URL = "491"`
- `ACCESS_ENTITY_NOT_FOUND = "492"`
- `ACCESS_ENTITY_ALREADY_EXISTS = "493"`
- `ACCESS_ENTITY_FOUND_DUPLICATES = "494"`
- `ACCESS_STORAGE_READONLY = "495"`
- `QUOTA_REQUIRES_CLIENT_KEY = "496"`
- `ACCESS_DENIED = "497"`
- `LIMIT_BY_WITH_TIES_IS_NOT_SUPPORTED = "498"`
- S3_ERROR = "499"
- `AZURE_BLOB_STORAGE_ERROR = "500"`
- `CANNOT_CREATE_DATABASE = "501"`
- `CANNOT_SIGQUEUE = "502"`
- `AGGREGATE_FUNCTION_THROW = "503"`
- `FILE_ALREADY_EXISTS = "504"`
- `CANNOT_DELETE_DIRECTORY = "505"`
- `UNEXPECTED_ERROR_CODE = "506"`
- `UNABLE_TO_SKIP_UNUSED_SHARDS = "507"`
- `UNKNOWN_ACCESS_TYPE = "508"`
- `INVALID_GRANT = "509"`
- `CACHE_DICTIONARY_UPDATE_FAIL = "510"`
- `UNKNOWN_ROLE = "511"`
- `SET_NON_GRANTED_ROLE = "512"`
- `UNKNOWN_PART_TYPE = "513"`
- `ACCESS_STORAGE_FOR_INSERTION_NOT_FOUND = "514"`
- `INCORRECT_ACCESS_ENTITY_DEFINITION = "515"`
- `AUTHENTICATION_FAILED = "516"`
- `CANNOT_ASSIGN_ALTER = "517"`
- `CANNOT_COMMIT_OFFSET = "518"`
- `NO_REMOTE_SHARD_AVAILABLE = "519"`
- `CANNOT_DETACH_DICTIONARY_AS_TABLE = "520"`
- `ATOMIC_RENAME_FAIL = "521"`
- `UNKNOWN_ROW_POLICY = "523"`
- `ALTER_OF_COLUMN_IS_FORBIDDEN = "524"`
- `INCORRECT_DISK_INDEX = "525"`
- `NO_SUITABLE_FUNCTION_IMPLEMENTATION = "527"`
- `CASSANDRA_INTERNAL_ERROR = "528"`
- `NOT_A_LEADER = "529"`
- `CANNOT_CONNECT_RABBITMQ = "530"`
- `CANNOT_FSTAT = "531"`
- `LDAP_ERROR = "532"`
- `INCONSISTENT_RESERVATIONS = "533"`
- `NO_RESERVATIONS_PROVIDED = "534"`
- `UNKNOWN_RAID_TYPE = "535"`
- `CANNOT_RESTORE_FROM_FIELD_DUMP = "536"`
- `ILLEGAL_MYSQL_VARIABLE = "537"`
- `MYSQL_SYNTAX_ERROR = "538"`
- `CANNOT_BIND_RABBITMQ_EXCHANGE = "539"`
- `CANNOT_DECLARE_RABBITMQ_EXCHANGE = "540"`
- `CANNOT_CREATE_RABBITMQ_QUEUE_BINDING = "541"`
- `CANNOT_REMOVE_RABBITMQ_EXCHANGE = "542"`
- `UNKNOWN_MYSQL_DATATYPES_SUPPORT_LEVEL = "543"`
- `ROW_AND_ROWS_TOGETHER = "544"`
- `FIRST_AND_NEXT_TOGETHER = "545"`
- `NO_ROW_DELIMITER = "546"`
- `INVALID_RAID_TYPE = "547"`
- `UNKNOWN_VOLUME = "548"`
- `DATA_TYPE_CANNOT_BE_USED_IN_KEY = "549"`
- `CONDITIONAL_TREE_PARENT_NOT_FOUND = "550"`
- `ILLEGAL_PROJECTION_MANIPULATOR = "551"`
- `UNRECOGNIZED_ARGUMENTS = "552"`
- `LZMA_STREAM_ENCODER_FAILED = "553"`
- `LZMA_STREAM_DECODER_FAILED = "554"`
- `ROCKSDB_ERROR = "555"`
- `SYNC_MYSQL_USER_ACCESS_ERROR = "556"`
- `UNKNOWN_UNION = "557"`
- `EXPECTED_ALL_OR_DISTINCT = "558"`
- `INVALID_GRPC_QUERY_INFO = "559"`
- `ZSTD_ENCODER_FAILED = "560"`
- `ZSTD_DECODER_FAILED = "561"`
- `TLD_LIST_NOT_FOUND = "562"`
- `CANNOT_READ_MAP_FROM_TEXT = "563"`
- `INTERSERVER_SCHEME_DOESNT_MATCH = "564"`
- `TOO_MANY_PARTITIONS = "565"`
- `CANNOT_RMDIR = "566"`
- `DUPLICATED_PART_UUIDS = "567"`
- `RAFT_ERROR = "568"`
- `MULTIPLE_COLUMNS_SERIALIZED_TO_SAME_PROTOBUF_FIELD = "569"`
- `DATA_TYPE_INCOMPATIBLE_WITH_PROTOBUF_FIELD = "570"`
- `DATABASE_REPLICATION_FAILED = "571"`
- `TOO_MANY_QUERY_PLAN_OPTIMIZATIONS = "572"`
- `EPOLL_ERROR = "573"`
- `DISTRIBUTED_TOO_MANY_PENDING_BYTES = "574"`
- `UNKNOWN_SNAPSHOT = "575"`
- `KERBEROS_ERROR = "576"`
- `INVALID_SHARD_ID = "577"`
- `INVALID_FORMAT_INSERT_QUERY_WITH_DATA = "578"`
- `INCORRECT_PART_TYPE = "579"`
- `CANNOT_SET_ROUNDING_MODE = "580"`
- `TOO_LARGE_DISTRIBUTED_DEPTH = "581"`
- `NO_SUCH_PROJECTION_IN_TABLE = "582"`
- `ILLEGAL_PROJECTION = "583"`
- `PROJECTION_NOT_USED = "584"`
- `CANNOT_PARSE_YAML = "585"`
- `CANNOT_CREATE_FILE = "586"`
- `CONCURRENT_ACCESS_NOT_SUPPORTED = "587"`
- `DISTRIBUTED_BROKEN_BATCH_INFO = "588"`
- `DISTRIBUTED_BROKEN_BATCH_FILES = "589"`
- `CANNOT_SYSCONF = "590"`
- `SQLITE_ENGINE_ERROR = "591"`
- `DATA_ENCRYPTION_ERROR = "592"`
- `ZERO_COPY_REPLICATION_ERROR = "593"`
- BZIP2_STREAM_DECODER_FAILED = "594"
- BZIP2_STREAM_ENCODER_FAILED = "595"
- `INTERSECT_OR_EXCEPT_RESULT_STRUCTURES_MISMATCH = "596"`
- `NO_SUCH_ERROR_CODE = "597"`
- `BACKUP_ALREADY_EXISTS = "598"`
- `BACKUP_NOT_FOUND = "599"`
- `BACKUP_VERSION_NOT_SUPPORTED = "600"`
- `BACKUP_DAMAGED = "601"`
- `NO_BASE_BACKUP = "602"`
- `WRONG_BASE_BACKUP = "603"`
- `BACKUP_ENTRY_ALREADY_EXISTS = "604"`
- `BACKUP_ENTRY_NOT_FOUND = "605"`
- `BACKUP_IS_EMPTY = "606"`
- `CANNOT_RESTORE_DATABASE = "607"`
- `CANNOT_RESTORE_TABLE = "608"`
- `FUNCTION_ALREADY_EXISTS = "609"`
- `CANNOT_DROP_FUNCTION = "610"`
- `CANNOT_CREATE_RECURSIVE_FUNCTION = "611"`
- `OBJECT_ALREADY_STORED_ON_DISK = "612"`
- `OBJECT_WAS_NOT_STORED_ON_DISK = "613"`
- `POSTGRESQL_CONNECTION_FAILURE = "614"`
- `CANNOT_ADVISE = "615"`
- `UNKNOWN_READ_METHOD = "616"`
- LZ4_ENCODER_FAILED = "617"
- LZ4_DECODER_FAILED = "618"
- `POSTGRESQL_REPLICATION_INTERNAL_ERROR = "619"`
- `QUERY_NOT_ALLOWED = "620"`
- `CANNOT_NORMALIZE_STRING = "621"`
- `CANNOT_PARSE_CAPN_PROTO_SCHEMA = "622"`
- `CAPN_PROTO_BAD_CAST = "623"`
- `BAD_FILE_TYPE = "624"`
- `IO_SETUP_ERROR = "625"`
- `CANNOT_SKIP_UNKNOWN_FIELD = "626"`
- `BACKUP_ENGINE_NOT_FOUND = "627"`
- `OFFSET_FETCH_WITHOUT_ORDER_BY = "628"`
- `HTTP_RANGE_NOT_SATISFIABLE = "629"`
- `HAVE_DEPENDENT_OBJECTS = "630"`
- `UNKNOWN_FILE_SIZE = "631"`
- `UNEXPECTED_DATA_AFTER_PARSED_VALUE = "632"`
- `QUERY_IS_NOT_SUPPORTED_IN_WINDOW_VIEW = "633"`
- `MONGODB_ERROR = "634"`
- `CANNOT_POLL = "635"`
- `CANNOT_EXTRACT_TABLE_STRUCTURE = "636"`
- `INVALID_TABLE_OVERRIDE = "637"`
- `SNAPPY_UNCOMPRESS_FAILED = "638"`
- `SNAPPY_COMPRESS_FAILED = "639"`
- `NO_HIVEMETASTORE = "640"`
- `CANNOT_APPEND_TO_FILE = "641"`
- `CANNOT_PACK_ARCHIVE = "642"`
- `CANNOT_UNPACK_ARCHIVE = "643"`
- `REMOTE_FS_OBJECT_CACHE_ERROR = "644"`
- `NUMBER_OF_DIMENSIONS_MISMATCHED = "645"`
- `CANNOT_BACKUP_DATABASE = "646"`
- `CANNOT_BACKUP_TABLE = "647"`
- `WRONG_DDL_RENAMING_SETTINGS = "648"`
- `INVALID_TRANSACTION = "649"`
- `SERIALIZATION_ERROR = "650"`
- `CAPN_PROTO_BAD_TYPE = "651"`
- `ONLY_NULLS_WHILE_READING_SCHEMA = "652"`
- `CANNOT_PARSE_BACKUP_SETTINGS = "653"`
- `WRONG_BACKUP_SETTINGS = "654"`
- `FAILED_TO_SYNC_BACKUP_OR_RESTORE = "655"`
- `MEILISEARCH_EXCEPTION = "656"`
- `UNSUPPORTED_MEILISEARCH_TYPE = "657"`
- `MEILISEARCH_MISSING_SOME_COLUMNS = "658"`
- `UNKNOWN_STATUS_OF_TRANSACTION = "659"`
- `HDFS_ERROR = "660"`
- `CANNOT_SEND_SIGNAL = "661"`
- `FS_METADATA_ERROR = "662"`
- `INCONSISTENT_METADATA_FOR_BACKUP = "663"`
- `ACCESS_STORAGE_DOESNT_ALLOW_BACKUP = "664"`
- `CANNOT_CONNECT_NATS = "665"`
- `NOT_INITIALIZED = "667"`
- `INVALID_STATE = "668"`
- `NAMED_COLLECTION_DOESNT_EXIST = "669"`
- `NAMED_COLLECTION_ALREADY_EXISTS = "670"`
- `NAMED_COLLECTION_IS_IMMUTABLE = "671"`
- `INVALID_SCHEDULER_NODE = "672"`
- `RESOURCE_ACCESS_DENIED = "673"`
- `RESOURCE_NOT_FOUND = "674"`
- CANNOT_PARSE_IPV4 = "675"
- CANNOT_PARSE_IPV6 = "676"
- `THREAD_WAS_CANCELED = "677"`
- `IO_URING_INIT_FAILED = "678"`
- `IO_URING_SUBMIT_ERROR = "679"`
- `MIXED_ACCESS_PARAMETER_TYPES = "690"`
- `UNKNOWN_ELEMENT_OF_ENUM = "691"`
- `TOO_MANY_MUTATIONS = "692"`
- `AWS_ERROR = "693"`
- `ASYNC_LOAD_CYCLE = "694"`
- `ASYNC_LOAD_FAILED = "695"`
- `ASYNC_LOAD_CANCELED = "696"`
- `CANNOT_RESTORE_TO_NONENCRYPTED_DISK = "697"`
- `INVALID_REDIS_STORAGE_TYPE = "698"`
- `INVALID_REDIS_TABLE_STRUCTURE = "699"`
- `USER_SESSION_LIMIT_EXCEEDED = "700"`
- `CLUSTER_DOESNT_EXIST = "701"`
- `CLIENT_INFO_DOES_NOT_MATCH = "702"`
- `INVALID_IDENTIFIER = "703"`
- `QUERY_CACHE_USED_WITH_NONDETERMINISTIC_FUNCTIONS = "704"`
- `TABLE_NOT_EMPTY = "705"`
- `LIBSSH_ERROR = "706"`
- `GCP_ERROR = "707"`
- `ILLEGAL_STATISTICS = "708"`
- `CANNOT_GET_REPLICATED_DATABASE_SNAPSHOT = "709"`
- `FAULT_INJECTED = "710"`
- `FILECACHE_ACCESS_DENIED = "711"`
- `TOO_MANY_MATERIALIZED_VIEWS = "712"`
- `BROKEN_PROJECTION = "713"`
- `UNEXPECTED_CLUSTER = "714"`
- `CANNOT_DETECT_FORMAT = "715"`
- `CANNOT_FORGET_PARTITION = "716"`
- `EXPERIMENTAL_FEATURE_ERROR = "717"`
- `TOO_SLOW_PARSING = "718"`
- `QUERY_CACHE_USED_WITH_SYSTEM_TABLE = "719"`
- `USER_EXPIRED = "720"`
- `DEPRECATED_FUNCTION = "721"`
- `ASYNC_LOAD_WAIT_FAILED = "722"`
- `PARQUET_EXCEPTION = "723"`
- `TOO_MANY_TABLES = "724"`
- `TOO_MANY_DATABASES = "725"`
- `DISTRIBUTED_CACHE_ERROR = "900"`
- `CANNOT_USE_DISTRIBUTED_CACHE = "901"`
- `KEEPER_EXCEPTION = "999"`
- `POCO_EXCEPTION = "1000"`
- `STD_EXCEPTION = "1001"`
- `UNKNOWN_EXCEPTION = "1002"`
---
URL: https://www.tinybird.co/docs/forward/dev-reference/datafiles
Last update: 2026-06-02T11:48:14.000Z
Content:
---
title: "Datafiles \| Tinybird"
theme-color: "#171612"
description: "Datafiles describe your Tinybird resources, like data sources, pipes, and so on. They're the source code of your project."
inkeep:version: "forward"
---
# Datafiles (https://www.tinybird.co/docs/forward/dev-reference/datafiles#datafiles)
Datafiles describe your Tinybird resources, like data sources, pipes, and so on. They're the source code of your project.
You can use datafiles to manage your projects as source code and take advantage of version control. Tinybird CLI helps you produce and push datafiles to the Tinybird platform.
## Data projects (https://www.tinybird.co/docs/forward/dev-reference/datafiles#data-projects)
A data project is a collection of datafiles that describe your Tinybird resources.
The minimal data project includes:
- A .datasource file, which describes the source and schema of your data.
- A pipe, typically a .endpoint file, which describes how to process and serve the data.
For example:
- Data source
- Pipe
```
SCHEMA >
`day` Date `json:$.day`,
`station` String `json:$.station`,
`abslevel` Nullable(Float32) `json:$.abslevel`,
`percentagevolume` Float32 `json:$.percentagevolume`,
`volume` Float32 `json:$.volume`
ENGINE "MergeTree"
ENGINE_PARTITION_KEY "station"
ENGINE_SORTING_KEY "day, station"
```
## Folder structure (https://www.tinybird.co/docs/forward/dev-reference/datafiles#folder-structure)
Tinybird projects are organized in a folder structure that helps you organize the source for each resource type.
The datafiles that you generate when you run `tb init` are organized in the following folders:
- `connections`: Contains [connection files](/docs/forward/dev-reference/datafiles/connection-files).
- `copies`: Contains the [copy pipes](/docs/forward/dev-reference/datafiles/pipe-files#copy-pipes).
- `datasources`: Contains the [data sources](/docs/forward/dev-reference/datafiles/datasource-files).
- `endpoints`: Contains the [endpoint pipes](/docs/forward/dev-reference/datafiles/pipe-files#endpoint-pipes).
- `fixtures`: Contains test fixtures and test data.
- `infra`: Contains the infrastructure files. See [tb infra](/docs/forward/dev-reference/commands/tb-infra).
- `materializations`: Contains [materialized views](/docs/forward/dev-reference/datafiles/pipe-files#materialized-pipes).
- `pipes`: Contains non-published [pipes](/docs/forward/dev-reference/datafiles/pipe-files).
- `sinks`: Contains the [sink pipes](/docs/forward/dev-reference/datafiles/pipe-files#sink-pipes).
- `tests`: Contains the test suites.
The following example shows a typical `tinybird` project folder structure that includes subfolders for supported types:
Example file structure
```
.
├── .tinyb
├── tinybird.config.json
├── connections
├── copies
├── datasources
│ └── user_actions.datasource
├── endpoints
│ ├── user_actions_line_chart.pipe
│ └── user_actions_total_widget.pipe
├── fixtures
│ ├── user_actions.prompt
│ └── user_actions_d1046873.ndjson
├── infra
│ └── aws
│ ├── config.json
│ ├── k8s.yaml
│ └── main.tf
├── materializations
├── pipes
├── sinks
└── tests
└── user_actions_line_chart.yaml
```
The `.tinyb` file contains local authentication context, and `tinybird.config.json` contains project settings like the included project path and default development mode. See [Config files](/docs/forward/dev-reference/config-files).
## Types of datafiles (https://www.tinybird.co/docs/forward/dev-reference/datafiles#types-of-datafiles)
Tinybird uses the following types of datafiles:
- Datasource files (.datasource) represent data sources. See [Datasource files](/docs/forward/dev-reference/datafiles/datasource-files).
- Pipe files (.pipe) represent pipes of various types. See [Pipe files](/docs/forward/dev-reference/datafiles/pipe-files).
## Syntactic conventions (https://www.tinybird.co/docs/forward/dev-reference/datafiles#syntactic-conventions)
Datafiles follow the same syntactic conventions.
### Casing (https://www.tinybird.co/docs/forward/dev-reference/datafiles#casing)
Instructions always appear at the beginning of a line in upper case. For example:
Basic syntax
```
COMMAND value
ANOTHER_INSTR "Value with multiple words"
```
### Multiple lines (https://www.tinybird.co/docs/forward/dev-reference/datafiles#multiple-lines)
Instructions can span multiples lines. For example:
Multiline syntax
```
SCHEMA >
`d` DateTime,
`total` Int32,
`from_novoa` Int16
```
### Comments (https://www.tinybird.co/docs/forward/dev-reference/datafiles#comments)
You can add comments using the `#` character. For example:
```
# This is a comment
COMMAND value
```
## Next steps (https://www.tinybird.co/docs/forward/dev-reference/datafiles#next-steps)
- Learn about [datasource files](/docs/forward/dev-reference/datafiles/datasource-files).
- Learn about [connection files](/docs/forward/dev-reference/datafiles/connection-files).
- Learn about [pipe files](/docs/forward/dev-reference/datafiles/pipe-files).
---
URL: https://www.tinybird.co/docs/forward/dev-reference/config-files
Last update: 2026-06-23T20:48:23.000Z
Content:
---
title: "Config files \| Tinybird"
theme-color: "#171612"
description: "Tinybird project configuration files, including tinybird.config.json and .tinyb."
inkeep:version: "forward"
---
# Config files (https://www.tinybird.co/docs/forward/dev-reference/config-files#config-files)
Tinybird projects use configuration files for two different jobs:
- `tinybird.config.json` defines project settings, such as which files to include and whether development runs in Cloud Branches or Tinybird Local.
- `.tinyb` stores local authentication and Workspace context created by `tb login`.
Commit `tinybird.config.json` to version control. Don't commit `.tinyb` because it can contain authentication tokens.
## tinybird.config.json (https://www.tinybird.co/docs/forward/dev-reference/config-files#tinybird-config-json)
Use `tinybird.config.json` to tell Tinybird where your project resources live. The development mode controls the default build target. Use `branch` for Cloud Branches or `local` for Tinybird Local.
- Tinybird CLI
- TypeScript SDK
- Python SDK
tinybird.config.json
```
{
"dev_mode": "branch",
"branch_data_mode": "last_partition",
"folder": "tinybird",
"include": [
"tinybird"
]
}
```
### Config keys (https://www.tinybird.co/docs/forward/dev-reference/config-files#config-keys)
| Key | Description |
| --- | --- |
| `include` | Paths to datafiles or SDK resource definitions that Tinybird builds and deploys. |
| `folder` | Project folder for datafiles. `tb init` writes this value to `tinybird.config.json`. When omitted, Tinybird infers the folder from `include`. |
| `dev_mode` / `devMode` | Default build target: `branch` for Cloud Branches, `local` for Tinybird Local, or `manual` to require explicit `--cloud`, `--local`, or `--branch` flags. |
| `branch_data_mode` | Data to attach when `tb build` auto-creates a Cloud Branch with `dev_mode=branch`. Only `last_partition` is supported. Defaults to `last_partition`. The legacy key `branch_data_on_create` is no longer supported. |
| `token` / `baseUrl` | SDK-only. Authentication token and API host, often loaded from environment variables. |
For development mode setup, see [Local development](/docs/forward/development-workflow/local-development) and [Using Cloud Branches](/docs/forward/development-workflow/using-cloud-branches).
## .tinyb (https://www.tinybird.co/docs/forward/dev-reference/config-files#tinyb)
The `.tinyb` file contains the Tinybird project configuration, including the authentication token obtained by running `tb login`.
Running commands requires a valid `.tinyb` file in the root of your project. If you don't have one, create one by running [tb login](/docs/forward/dev-reference/commands/tb-login).
### Location (https://www.tinybird.co/docs/forward/dev-reference/config-files#location)
You can place the `.tinyb` file in the root of your app repository or keep it together with your Tinybird datafiles. Tinybird looks for the `.tinyb` file in all parent folders until it reaches the home directory.
If your Tinybird files and folders are located in a subdirectory, such as `tinybird/`, add a `cwd` field to specify the relative path to that directory.
### File structure (https://www.tinybird.co/docs/forward/dev-reference/config-files#file-structure)
The following is a sample `.tinyb` file:
.tinyb
```
{
"host": "",
"id": "",
"name": "",
"scope": "user",
"token": "",
"tokens": {
"": "",
"": ""
},
"cwd": "",
"user_email": "",
"user_id": "",
"user_token": "",
"version": ""
}
```
---
URL: https://www.tinybird.co/docs/forward/dev-reference/common-error-patterns
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Common error patterns \| Tinybird"
theme-color: "#171612"
description: "Learn about common error patterns in Tinybird and how to resolve them."
inkeep:version: "forward"
---
# Common error patterns (https://www.tinybird.co/docs/forward/dev-reference/common-error-patterns#common-error-patterns)
This guide explains common error patterns you might encounter when working with Tinybird and provides solutions to resolve them. Check here for the [full list of errors](./list-of-errors).
### MEMORY_LIMIT_EXCEEDED (241) (https://www.tinybird.co/docs/forward/dev-reference/common-error-patterns#memory-limit-exceeded-241)
This error occurs when a query requires more memory than the available limit.
**Common causes:**
- Large result sets
- Complex joins or query time computations
- Insufficient sorting key optimization
**Solutions:**
- Optimize your sorting key to better match query patterns
- Add appropriate indexes
- Break down complex queries into smaller parts
- Precompute functions, such as [JSONExtract()](/docs/sql-reference/functions/json-functions#jsonextract-functions), and materialize them in an intermediate data source.
- Use `LIMIT` clauses when appropriate
### TIMEOUT_EXCEEDED (159) (https://www.tinybird.co/docs/forward/dev-reference/common-error-patterns#timeout-exceeded-159)
This error indicates that a query took longer than the default limit of 10 seconds to execute.
**Common causes:**
- Complex queries without proper optimization
- Large data scans
- Inefficient sorting keys
**Solutions:**
- Review and optimize your sorting key (do they align with the query filters?)
- Add appropriate indexes
- Consider materializing frequently used queries
- Break down complex queries
### TOO_MANY_PARTS (252) (https://www.tinybird.co/docs/forward/dev-reference/common-error-patterns#too-many-parts-252)
This error occurs when you have too many data parts in a partition.
**Common causes:**
- Writing to more than 30 partitions in a single ingest.
- Partition Key is too granular for regular ingests, such as a unique ID or a full-precision timestamp
- Partition Key is too granular for large historic backfills, such as a month
**Solutions:**
- Break your inserts into smaller chunks
- Reduce partition granularity
---
URL: https://www.tinybird.co/docs/forward/dev-reference/commands
Last update: 2026-06-18T09:36:38.000Z
Content:
---
title: "Tinybird CLI command reference \| Tinybird"
theme-color: "#171612"
description: "The Tinybird CLI allows you to use all the Tinybird functionality directly from the command line. Get to know the command reference."
inkeep:version: "forward"
---
# CLI command reference (https://www.tinybird.co/docs/forward/dev-reference/commands#cli-command-reference)
Recent CLI versions removed AI-assisted command generation and introduced `tb init` for project setup.
- `tb init` replaces `tb create` for scaffolding projects.
- `tb mock` has been removed. Create fixture files manually (or asking a codign agent) in `fixtures/`.
- `tb test create` has been removed. Create YAML tests manually (or asking a coding agent), then use `tb test run` and `tb test update`.
- `tb --prompt` and interactive `tb` mode are no longer available.
The following list shows all available commands in the Tinybird command-line interface, their options, and their arguments.
For SDK-specific command references, see:
- TypeScript SDK CLI (`npx tinybird`): [TypeScript SDK CLI commands](/docs/forward/dev-reference/commands/typescript-sdk-cli)
- Python SDK CLI (`tinybird`): [Python SDK CLI commands](/docs/forward/dev-reference/commands/python-sdk-cli)
- [Global options](./commands/global-options)
- [tb](./commands/tb)
- [tb branch](./commands/tb-branch)
- [tb build](./commands/tb-build)
- [tb connection](./commands/tb-connection)
- [tb copy](./commands/tb-copy)
- [tb datasource](./commands/tb-datasource)
- [tb deploy](./commands/tb-deploy)
- [tb deployment](./commands/tb-deployment)
- [tb dev](./commands/tb-dev)
- [tb endpoint](./commands/tb-endpoint)
- [tb fmt](./commands/tb-fmt)
- [tb info](./commands/tb-info)
- [tb init](./commands/tb-init)
- [tb infra](./commands/tb-infra)
- [tb job](./commands/tb-job)
- [tb local](./commands/tb-local)
- [tb login](./commands/tb-login)
- [tb logout](./commands/tb-logout)
- [tb materialization](./commands/tb-materialization)
- [tb open](./commands/tb-open)
- [tb pipe](./commands/tb-pipe)
- [tb pull](./commands/tb-pull)
- [tb secret](./commands/tb-secret)
- [tb sink](./commands/tb-sink)
- [tb sql](./commands/tb-sql)
- [tb test](./commands/tb-test)
- [tb token](./commands/tb-token)
- [tb update](./commands/tb-update)
- [tb workspace](./commands/tb-workspace)
---
URL: https://www.tinybird.co/docs/forward/core-concepts/workspaces
Last update: 2026-05-29T14:27:44.000Z
Content:
---
title: "Workspaces \| Tinybird"
theme-color: "#171612"
description: "Workspaces contain your Tinybird project resources, data, tokens, branches, and deployments."
inkeep:version: "forward"
---
# Workspaces (https://www.tinybird.co/docs/forward/core-concepts/workspaces#workspaces)
A Workspace is the cloud environment for a Tinybird project. It contains your resources, data, tokens, branches, deployments, service data sources, and member permissions.
Use Workspaces for long-lived project boundaries: production, staging, separate products, or separate customers when you need hard isolation. Use [Branches](/docs/forward/core-concepts/branches) for short-lived development and preview environments inside a Workspace.
Every Workspace belongs to an [Organization](/docs/forward/core-concepts/organizations). Organization settings control billing, plan limits, member policy, and the regions or clusters where Workspaces can run.
## What belongs to a Workspace (https://www.tinybird.co/docs/forward/core-concepts/workspaces#what-belongs-to-a-workspace)
A Workspace contains the resources you define and deploy:
- [Data Sources](/docs/forward/core-concepts/data-sources)
- [Connections](/docs/forward/core-concepts/connections)
- [Pipes](/docs/forward/core-concepts/pipes)
- [API Endpoints](/docs/forward/core-concepts/api-endpoints)
- [Materialized Views](/docs/forward/core-concepts/materialized-views)
- [Copy Pipes](/docs/forward/core-concepts/copy-pipes)
- [Sinks](/docs/forward/core-concepts/sinks)
- [Tokens](/docs/forward/core-concepts/tokens)
Workspace state also includes ingested data, secrets, service data sources, branch metadata, deployments, and member roles.
You define resources in code, test them locally or in a branch, then deploy to the workspace.
You can build Forward projects with:
- [TypeScript SDK](/docs/forward/quickstarts/typescript-sdk)
- [Python SDK](/docs/forward/quickstarts/python-sdk)
- [Tinybird CLI datafiles](/docs/forward/quickstarts/cli)
## Create a workspace (https://www.tinybird.co/docs/forward/core-concepts/workspaces#create-a-workspace)
Create new workspaces from Tinybird Cloud:
1. Open [Tinybird Cloud](https://cloud.tinybird.co/).
2. Use the workspace switcher or the **Organization settings** > **Workspaces** page.
3. Select **Create workspace**.
4. Choose the organization, region, or cluster if your organization has more than one option.
5. Choose how you want to start: TypeScript SDK, Python SDK, CLI datafiles, or the UI.
New workspaces can show SDK onboarding directly in the UI. For example, the TypeScript SDK path gives you project setup instructions, credentials, and an agent-ready prompt you can copy into your coding agent.
You can also create a workspace with the Tinybird CLI:
```
tb workspace create
```
If your account has access to multiple organizations, pass the organization ID:
```
tb workspace create --organization-id
```
## Manage workspace members (https://www.tinybird.co/docs/forward/core-concepts/workspaces#manage-workspace-members)
In Tinybird Cloud, open the workspace and go to **Settings** > **Members**.
From there you can:
- Invite users by email.
- Assign a workspace role.
- Resend or copy invite links.
- Remove members from the workspace.
Organization admins can also manage workspace membership from **Organization settings** > **Members** without being a member of every workspace.
## Workspace roles (https://www.tinybird.co/docs/forward/core-concepts/workspaces#workspace-roles)
Members always have a workspace role. You can change a member's role at any time.
| Role | Manage resources | Manage users | Create a branch |
| --- | --- | --- | --- |
| `Admin` | Yes | Yes | Yes |
| `Guest` | Yes | No | Yes |
| `Viewer` | No | No | Yes |
## Connect local tooling to a workspace (https://www.tinybird.co/docs/forward/core-concepts/workspaces#connect-local-tooling-to-a-workspace)
After creating a workspace, connect your local project to it.
For CLI datafile projects, authenticate with `tb login`:
```
tb login
```
This writes workspace credentials to `.tinyb`.
For TypeScript SDK projects, authenticate with the SDK CLI:
```
tinybird login
```
This writes workspace credentials to `.env.local`.
To see the workspace your CLI session points to, run:
```
tb workspace current
```
To list the workspaces you can access:
```
tb workspace ls
```
To switch your CLI session to another workspace:
```
tb workspace use
```
## Workspace ID (https://www.tinybird.co/docs/forward/core-concepts/workspaces#workspace-id)
Each workspace has a unique ID. You need it for some API calls, JWT configuration, and support requests.
Find it in Tinybird Cloud from the workspace list, or run:
```
tb workspace current
```
Example output:
```
** Current workspace:
--------------------------------------------------------------------------------------------
| name | id | role | plan | current |
--------------------------------------------------------------------------------------------
| analytics_prod | xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | admin | Custom | True |
--------------------------------------------------------------------------------------------
```
## Delete a workspace (https://www.tinybird.co/docs/forward/core-concepts/workspaces#delete-a-workspace)
Deleting a workspace deletes all resources and data in it, including Data Sources, ingested data, Pipes, published endpoints, tokens, branches, and deployments.
Deleted workspaces can't be recovered.
You can delete a workspace from Tinybird Cloud, or with the CLI if you are a workspace admin:
```
tb workspace delete
```
Use `--yes` to skip the confirmation prompt in automation:
```
tb workspace delete --yes
```
---
URL: https://www.tinybird.co/docs/forward/core-concepts/tokens
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Authentication \| Tinybird"
theme-color: "#171612"
description: "Learn how to authenticate your requests to Tinybird."
inkeep:version: "forward"
---
# Tokens (https://www.tinybird.co/docs/forward/core-concepts/tokens#tokens)
Tinybird uses tokens to authenticate CLI and API requests. Tokens protect access to your resources. Any operation to manage your resources using the CLI or REST API requires a valid token with the necessary permissions.
There are two types of tokens:
- [Static Tokens](/docs/forward/core-concepts/static-tokens): Use them to perform operations on your account, like importing data, creating Data Sources, or publishing APIs using the CLI or REST API. Use them to read data as well, and be mindful of their permanent nature.
- [JSON Web Tokens](/docs/forward/core-concepts/jwt): Use them to read from published API Endpoints that expose your data to an app when you want to implement filtering per user through fixed parameters (RBAC) or apply rate limiting for different end users of Tinybird Endpoints.
## Where tokens are stored and used (https://www.tinybird.co/docs/forward/core-concepts/tokens#where-tokens-are-stored-and-used)
How you store and pass tokens depends on the environment:
- **Tinybird CLI**: `tb login` stores your CLI credentials in the local `.tinyb` file.
- **SDK projects**: the TypeScript and Python SDK CLIs read `TINYBIRD_TOKEN` and `TINYBIRD_URL` from `.env.local` during local development.
- **CI/CD**: store `TINYBIRD_TOKEN` and `TINYBIRD_URL` as secret environment variables in your CI provider.
- **API calls**: pass the token in the `Authorization` header.
SDK projects commonly use a `.env.local` file during development:
.env.local
```
TINYBIRD_TOKEN=p.your_token_here
TINYBIRD_URL=https://api.tinybird.co
```
In CI/CD, set the same values as protected secrets and use them when deploying:
CI/CD deploy
```
export TINYBIRD_TOKEN="$TINYBIRD_TOKEN"
export TINYBIRD_URL="$TINYBIRD_URL"
tb deploy
```
For direct API calls, pass the token as a bearer token:
API request with a token
```
curl \
-H "Authorization: Bearer " \
"https://api.tinybird.co/v0/pipes/.json"
```
---
URL: https://www.tinybird.co/docs/forward/core-concepts/tinybird-local
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Tinybird Local \| Tinybird"
theme-color: "#171612"
description: "Run Tinybird on your machine for local development, testing, and CI."
inkeep:version: "forward"
---
# Tinybird Local (https://www.tinybird.co/docs/forward/core-concepts/tinybird-local#tinybird-local)
Tinybird Local is a containerized Tinybird runtime for development, testing, and CI. It runs on your machine and gives you the same project model you use in Tinybird Cloud: Data Sources, Pipes, Endpoints, Tokens, APIs, and local services.
Use Tinybird Local when you want a fast, isolated environment with fixture data. Use [Cloud Branches](/docs/forward/development-workflow/using-cloud-branches) when you need a shared Cloud environment, production-like data, or connector validation before deployment.
## Prerequisites (https://www.tinybird.co/docs/forward/core-concepts/tinybird-local#prerequisites)
To install and run Tinybird Local, you need:
- A [Tinybird account](https://cloud.tinybird.co/).
- Linux or macOS.
- A container runtime, such as Docker or OrbStack.
OrbStack containers do not require extra configuration. Docker users should check the following settings:
- **Apple Silicon**: Enable Rosetta in **Docker Desktop > Settings > General > Use Rosetta for x86/amd64 emulation on Apple Silicon**. Without Rosetta, Docker falls back to QEMU emulation, which can be slower and less stable.
- **Memory**: Set Docker memory to at least 8 GB in **Docker Desktop > Settings > Advanced > Memory**.
## Install Tinybird (https://www.tinybird.co/docs/forward/core-concepts/tinybird-local#install-tinybird)
Install the Tinybird CLI and Tinybird Local:
- For macOS and Linux
- For Windows
```
curl https://tinybird.co | sh
```
Authenticate with Tinybird Cloud:
```
tb login
```
This opens a browser window where you can sign in to Tinybird Cloud. If you need a specific region, see [Regions and endpoints](/docs/api-reference#regions-and-endpoints) and pass the host to `tb login`.
## Start Tinybird Local (https://www.tinybird.co/docs/forward/core-concepts/tinybird-local#start-tinybird-local)
Start Tinybird Local with the CLI:
```
tb local start
```
Run it in the background with `--daemon`:
```
tb local start --daemon
```
The CLI waits until the local services are ready. Do not start building your project until you see `Tinybird Local is ready!`.
Stop Tinybird Local with `Ctrl+C` if it is running in the foreground, or with:
```
tb local stop
```
If Tinybird Local becomes unhealthy, restart it:
```
tb local restart
```
## Services (https://www.tinybird.co/docs/forward/core-concepts/tinybird-local#services)
Tinybird Local starts the services needed to run a Tinybird project locally:
- ClickHouse®: Stores data and runs queries.
- Redis: Stores user and Workspace metadata.
- Kafka consumer: Ingests data from Kafka topics into Data Sources.
- Events API server: Ingests HTTP events into Data Sources.
- Tinybird server: Exposes Tinybird APIs, such as authentication and Endpoints.
- Project server: Exposes your project to Tinybird UI.
- MCP server: Exposes your Workspace as tools for agents.
The local API base URL is:
```
http://localhost:7181/v0/
```
For example, you can write to a local Events API Data Source:
```
curl \
-X POST 'http://localhost:7181/v0/events?name=' \
-H "Authorization: Bearer " \
-d $''
```
## Persist local data (https://www.tinybird.co/docs/forward/core-concepts/tinybird-local#persist-local-data)
By default, local state is tied to the Tinybird Local container. To persist data between development sessions, pass a custom volume path:
```
tb local start --volumes-path
```
`tb local stop` does not remove local data. `tb local restart` recreates the local runtime. To return to an earlier local state, restart with a previous volume path:
```
tb local restart --volumes-path ./tb_previous_snapshot
```
## Run with Docker directly (https://www.tinybird.co/docs/forward/core-concepts/tinybird-local#run-with-docker-directly)
The CLI is the recommended way to start Tinybird Local. If you need to run the container directly, use Docker:
```
docker run --platform linux/amd64 -p 7181:7181 --name tinybird-local -d tinybirdco/tinybird-local:latest
```
To connect the `tb` CLI to a manually started container, create a `.tinyb` file in your project directory:
```
{
"name": ""
}
```
## Run with Docker Compose (https://www.tinybird.co/docs/forward/core-concepts/tinybird-local#run-with-docker-compose)
Use Docker Compose when you want explicit volume mounts for ClickHouse® and metadata storage:
docker-compose.yaml
```
version: "3.8"
services:
tinybird-local:
image: tinybirdco/tinybird-local:latest
container_name: tinybird-local
platform: linux/amd64
ports:
- "7181:7181"
volumes:
- ./data/data:/var/lib/clickhouse
- ./data/metadata:/redis-data
restart: unless-stopped
```
Start it:
```
docker compose up -d
```
The `./data/data` directory stores ClickHouse® data. The `./data/metadata` directory stores internal metadata.
## Local Tokens (https://www.tinybird.co/docs/forward/core-concepts/tinybird-local#local-tokens)
Tinybird Local can generate default user and Workspace Tokens for development and tests:
```
tb local generate-tokens
```
Pass generated Token values when starting Tinybird Local:
```
tb local start --user-token= --workspace-token=
```
See [Tinybird Local Tokens](/docs/forward/core-concepts/static-tokens#tinybird-local-tokens).
## Next steps (https://www.tinybird.co/docs/forward/core-concepts/tinybird-local#next-steps)
- [Local development](/docs/forward/development-workflow/local-development)
- [Cloud Branches](/docs/forward/development-workflow/using-cloud-branches)
- [Test your project](/docs/forward/guides/test-your-project)
- [Tinybird CLI commands](/docs/forward/dev-reference/commands)
---
URL: https://www.tinybird.co/docs/forward/core-concepts/templating-language
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Templating language \| Tinybird"
theme-color: "#171612"
description: "Learn how Tinybird's templating language works in Pipes, Connections, and Data Sources."
inkeep:version: "forward"
---
# Templating language (https://www.tinybird.co/docs/forward/core-concepts/templating-language#templating-language)
Use Tinybird's templating language to make project definitions dynamic while keeping them safe and deployable as code. Use it in SQL nodes to define Query Parameters, in connector settings to reference secrets, and in datafiles to reuse values across environments.
Templates are rendered before Tinybird runs the SQL or creates the resource. The rendered result must be valid SQL or a valid datafile setting.
## Where templates are used (https://www.tinybird.co/docs/forward/core-concepts/templating-language#where-templates-are-used)
You use templates in different places depending on the resource:
- **API Endpoints, SQL API queries, Copy Pipes, and Sink Pipes**: Use typed parameters in SQL, such as `{{DateTime(start_date)}}` or `{{String(country, 'US')}}`.
- **Connection files and connector Data Sources**: Use secrets in settings, such as `{{ tb_secret("KAFKA_KEY") }}`.
- **Conditional SQL**: Use control flow with `{% if %}`, `{% else %}`, `{% for %}`, and `defined()` to include SQL only when a value exists.
In `.pipe` files, start dynamic SQL with `%` so Tinybird parses template parameters:
Parameterized SQL node
```
%
SELECT pathname, count() AS views
FROM page_views
WHERE timestamp >= {{DateTime(start_date)}}
AND timestamp < {{DateTime(end_date)}}
AND country = {{String(country, 'US')}}
GROUP BY pathname
ORDER BY views DESC
LIMIT {{Int32(limit, 10)}}
```
In SDK projects, declare parameters in the `params` object and use the same SQL template syntax inside SQL strings. The `%` prefix is not required:
TypeScript SDK endpoint parameters
```
import { defineEndpoint, node, p } from "@tinybirdco/sdk";
export const topPages = defineEndpoint("top_pages", {
params: {
start_date: p.dateTime(),
end_date: p.dateTime(),
country: p.string().optional("US"),
limit: p.int32().optional(10),
},
nodes: [
node({
name: "top_pages",
sql: `
SELECT pathname, count() AS views
FROM page_views
WHERE timestamp >= {{DateTime(start_date)}}
AND timestamp < {{DateTime(end_date)}}
AND country = {{String(country, 'US')}}
GROUP BY pathname
ORDER BY views DESC
LIMIT {{Int32(limit, 10)}}
`,
}),
],
});
```
## Parameters (https://www.tinybird.co/docs/forward/core-concepts/templating-language#parameters)
Parameters are values supplied when an API Endpoint, SQL API query, on-demand Copy Pipe, or on-demand Sink Pipe runs. Define parameters with type functions so Tinybird validates and casts incoming values before injecting them into SQL.
The basic pattern is:
```
{{(, , description=, required=)}}
```
For example:
```
{{Int32(limit, 10, description="Maximum number of rows", required=False)}}
```
Use defaults for scheduled resources and for optional filters. If a parameter has no default, use `defined()` before referencing it:
Optional filter
```
%
SELECT *
FROM events
WHERE 1
{% if defined(country) %}
AND country = {{String(country)}}
{% end %}
```
Common parameter functions include `String`, `Boolean`, `Date`, `DateTime`, `DateTime64`, `Float32`, `Float64`, integer types such as `Int32` and `UInt64`, and `Array`. Use `column()` when you need to parameterize a column name:
```
ORDER BY {{column(order_by, 'timestamp')}}
```
For the full list of functions and arguments, see [Template functions](/docs/forward/dev-reference/template-functions).
Do not concatenate untyped user input into SQL. Use typed template functions so Tinybird validates values and renders safe SQL literals.
## Settings (https://www.tinybird.co/docs/forward/core-concepts/templating-language#settings)
Settings are datafile instructions that configure resources. They include values like `ENGINE_SORTING_KEY`, `KAFKA_BOOTSTRAP_SERVERS`, `IMPORT_BUCKET_URI`, and `EXPORT_SCHEDULE`.
Settings can use template functions when the value must vary by environment or must stay out of source control:
Connection settings with secrets
```
TYPE kafka
KAFKA_BOOTSTRAP_SERVERS {{ tb_secret("KAFKA_BOOTSTRAP_SERVERS", "localhost:9092") }}
KAFKA_SECURITY_PROTOCOL SASL_SSL
KAFKA_KEY {{ tb_secret("KAFKA_KEY") }}
KAFKA_SECRET {{ tb_secret("KAFKA_SECRET") }}
```
The rendered setting must still be valid for the resource type. For available settings, see the relevant datafile reference:
- [Datasource files](/docs/forward/dev-reference/datafiles/datasource-files)
- [Connection files](/docs/forward/dev-reference/datafiles/connection-files)
- [Pipe files](/docs/forward/dev-reference/datafiles/pipe-files)
## Secrets (https://www.tinybird.co/docs/forward/core-concepts/templating-language#secrets)
Secrets store credentials and environment-specific values outside your source files. In datafiles, read them with `tb_secret()`:
```
S3_ARN {{ tb_secret("S3_ARN") }}
```
In TypeScript and Python SDK projects, use `secret()`:
```
import { secret } from "@tinybirdco/sdk";
const arn = secret("S3_ARN");
```
```
from tinybird_sdk import secret
arn = secret("S3_ARN")
```
Set secrets in each environment where the resource runs:
```
tb secret set KAFKA_KEY
tb --cloud secret set KAFKA_KEY
tb --branch=my_branch secret set KAFKA_KEY
```
You can provide a default value as the second argument to `tb_secret()`. Defaults are useful for local development, but Cloud resources should use real secret values:
```
KAFKA_BOOTSTRAP_SERVERS {{ tb_secret("KAFKA_BOOTSTRAP_SERVERS", "localhost:9092") }}
```
Secrets are replaced when resources are deployed. If you change a secret, redeploy the project for the affected resources to use the new value.
See [tb secret](/docs/forward/dev-reference/commands/tb-secret) to set, list, and remove secrets.
## Control flow (https://www.tinybird.co/docs/forward/core-concepts/templating-language#control-flow)
Tinybird templates support Python-style control flow for conditional SQL. Use `{% if %}`, `{% elif %}`, `{% else %}`, `{% for %}`, and `{% end %}` with escaped Markdoc syntax in Markdown examples.
Conditional SQL
```
%
SELECT *
FROM events
WHERE 1
{% if defined(plan) %}
AND plan = {{String(plan)}}
{% end %}
{% if defined(min_amount) %}
AND amount >= {{Float64(min_amount)}}
{% end %}
```
Use `WHERE 1` before optional `AND` clauses to avoid invalid SQL when no optional filters are rendered.
## Helper functions (https://www.tinybird.co/docs/forward/core-concepts/templating-language#helper-functions)
In addition to typed parameters and control flow, Tinybird includes helper functions for common dynamic SQL patterns:
- `defined(param)`: Check if a parameter exists before rendering SQL that uses it.
- `column(param, default)`: Render a validated column name, often for dynamic `ORDER BY` or `GROUP BY` clauses.
- `columns(param)`: Render multiple column names from a parameter.
- `sql_and(...)`: Build optional `AND` filters from parameterized comparisons, such as `in`, `not_in`, `gt`, `lt`, `gte`, and `lte`.
- `split_to_array(param, default, separator=',')`: Split a parameter into an array.
- `date_diff_in_seconds()`, `date_diff_in_minutes()`, `date_diff_in_hours()`, and `date_diff_in_days()`: Compare dates in template control flow, for example to choose between raw and aggregated Data Sources based on the requested time range.
- `JSON(param, default)`: Parse a JSON parameter so you can iterate over structured filters.
- `max_threads(n)`: Set the maximum number of threads for a query. This is useful for Copy Pipes in local environments with limited resources.
- `error()` and `custom_error()`: Stop rendering and return a custom error when a parameter is invalid or missing.
For example, use `sql_and()` when you need to add several optional filters without manually assembling every `AND` clause:
Optional filters with sql_and
```
%
SELECT *
FROM events
WHERE 1
{% if defined(plan) or defined(countries) %}
AND {{sql_and(
plan__in=Array(plan, 'String', defined=False),
country__in=Array(countries, 'String', defined=False)
)}}
{% end %}
```
See [Template functions](/docs/forward/dev-reference/template-functions) for the full helper reference and examples.
## Scope and limitations (https://www.tinybird.co/docs/forward/core-concepts/templating-language#scope-and-limitations)
- Query parameters are available in Endpoints, SQL API queries, and in on-demand Copy Pipes and Sink Pipes.
- Scheduled Copy Pipes and Sink Pipes can use parameters only when those parameters have defaults, because there is no request-time value.
- Materialized Views don't support query parameters.
- Parameters with the same name cascade through dependent Pipes.
- Fixed parameters from JWTs override URL parameters with the same name.
## Next steps (https://www.tinybird.co/docs/forward/core-concepts/templating-language#next-steps)
- Add request-time filters with [Query parameters](/docs/forward/query-data/query-parameters).
- Publish parameterized [Endpoints](/docs/forward/core-concepts/api-endpoints).
- Store credentials with [tb secret](/docs/forward/dev-reference/commands/tb-secret).
- Review every available [Template function](/docs/forward/dev-reference/template-functions).
---
URL: https://www.tinybird.co/docs/forward/core-concepts/storage-architecture
Last update: 2026-07-31T11:10:11.000Z
Content:
---
title: "Storage Architecture \| Tinybird"
theme-color: "#171612"
description: "How Tinybird Cloud stores and queries data using object storage, local SSD caching, and zero-copy replication."
inkeep:version: "forward"
---
# Tinybird Cloud storage architecture (https://www.tinybird.co/docs/forward/core-concepts/storage-architecture#tinybird-cloud-storage-architecture)
Tinybird Cloud uses compute-storage separation. Tinybird stores all data in object storage, using S3 on AWS and GCS on GCP. Each ClickHouse® replica has a local SSD cache for fast reads. Replicas share data in object storage via zero-copy replication.
## Storage (https://www.tinybird.co/docs/forward/core-concepts/storage-architecture#storage)
Tinybird writes all data to **object storage** (S3 or GCS, depending on your [region](/docs/api-reference#regions-and-endpoints)), and object storage is the source of truth for durability and replication. Writes always go there first; the local SSD cache is only a read-path optimization.
## Local SSD cache (https://www.tinybird.co/docs/forward/core-concepts/storage-architecture#local-ssd-cache)
Each replica has a local SSD that caches data parts fetched from object storage. When the database plans a new query, it checks the local cache first. If the part is cached, it reads at SSD speed. If not, it fetches from object storage and caches the part for future reads.
The cache lives on fast local SSD, with capacity determined by cloud provider and machine type. ClickHouse® manages eviction: when cache pressure increases, older or less frequently accessed parts are removed first.
To protect cache stability, Tinybird applies limits to how much data a single query can write into cache, so large scans don't flush useful working sets. There is also no TTL-based expiration: ClickHouse® data parts are immutable, so cached parts stay valid until they are superseded by merges and naturally evicted.
## Ingestion (https://www.tinybird.co/docs/forward/core-concepts/storage-architecture#ingestion)
Data enters the ClickHouse® cluster through two paths. **Streaming ingest** (Events API, Kafka) goes through the **Gatherer**, which batches and buffers incoming events before writing to ClickHouse®. This reduces small-part fragmentation in object storage and improves merge behavior. **Batch ingest** (S3/GCS connectors, files via the datasources API) writes directly to the cluster.
In both paths, persisted writes end up in object storage.
## Zero-copy replication (https://www.tinybird.co/docs/forward/core-concepts/storage-architecture#zero-copy-replication)
Replicas share the same data in object storage rather than each storing its own copy. This means that adding a replica doesn't require copying data. The new replica syncs metadata and starts caching data locally when reading.
Tinybird maintains zero-copy replication in its [ClickHouse® fork](https://www.tinybird.co/blog/why-we-maintain-a-clickhouse-fork-at-tinybird).
## Query path (https://www.tinybird.co/docs/forward/core-concepts/storage-architecture#query-path)
When a query reaches Tinybird, ClickHouse® determines which data parts it needs, checks the **local SSD cache** first, and fetches any misses from **object storage**, caching those parts locally for subsequent reads.
## Infrastructure options (https://www.tinybird.co/docs/forward/core-concepts/storage-architecture#infrastructure-options)
| | Shared | Dedicated |
| --- | --- | --- |
| Cluster | Shared with other customers | Exclusive cluster |
| Replica management | Managed by Tinybird | [Self-serve](/docs/forward/guides/cluster-management) |
| Cache sizing | Automatic | Configurable via replica size |
| Plans | Free, Developer, SaaS | Enterprise |
See [Developer](/docs/forward/pricing/developer), [SaaS](/docs/forward/pricing/saas), and [Enterprise](/docs/forward/pricing/enterprise) for plan and billing details.
## Related (https://www.tinybird.co/docs/forward/core-concepts/storage-architecture#related)
- [Core concepts](/docs/forward/core-concepts)
- [Pricing plans](/docs/forward/pricing)
- [Enterprise](/docs/forward/pricing/enterprise)
- [Cluster management](/docs/forward/guides/cluster-management)
- [Why Tinybird maintains a ClickHouse® fork](https://www.tinybird.co/blog/why-we-maintain-a-clickhouse-fork-at-tinybird)
- [How Tinybird's storage architecture works: S3, local caching, and zero-copy replication](https://www.tinybird.co/blog/tinybird-architecture)
---
URL: https://www.tinybird.co/docs/forward/core-concepts/static-tokens
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Static tokens \| Tinybird"
theme-color: "#171612"
description: "Static tokens are permanent and long-term."
inkeep:version: "forward"
---
# Static tokens (https://www.tinybird.co/docs/forward/core-concepts/static-tokens#static-tokens)
Static Tokens are permanent and long-term. Tinybird stores them without an expiration date or time. They remain valid until you delete or refresh them. They're useful for backend-to-backend integrations, where you call Tinybird as another service.
## Default tokens (created by Tinybird) (https://www.tinybird.co/docs/forward/core-concepts/static-tokens#default-tokens-created-by-tinybird)
All workspaces come with a set of default tokens:
| Token name | Description |
| --- | --- |
| `Workspace admin token` | The Workspace token. This token is workspace-bound and enables any operation over it. Note: only workspace admins have access to this token. |
| `Admin token` | The CLI token. This token is managed by Tinybird for you and the CLI uses it to authenticate via 'tb login' (stores it locally in the `.tinyb` file). |
| `User token` | Required only for certain operations through the API (like creating Workspaces). The system asks you for it if required. |
Learn how to [list existing Tokens](https://www.tinybird.co/docs/forward/core-concepts/static-tokens#list-existing-tokens).
## User created tokens (https://www.tinybird.co/docs/forward/core-concepts/static-tokens#user-created-tokens)
Users can create additional tokens with different authorization scopes. This allow you to grant granular access to resources or to create tokens for CI/CD or for other purposes.
There are two types of static tokens:
- **[Resource-scoped tokens](https://www.tinybird.co/docs/forward/core-concepts/static-tokens#resource-scoped-tokens):** grant specific permissions on specific resources, such as reading from a given endpoint or appending to a given data source. Created in *.pipe* and *.datasource* files and managed via deployments.
- **[Workspace and Org. level tokens](https://www.tinybird.co/docs/forward/core-concepts/static-tokens#other-tokens):** tokens with workspace or organization-wide scopes: `WORKSPACE:READ_ALL`, `WORKSPACE:DEPLOY`, `ADMIN`, `TOKENS` or `ORG_DATASOURCES:READ`. Created and managed via the CLI or API.
### Resource-scoped tokens (https://www.tinybird.co/docs/forward/core-concepts/static-tokens#resource-scoped-tokens)
When you create a resource-scoped token, you can define which resources can be accessed by that token, and which methods can be used to access them.
In datafile projects, resource-scoped tokens are managed using the `TOKEN` directive with the structure `TOKEN `. In SDK projects, declare reusable token names and attach them to Data Sources, Pipes, or Endpoints with each resource's `tokens` option.
- Tinybird CLI
- TypeScript SDK
- Python SDK
events.datasource
```
TOKEN events_append APPEND
TOKEN events_read READ
SCHEMA >
`timestamp` DateTime `json:$.timestamp`,
`payload` String `json:$.payload`
ENGINE "MergeTree"
ENGINE_SORTING_KEY "timestamp"
```
events_api.pipe
```
TOKEN events_read READ
NODE latest
SQL >
SELECT timestamp, payload FROM events ORDER BY timestamp DESC LIMIT 100
TYPE ENDPOINT
```
Deployments create and update resource-scoped Tokens. Tinybird tracks which ones to create or destroy based on all the Tokens defined within the datafiles in your project. Find deployment-generated Tokens in the Workspace UI or by running `tb (--cloud) token ls`.
The following scopes are available for resource-scoped tokens:
| Token Scope (API) | Token Scope (CLI) | Description |
| --- | --- | --- |
| `DATASOURCES:READ:datasource_name` | `TOKEN READ` in `.datasource` files | Grants the token read permissions on the specified data sources |
| `DATASOURCES:APPEND:datasource_name` | `TOKEN APPEND` in `.datasource` files | Grants the token permission to append data to the specified data source. |
| `PIPES:READ:pipe_name` | `TOKEN APPEND` in `.pipe` files | Grants the token read permissions for the specified pipe. |
When adding the `DATASOURCES:READ` scope to a token, it automatically grants read permissions to the [quarantine data source](/docs/forward/guides/quarantine) associated with it.
SQL filters (`:sql_filter` suffix) are not supported in Tinybird Forward. Use fixed parameters in JWTs for row-level security instead.
### Other tokens (https://www.tinybird.co/docs/forward/core-concepts/static-tokens#other-tokens)
These are operational tokens that are not tied to specific resources. Run the following command in the CLI:
```
tb token create static new_admin_token --scope
```
The following scopes are available for general tokens:
| Value | Description |
| --- | --- |
| `TOKENS` | Grants the token permission to create, delete or refresh tokens. |
| `ADMIN` | Grants full access to the workspace. Use sparingly. |
| `WORKSPACE:READ_ALL` | Grants read access to all workspace resources: datasources, pipes, `tinybird.*` service data sources, and `system.*` tables. Particularly useful for BI Tools. |
| `WORKSPACE:DEPLOY` | Grants permission to run deploy-related operations in the workspace. |
| `ORG_DATASOURCES:READ` | Grants the token read access to organization service datasources. |
## List existing tokens (https://www.tinybird.co/docs/forward/core-concepts/static-tokens#list-existing-tokens)
You can review your existing tokens using:
- **CLI**: Run `tb token ls` to list all tokens in your workspace. See [tb token](/docs/forward/dev-reference/commands/tb-token) for reference.
- **UI**: Navigate to the "Tokens" section in the sidebar of your Tinybird workspace.
## Refresh a static token (https://www.tinybird.co/docs/forward/core-concepts/static-tokens#refresh-a-static-token)
To refresh a token, run the `tb token refresh` command:
```
tb token refresh my_static_token
```
See [tb token](/docs/forward/dev-reference/commands/tb-token) for more information.
## Delete a static token (https://www.tinybird.co/docs/forward/core-concepts/static-tokens#delete-a-static-token)
### Resource-scoped tokens (https://www.tinybird.co/docs/forward/core-concepts/static-tokens#resource-scoped-tokens)
Deployments update resource-scoped Tokens. Tinybird tracks which ones to destroy based on all the Tokens defined within the datafiles in your project.
To remove a resource-scoped Token, **delete it from the datafiles and deploy the project.** Tinybird applies the changes automatically.
### Other tokens (https://www.tinybird.co/docs/forward/core-concepts/static-tokens#other-tokens)
To delete [other tokens](/docs/forward/core-concepts/static-tokens#other-tokens) that are not tied to specific resources, run the following command:
```
tb token rm
```
See [tb token](/docs/forward/dev-reference/commands/tb-token) for more information.
## Tinybird Local tokens (https://www.tinybird.co/docs/forward/core-concepts/static-tokens#tinybird-local-tokens)
When working with [Tinybird Local](/docs/forward/core-concepts/tinybird-local), you can authenticate by running `tb login`:
```
tb login
```
The command opens a browser window where you can sign in. See [tb login](/docs/forward/dev-reference/commands/tb-login).
Credentials are stored in the `.tinyb` file. See [.tinyb file](/docs/forward/dev-reference/datafiles/tinyb-file).
Tinybird Local also supports generating default user and workspace tokens for development and testing. This is especially useful in CI/CD pipelines, tests, or automated setups where dynamically [fetching tokens](/docs/api-reference/token-api) adds unnecessary complexity.
### Generate Tinybird Local tokens (https://www.tinybird.co/docs/forward/core-concepts/static-tokens#generate-tinybird-local-tokens)
You can generate valid Tinybird Local tokens using the following [command](/docs/forward/dev-reference/commands/tb-local):
```
tb local generate-tokens
```
This command outputs valid tokens for both a user token and a workspace token. You can then export them as environment variables:
```
TB_LOCAL_WORKSPACE_TOKEN=$(tb --output=json local generate-tokens | jq -r '.workspace_token')
TB_LOCAL_USER_TOKEN=$(tb --output=json local generate-tokens | jq -r '.user_token')
tb local start
```
Alternatively, you can pass the generated token values in the arguments to `tb local start`. The `generate-tokens` command prints the tokens to your console; you must copy these values to use in the start command:
```
tb local generate-tokens
tb local start --user-token= --workspace-token=
```
Once Tinybird Local has started, you can reference `$TB_LOCAL_USER_TOKEN` and `$TB_LOCAL_WORKSPACE_TOKEN` as environment variables in your API calls or scripts from that shell session.
---
URL: https://www.tinybird.co/docs/forward/core-concepts/sinks
Last update: 2026-06-02T11:48:14.000Z
Content:
---
title: "Sinks \| Tinybird"
theme-color: "#171612"
description: "Sinks are the destinations for your data. They are the places where you can store your data after it has been transformed."
inkeep:version: "forward"
---
# Sinks (https://www.tinybird.co/docs/forward/core-concepts/sinks#sinks)
Tinybird Sinks allow you to export data from your Tinybird Workspace to external systems on a scheduled or on-demand basis. Sinks are built on top of [Pipes](/docs/forward/core-concepts/pipes) and provide a fully managed way to push data to various destinations.
## Available Sinks (https://www.tinybird.co/docs/forward/core-concepts/sinks#available-sinks)
Tinybird supports the following Sink destinations:
- [Kafka Sink](/docs/forward/copy-export-data/kafka-sink)
- [S3 Sink](/docs/forward/copy-export-data/s3-sink)
- [GCS Sink](/docs/forward/copy-export-data/gcs-sink)
Sinks are available on the Developer and Enterprise plans. See [Plans](/docs/forward/pricing).
## Create a Sink (https://www.tinybird.co/docs/forward/core-concepts/sinks#create-a-sink)
Sinks are defined with a connection and a pipe that exports query results to that connection. Use `TYPE SINK` in `.pipe` files, `defineSinkPipe` in the TypeScript SDK, or `define_sink_pipe` in the Python SDK.
The following example creates an S3 Sink that exports processed events to an S3 bucket on a schedule.
- Tinybird CLI
- TypeScript SDK
- Python SDK
connections/landing_s3.connection
```
TYPE s3
S3_REGION "us-east-1"
S3_ARN {{ tb_secret("AWS_ROLE_ARN") }}
```
datasources/events.datasource
```
SCHEMA >
`timestamp` DateTime,
`session_id` String,
`status` String
ENGINE "MergeTree"
ENGINE_SORTING_KEY "timestamp"
```
pipes/events_export_sink.pipe
```
NODE export
SQL >
SELECT timestamp, session_id
FROM events
WHERE status = 'processed'
TYPE SINK
EXPORT_CONNECTION_NAME "landing_s3"
EXPORT_BUCKET_URI "s3://my-bucket/exports/"
EXPORT_FILE_TEMPLATE "events_{date}"
EXPORT_FORMAT "csv"
EXPORT_SCHEDULE "0 * * * *"
```
See all syntax options in the [Sink pipes reference](/docs/forward/dev-reference/datafiles/pipe-files#sink-pipes), [TypeScript SDK reference](/docs/forward/dev-reference/typescript-sdk-resources#sink-pipes), and [Python SDK reference](/docs/forward/dev-reference/python-sdk-resources#sink-pipes).
## Key features (https://www.tinybird.co/docs/forward/core-concepts/sinks#key-features)
- Fully Managed: Sinks require no additional tooling or infrastructure management.
- Scheduled or On-Demand: Run exports on a defined schedule using cron expressions or trigger them manually when needed.
- Query Parameters: Support for parameterized queries allows flexible data filtering and transformation.
- Observability: Monitor Sink operations and data transfer through Service Data Sources.
## Common use cases (https://www.tinybird.co/docs/forward/core-concepts/sinks#common-use-cases)
Sinks enable various data integration scenarios:
- Regular data exports to clients or partner systems.
- Feeding data lakes and data warehouses.
- Real-time data synchronization with external systems.
- Event-driven architectures and data pipelines.
---
URL: https://www.tinybird.co/docs/forward/core-concepts/shared-vs-dedicated-infrastructure
Last update: 2026-07-31T11:10:11.000Z
Content:
---
title: "Shared vs dedicated infrastructure \| Tinybird"
theme-color: "#171612"
description: "Understand the main differences between shared and dedicated Tinybird infrastructure."
inkeep:version: "forward"
---
# Shared vs dedicated infrastructure (https://www.tinybird.co/docs/forward/core-concepts/shared-vs-dedicated-infrastructure#shared-vs-dedicated-infrastructure)
Tinybird runs Workspaces on either shared infrastructure or dedicated infrastructure. Both give you managed ClickHouse®, ingestion, APIs, and Tinybird tooling. The difference is how much infrastructure isolation and operational control you need.
This maps to the familiar multi-tenant versus single-tenant trade-off: shared infrastructure is multi-tenant, while dedicated infrastructure gives your Organization isolated compute.
## Shared infrastructure (https://www.tinybird.co/docs/forward/core-concepts/shared-vs-dedicated-infrastructure#shared-infrastructure)
On shared infrastructure, your Workspace runs on infrastructure shared with other Tinybird customers. Tinybird manages the cluster topology, ClickHouse® version, upgrades, and operational limits.
Shared infrastructure is a good fit when you want a managed experience with less operational overhead:
- You co-live on shared clusters managed by Tinybird.
- You resize vertically by choosing a larger plan size, up to the limits of shared infrastructure.
- You don't manage replicas, traffic weights, or cluster topology.
- Tinybird manages ClickHouse® versions and upgrade timing.
- Limits and safeguards protect shared resources from noisy-neighbor effects.
For billing details, see [Developer](/docs/forward/pricing/developer), [SaaS](/docs/forward/pricing/saas), and [Free](/docs/forward/pricing/free).
## Dedicated infrastructure (https://www.tinybird.co/docs/forward/core-concepts/shared-vs-dedicated-infrastructure#dedicated-infrastructure)
On dedicated infrastructure, your organization gets an isolated ClickHouse® cluster. Your ingestion and query workloads run on replicas dedicated to you.
Dedicated infrastructure is a good fit when you need more control, isolation, or scale:
- Your workload runs on dedicated replicas, not shared compute.
- You can add, remove, resize, and rebalance replicas.
- You can control how read, write, and copy workloads are distributed across replicas.
- You have more control over ClickHouse® versioning and when upgrades happen.
- You can tune topology and capacity for workload isolation, high throughput, or stricter production requirements.
For billing details, see [Enterprise](/docs/forward/pricing/enterprise). For operational controls, see [Cluster management](/docs/forward/guides/cluster-management).
## Which one should you use? (https://www.tinybird.co/docs/forward/core-concepts/shared-vs-dedicated-infrastructure#which-one-should-you-use)
Start with shared infrastructure if you want the simplest managed path and your workload fits within shared plan limits. Move to dedicated infrastructure when you need stronger isolation, higher scale, custom replica topology, or more control over operational changes like upgrades.
---
URL: https://www.tinybird.co/docs/forward/core-concepts/self-managed
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Self-managed \| Tinybird"
theme-color: "#171612"
description: "Run Tinybird infrastructure in your own cloud environment with self-managed regions."
inkeep:version: "forward"
---
# Self-managed (https://www.tinybird.co/docs/forward/core-concepts/self-managed#self-managed)
By default, Tinybird runs on [Tinybird managed regions](/docs/api-reference#regions-and-endpoints). Self-managed regions let you run Tinybird infrastructure in your own cloud environment while Tinybird Cloud manages the control plane. Use them when you need control over data residency, compliance requirements, or infrastructure costs.
Self-managed regions are currently in beta. Use them for development environments only. Avoid production workloads. Features, requirements, and implementation details might change as Tinybird improves this capability.
## Key benefits (https://www.tinybird.co/docs/forward/core-concepts/self-managed#key-benefits)
Key benefits of self-managed regions include:
- Data sovereignty and compliance: Keep your data in specific geographic locations to meet regulatory requirements.
- Integration: Connect directly to your existing data sources and internal systems without data leaving your network.
- Infrastructure control: Tune hardware resources for your workload patterns.
- Cost efficiency: Avoid egress fees and use your existing cloud commitments and reserved instances.
## Limitations (https://www.tinybird.co/docs/forward/core-concepts/self-managed#limitations)
Self-managed regions have some limitations compared to Tinybird managed regions:
- You are responsible for infrastructure maintenance, scaling, and security.
- Some advanced features may require additional configuration.
- Performance tuning and optimization are your responsibility.
- Upgrades must be managed manually by redeploying with newer container versions.
## How self-managed regions work (https://www.tinybird.co/docs/forward/core-concepts/self-managed#how-self-managed-regions-work)
When you set up a self-managed region:
1. You register a region with Tinybird Cloud.
2. You deploy the Tinybird Local container in your cloud environment.
3. You interact with your self-managed region directly using the Tinybird CLI with the `--cloud` flag.
This approach keeps your data and infrastructure in your cloud environment while Tinybird Cloud manages the region metadata.
## Add a self-managed region (https://www.tinybird.co/docs/forward/core-concepts/self-managed#add-a-self-managed-region)
There are two main ways to create a self-managed region:
- **Manually**: See [Create a self-managed region manually](/docs/forward/guides/add-self-managed-region-manually).
- **Using the CLI**: See [Use the CLI to add a self-managed region](/docs/forward/guides/add-self-managed-region-cli).
## Supported clouds (https://www.tinybird.co/docs/forward/core-concepts/self-managed#supported-clouds)
The following cloud service providers are supported:
- AWS
- GCP (Coming soon)
- Azure (Coming soon)
You can also create a self-managed region by deploying the container on the following Platform as a Service (PaaS) providers:
- Fly.io
- Zeet
- Google Cloud Run
- CloudFoundry
- Azure Container Instances
- DigitalOcean App Platform
When deploying Tinybird Local on a PaaS, make sure to expose the following environment variables to the container:
- `TB_INFRA_TOKEN`
- `TB_INFRA_WORKSPACE`
- `TB_INFRA_ORGANIZATION`
- `TB_INFRA_USER`
See [tb infra](/docs/forward/dev-reference/commands/tb-infra) for more information.
---
URL: https://www.tinybird.co/docs/forward/core-concepts/pipes
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Pipes \| Tinybird"
theme-color: "#171612"
description: "Pipes help you to bring your SQL queries together to achieve a purpose, like publishing an endpoint."
inkeep:version: "forward"
---
# Pipes (https://www.tinybird.co/docs/forward/core-concepts/pipes#pipes)
A pipe is a collection of one or more SQL queries. Each query is called a node.
Use Pipes to build features over your data. For example, you can write SQL that joins, aggregates, or otherwise transforms your data. Pipes may be internal, which means your project references them, or they can have outputs such as API Endpoints, Sinks, Materialized Views, and Copy Jobs.
## Nodes (https://www.tinybird.co/docs/forward/core-concepts/pipes#nodes)
A node is a container for a single SQL `SELECT` statement. Nodes live within pipes, and you can have many sequential nodes inside the same pipe. They allow you to break your query logic down into multiple smaller queries. You can then chain nodes together to build the logic incrementally.
A query in a node can read data from a data source, other nodes inside the same pipe, or from endpoint nodes in other pipes. Each node can be developed and tested individually. This makes it much easier to build complex query logic in Tinybird as you avoid creating large monolithic queries with many subqueries.
Pipes are defined with `.pipe` files in regular projects, in `.ts` files with the TypeScript SDK, or in `.py` files with the Python SDK.
SQL nodes can use Tinybird's [templating language](/docs/forward/core-concepts/templating-language) for typed parameters, conditional SQL, and environment-specific values.
- Tinybird CLI
- TypeScript SDK
- Python SDK
pipes/events_by_path.pipe
```
DESCRIPTION >
Count events by path for a date range
NODE recent_events
SQL >
%
SELECT timestamp, pathname
FROM events
WHERE timestamp >= {{DateTime(start_date)}}
AND timestamp <= {{DateTime(end_date)}}
NODE aggregation
SQL >
SELECT pathname, count() AS events
FROM recent_events
GROUP BY pathname
ORDER BY events DESC
```
See all syntax options in the [Pipe files reference](/docs/forward/dev-reference/datafiles/pipe-files), [TypeScript SDK reference](/docs/forward/dev-reference/typescript-sdk-resources#pipes), and [Python SDK reference](/docs/forward/dev-reference/python-sdk-resources#pipes).
## Create generic pipes (https://www.tinybird.co/docs/forward/core-concepts/pipes#create-generic-pipes)
Generic Pipes are internal Pipes that you use from other Pipes or Endpoints. The preceding examples create generic Pipes because they don't declare an output type such as Endpoint, Materialized View, Copy Job, or Sink.
Use generic pipes to split complex SQL into reusable steps or to replace include-file patterns from Classic projects. To publish the result as an API endpoint, see [Endpoints](/docs/forward/core-concepts/api-endpoints).
## Next steps (https://www.tinybird.co/docs/forward/core-concepts/pipes#next-steps)
- Learn about [Endpoints](/docs/forward/core-concepts/api-endpoints).
- Learn about [Copy pipes](/docs/forward/core-concepts/copy-pipes).
- Learn about [Materialized views](/docs/forward/core-concepts/materialized-views).
---
URL: https://www.tinybird.co/docs/forward/core-concepts/organizations
Last update: 2026-08-04T10:22:54.000Z
Content:
---
title: "Organizations \| Tinybird"
theme-color: "#171612"
description: "Tinybird Organizations group workspaces, members, billing, limits, and infrastructure settings."
inkeep:version: "forward"
---
# Organizations (https://www.tinybird.co/docs/forward/core-concepts/organizations#organizations)
Organizations group Workspaces, members, billing, limits, and infrastructure settings in one region. Every [Workspace](/docs/forward/core-concepts/workspaces) belongs to an Organization.
Use Organizations to manage access across Workspaces, monitor usage, review billing, and control dedicated infrastructure settings when your plan includes them. See [Plans](/docs/forward/pricing).
The **Organization settings** section consists of the following areas:
- Overview
- Cluster (dedicated infrastructure only)
- Observability
- Logs
- Workspaces
- Members
- Billing (organization admins only)
- Managed regions
## Users and organizations (https://www.tinybird.co/docs/forward/core-concepts/organizations#users-and-organizations)
Users can be members of one or more Organizations. A user can only be the admin of one Organization.
## Access the organization settings (https://www.tinybird.co/docs/forward/core-concepts/organizations#access-the-organization-settings)
To access **Organization settings**, select the organization name in the page header. You can also select **Organization settings** from any **Settings** page.
## Overview (https://www.tinybird.co/docs/forward/core-concepts/organizations#overview)
The **Overview** page is the Organization landing page. It gives you a snapshot of your Organization's health and usage at a glance.
The page is organized in three sections:
- **Recently visited workspaces**: shows the Workspaces you visited most recently. Up to 4 Workspaces are shown. Select **View all workspaces** to see all Workspaces in the Organization.
- **Usage metrics**: 7-day usage cards with maximum aggregation for CPU, API vCPU, Memory, and other metrics depending on your plan.
- **Errors**: a 7-day errors chart with the 5 most recent error log entries. Select the chart to go to the [Logs](https://www.tinybird.co/docs/forward/core-concepts/organizations#logs) page filtered by errors.
The **Usage metrics** section varies by plan:
- **Free, Developer, and SaaS**: 7-day usage cards with maximum values.
- **Enterprise**: per-cluster Load and Memory indicators with one row per replica. You can toggle between **Now**, **6 h avg**, and **24 h avg** views.
## Cluster (https://www.tinybird.co/docs/forward/core-concepts/organizations#cluster)
The **Cluster** section is only available for Organizations with dedicated infrastructure.
The **Cluster** page is the central place to monitor and manage your dedicated cluster. It includes:
- **Load and memory charts** showing cluster resource usage over time.
- **Replica information** with the status and metrics for each replica in your cluster.
- **Manage Cluster** flow to adjust your cluster configuration. Only Organization admins can perform this action.
For replica operations, see [Cluster management](/docs/forward/guides/cluster-management).
## Observability (https://www.tinybird.co/docs/forward/core-concepts/organizations#observability)
The **Observability** page shows details about your resource usage.
### Charts view (https://www.tinybird.co/docs/forward/core-concepts/organizations#charts-view)
The following charts are available depending on your plan:
| Plan | Charts |
| --- | --- |
| Free | Max/Average vCPU time, Max/Average API vCPU time, Max/Average memory, Storage, Accumulated daily requests, Total errors |
| Developer | Total errors, Max/Average vCPU time, Max/Average API vCPU time, Max/Average memory, Requests, Storage, Ingested rows, Quarantine rows, Data Transfer |
| SaaS | Max/Average vCPU time, Max/Average QPS, Max/Average memory, Requests, Storage, Data Transfer, Total errors |
| Enterprise (Dedicated infrastructure) | Max load, Max memory, Requests, Storage, Data Transfer, Total errors |
You can select between average and maximum values for all usage charts, as well as the period, from **Last hour** to **Last 7 days**.
### Table view (https://www.tinybird.co/docs/forward/core-concepts/organizations#table-view)
Use the view toggle to switch between charts and table view.
The table view shows resource usage by Workspace or resource. You can filter by resource name by typing in the text box.
## Logs (https://www.tinybird.co/docs/forward/core-concepts/organizations#logs)
The **Logs** page is a cross-workspace logs explorer. It surfaces logs from all Workspaces in your Organization in one place, with the same filters and layout as the per-workspace logs page.
The page includes:
- A **chart** at the top showing log volume over time. Drag to zoom in on a time range.
- A **filter sidebar** with time range presets and a custom range picker, plus filters for errors, status codes, log types, sources, Workspace, and a text search.
- A **paginated table** showing log entries. Select a row to open a side panel with full log details.
- A **real-time toggle** that refreshes the view every 5 seconds.
With the **Workspace filter**, you can quickly narrow the results to Workspaces that have logs in the current time range, ranked by log count.
Filter state is reflected in the URL, so you can share links to a specific view with colleagues.
## Workspaces (https://www.tinybird.co/docs/forward/core-concepts/organizations#workspaces)
The **Workspaces** page shows details of all your Workspaces, including their name, members, and resources used by each Workspace.
New Workspaces created by a user with an email domain linked to, or matching an Organization are automatically added to that Organization.
## Members (https://www.tinybird.co/docs/forward/core-concepts/organizations#members)
The **Members** page shows details of your organization members, the workspaces they belong to, and their roles. From this page you can manage existing members, their permissions, or invite new members.
The table shows the following information:
- email
- Workspaces and roles
To view the detail of a member’s workspaces and roles, select the arrow next to the Workspace count. A menu shows all the workspaces that user is part of, plus their role in each Workspace.
To change a user’s role or remove them from a Workspace, hover over the Workspace name and follow the arrow. Select a new role from **Admin** or **Viewer**, or remove them from the Workspace. You don't need to be a user in that Workspace to make changes to its users.
### Add or remove members (https://www.tinybird.co/docs/forward/core-concepts/organizations#add-or-remove-members)
To add a user, select **Add member**.
To remove a user from the Organization, select **Remove member** in the menu.
All Organization members can access the **Members** page. Members can only see users from Workspaces they belong to and cannot manage other members. Only Organization administrators can invite or remove members and change roles.
### Add an organization admin (https://www.tinybird.co/docs/forward/core-concepts/organizations#add-an-organization-admin)
To add another user as an organization admin, follow these steps:
1. Navigate to the **Organization settings** page.
2. Go to the **Members** section.
3. Locate the user you want to make an admin.
4. Select the **More actions (⋯)** icon.
5. Select **Organization admin** in the menu.
This grants organization admin access to the selected users.
## Billing (https://www.tinybird.co/docs/forward/core-concepts/organizations#billing)
Only organization administrators can access the **Billing** section.
The **Billing** page contains a summary of the credits balance for your plan, with links to billing details and a summary of your plan. See [Billing](/docs/forward/pricing).
From the **Billing** page you can upgrade or resize your plan, or cancel it if you wish to downgrade to Free.
If you're on a Developer plan, the usage diagram shows the total expenditure by monthly invoice.
## Managed regions (https://www.tinybird.co/docs/forward/core-concepts/organizations#managed-regions)
The **Managed regions** page shows the self-managed regions where your Organization has Workspaces. See [Self-managed](/docs/forward/core-concepts/self-managed).
### Add a region (https://www.tinybird.co/docs/forward/core-concepts/organizations#add-a-region)
To add a region, select **Add region**.
### Remove a region (https://www.tinybird.co/docs/forward/core-concepts/organizations#remove-a-region)
To remove a region, select the **More actions (⋯)** and then select **Delete**.
---
URL: https://www.tinybird.co/docs/forward/core-concepts/materialized-views
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Materialized views \| Tinybird"
theme-color: "#171612"
description: "Materialized views process your data at ingest time to increase the performance of your queries and endpoints."
inkeep:version: "forward"
---
# Materialized views (https://www.tinybird.co/docs/forward/core-concepts/materialized-views#materialized-views)
A materialized view is the continuous, streaming result of a pipe saved as a new data source. As new data is ingested into the origin data source, the transformed results from the pipe are continually inserted in the new materialized view, which you can query as any other data source.
Processing data at ingest time reduces latency and cost-per-query, and can significantly improve the performance of your queries. If your Endpoint, MCP, or regular query usually needs calculations such as counts, sums, averages, or arrays, or transformations like string manipulations or joins, materializations are the best optimization. Materialize the operation and use the resulting Materialized View as a Data Source for the published query.
Typical use cases of Materialized views include:
- Improving the speed of a query that's taking too much time to run.
- Aggregating, sorting, or filtering data at ingest time.
- Simplifying query development by automating common filters and aggregations.
- Changing an existing schema for a different use case.
## Create materialized views (https://www.tinybird.co/docs/forward/core-concepts/materialized-views#create-materialized-views)
Materialized views are defined with a target Data Source and a pipe that writes into it. Use the `TYPE MATERIALIZED` directive in `.pipe` files, `defineMaterializedView` in the TypeScript SDK, or `define_materialized_view` in the Python SDK.
The following example starts with a `page_views` Data Source and creates a `daily_page_views` Data Source populated by a materialized view.
When you define a Materialized View target Data Source with an SDK, turn off JSON paths on the target. The target receives rows from the Materialized View, not raw JSON events from the Events API.
- Tinybird CLI
- TypeScript SDK
- Python SDK
datasources/page_views.datasource
```
SCHEMA >
`timestamp` DateTime,
`pathname` String,
`session_id` String
ENGINE "MergeTree"
ENGINE_SORTING_KEY "pathname, timestamp"
```
datasources/daily_page_views.datasource
```
SCHEMA >
`date` Date,
`pathname` String,
`views` SimpleAggregateFunction(sum, UInt64)
ENGINE "AggregatingMergeTree"
ENGINE_SORTING_KEY "date, pathname"
```
pipes/daily_page_views_mv.pipe
```
NODE aggregate
SQL >
SELECT
toDate(timestamp) AS date,
pathname,
count() AS views
FROM page_views
GROUP BY date, pathname
TYPE materialized
DATASOURCE daily_page_views
```
See all syntax options in the [Materialized pipes reference](/docs/forward/dev-reference/datafiles/pipe-files#materialized-pipes), [TypeScript SDK reference](/docs/forward/dev-reference/typescript-sdk-resources#materialized-views), and [Python SDK reference](/docs/forward/dev-reference/python-sdk-resources#materialized-views).
Once you have the origin and destination data sources and the materialized pipe defined, deploy the project to apply the changes. Materialized views are populated automatically during deployment process.
The order of columns in your transformation Pipe's SELECT statement doesn't need to match the order defined in the destination Data Source schema. Tinybird maps columns by name, not by position. As long as the column names and data types are compatible, the materialization works correctly.
## Populating materialized views (https://www.tinybird.co/docs/forward/core-concepts/materialized-views#populating-materialized-views)
Materialized Views destinations receive data on each insert on the origin data source. A populate is the process to move the data that was already in the origin data source to the destination data source.
Materialized views are populated automatically when you deploy your project. The deployment process handles the initial population and subsequent data migrations.
When you first deploy a project with a Materialized View, Tinybird creates the view and populates it with all existing data from the origin Data Source. Tinybird's deployment system manages this process automatically.
## Changing a materialized view (https://www.tinybird.co/docs/forward/core-concepts/materialized-views#changing-a-materialized-view)
If you need to make changes to a Materialized View after deployment (such as changing its schema or engine settings), follow the Data Source evolution process. This might require using a [forward query](/docs/forward/guides/evolve-data-source#forward-query) to ensure historical data is properly migrated.
Learn more about:
- [Deployments in Tinybird](/docs/forward/core-concepts/deployments) to understand how deployments manage state changes
- [Evolving data sources](/docs/forward/guides/evolve-data-source#materialized-data-sources) for details on making changes to materialized views after deployment
### Altering materialized views (https://www.tinybird.co/docs/forward/core-concepts/materialized-views#altering-materialized-views)
When you need to update the query logic of an existing materialized view without recreating the entire table, you can use the `DEPLOYMENT_METHOD alter` directive. This approach uses `ALTER TABLE ... MODIFY QUERY` instead of dropping and recreating the materialized view, which minimizes data movement and improves deployment performance.
#### When to use `ALTER` (https://www.tinybird.co/docs/forward/core-concepts/materialized-views#when-to-use)
Use the `DEPLOYMENT_METHOD alter` directive when you want to:
- Update the query logic of an existing materialized view
- Preserve existing data while modifying the transformation
- Minimize deployment time and resource usage
- Avoid data movement in production environments with large datasets
#### Syntax (https://www.tinybird.co/docs/forward/core-concepts/materialized-views#syntax)
Add the `DEPLOYMENT_METHOD alter` directive to your materialized pipe file:
my_transformation.pipe
```
NODE transformation_node
SQL >
SELECT
id,
local_date,
name,
sum(count) as total_count
FROM
my_origin
WHERE
local_date >= today() - interval 90 day -- Updated filter from 30 to 90 days
GROUP BY
id,
local_date,
name
TYPE materialized
DATASOURCE my_destination
DEPLOYMENT_METHOD alter
```
#### Limitations of `ALTER` usage (https://www.tinybird.co/docs/forward/core-concepts/materialized-views#limitations-of-usage)
- **Existing materialized views only**: Can only be used on materialized views that already exist in the target environment
- **Query changes only**: The directive only works when there are actual changes to the SQL query
- **No target changes**: Cannot change the target data source (`DATASOURCE` directive)
- **Deployment limitations**: Staging ingestion and reads are not available during ALTER deployments
## Limitations (https://www.tinybird.co/docs/forward/core-concepts/materialized-views#limitations)
As explained, materialized views work as insert triggers, which means a delete or truncate operation on your original data source doesn't affect the related materialized views. But be sure to understand [materialized views iterations](/docs/forward/guides/evolve-data-source#materialized-data-sources) before deleting data or setting TTLs to the origin data source.
As transformation and ingestion in the materialization is done on each block of inserted data in the original data source, some operations such as `GROUP BY`, `ORDER BY`, `DISTINCT` and `LIMIT` might need a specific `engine`, such as `AggregatingMergeTree` or `SummingMergeTree`, which can handle data aggregations.
The data source resulting from a materialization generated using `JOIN` is automatically updated **only** if and when a new operation is performed over the data source in the `FROM`.
You can't create Materialized Views that depend on the `UNION` of several Data Sources.
## Error messages (https://www.tinybird.co/docs/forward/core-concepts/materialized-views#error-messages)
### The performance of this query is not compatible with realtime ingestion (https://www.tinybird.co/docs/forward/core-concepts/materialized-views#the-performance-of-this-query-is-not-compatible-with-realtime-ingestion)
When you deploy a Materialized View, Tinybird validates that the transformation query runs fast enough to keep up with real-time ingestion. The validation works as follows:
1. Tinybird replaces the source Data Source in your query with a subquery that reads a limited batch of rows (2,000 by default).
2. It executes the batch query with tight resource constraints: a 1-second time limit and 2 threads.
3. If the query times out, Tinybird retries once. If it times out again, deployment fails with this error.
This check is a proactive guardrail. Rather than letting a slow Materialized View degrade your ingestion pipeline at runtime, Tinybird rejects it at deploy time.
**Common causes and fixes:**
- **Complex JOINs**: Joins against large Data Sources are expensive to run per batch. Pre-aggregate the joined Data Source or restructure your query.
- **Heavy aggregations or subqueries**: Simplify the transformation logic. If the query is complex, break it into multiple chained Materialized Views.
- **Expensive functions**: Some ClickHouse® functions are slow at scale, such as complex regular expressions or array operations on large arrays. Use simpler alternatives where possible.
- **Too many columns**: Selecting more columns than you need increases the work per batch. Select only the columns you need instead of `SELECT *`.
## Next steps (https://www.tinybird.co/docs/forward/core-concepts/materialized-views#next-steps)
- Learn about [Copy pipes](/docs/forward/core-concepts/copy-pipes).
- Publish your data with [API Endpoints](/docs/forward/core-concepts/api-endpoints).
---
URL: https://www.tinybird.co/docs/forward/core-concepts/jwt
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "JSON Web tokens (JWTs) \| Tinybird"
theme-color: "#171612"
description: "JWTs are signed tokens that allow you to securely authorize and share data between your application and Tinybird."
inkeep:version: "forward"
---
# JSON Web Tokens (JWTs) (https://www.tinybird.co/docs/forward/core-concepts/jwt#json-web-tokens-jwts)
JWTs are signed tokens that allow you to securely and independently authorize and consume data from Tinybird.
Unlike static tokens, **JWTs are not stored in Tinybird**. You create them inside your app and sign them with a shared secret between your app and Tinybird. Tinybird validates the signature of the JWT, using the shared secret, to ensure it's authentic.
JWTs are not defined as Tinybird datafiles or SDK resources. If you use the TypeScript or Python SDKs for your Tinybird project, generate JWTs in your app backend. You can self-sign them with a JWT library and a signing key such as `TINYBIRD_SIGNING_KEY`, or use the SDK token helpers to ask Tinybird to create the JWT through the Token API.
A great use case for JWTs is when you want to allow your app to call Tinybird API endpoints directly from the browser without proxying through your backend.
The typical pattern looks like this:
1. A user starts a session in your app.
2. The frontend requests a JWT from your backend.
3. Your backend generates a new JWT, signed with the Tinybird shared secret, and returns to the frontend.
4. The frontend uses the JWT to call the Tinybird API endpoints directly.
## JWT payload (https://www.tinybird.co/docs/forward/core-concepts/jwt#jwt-payload)
The payload of a JWT is a JSON object that contains the following fields:
| Key | Example Value | Required | Description |
| --- | --- | --- | --- |
| workspace_id | workspaces_id | Yes | The UUID of your Tinybird workspace, found in the workspace list. |
| name | frontend_jwt | Yes | Used to identify the token in the `tinybird.pipe_stats_rt` table, useful for analytics. Doesn't need to be unique. |
| exp | 123123123123 | Yes | The Unix timestamp (UTC) showing the expiry date & time. After a token has expired, Tinybird returns a 403 HTTP status code. |
| scopes | [{"type": "PIPES:READ", "resource": "requests_per_day", "fixed_params": {"org_id": "testing"}}] | Yes | Used to pass data to Tinybird, including the Tinybird scope, resources and fixed parameters. |
| scopes.type | PIPES:READ or DATASOURCES:READ | Yes | The type of scope, for example `PIPES:READ`. See [JWT scopes](https://www.tinybird.co/docs/forward/core-concepts/jwt#jwt-scopes) for supported scopes. |
| scopes.resource | t_b9427fe2bcd543d1a8923d18c094e8c1 or top_airlines | Yes | The ID or name of the pipe that the scope applies to, like which API endpoint the token can access. |
| scopes.fixed_params | {"org_id": "testing"} | No | Valid for scope `PIPES:READ`. Pass arbitrary fixed values to the API endpoint. These values can be accessed by pipe templates to supply dynamic values at query time. |
| scopes.filter | "org_id = 'testing'" | No | Valid for scope `DATASOURCES:READ`. Passes a WHERE filter that Tinybird appends to the specified scope resource. |
| limits | {"rps": 10} | No | You can limit the number of requests per second the JWT can perform. See [JWT rate limit](https://www.tinybird.co/docs/forward/core-concepts/jwt#rate-limits-for-jwts). |
Check out the [JWT example](https://www.tinybird.co/docs/forward/core-concepts/jwt#jwt-example) to see what a complete payload looks like.
## JWT algorithm (https://www.tinybird.co/docs/forward/core-concepts/jwt#jwt-algorithm)
Tinybird always uses HS256 as the algorithm for JWTs and doesn't read the `alg` field in the JWT header. You can skip the `alg` field in the header.
## JWT scopes (https://www.tinybird.co/docs/forward/core-concepts/jwt#jwt-scopes)
| Value | Description |
| --- | --- |
| `PIPES:READ:pipe_name` | Gives your token read permissions for the specified pipe. Use `fixed_params` to filter by the pipe parameters. |
| `DATASOURCES:READ:datasource_name` | Gives your token read permissions for the specified datasource. Use `filter` to filter by the data source columns. |
## JWT expiration (https://www.tinybird.co/docs/forward/core-concepts/jwt#jwt-expiration)
JWTs can have an expiration time that gives each token a finite lifespan.
Setting the `exp` field in the JWT payload is mandatory, and not setting it results in a 403 HTTP status code from Tinybird when requesting the API endpoint.
Tinybird validates that a JWT hasn't expired before allowing access to the API endpoint.
If a token has expired, Tinybird returns a 403 HTTP status code.
## JWT fixed parameters (https://www.tinybird.co/docs/forward/core-concepts/jwt#jwt-fixed-parameters)
Fixed parameters allow you to pass arbitrary values to the API endpoint. These values can be accessed by pipe templates to supply dynamic values at query time. See [Templating language](/docs/forward/core-concepts/templating-language) to learn how parameters are validated and rendered in SQL.
Consider the following API Endpoint that accepts a parameter called `org` that filters by the `org_id` column:
example.pipe
```
SELECT fieldA, fieldB FROM my_ds WHERE org_id = '{{ String(org) }}'
TYPE ENDPOINT
```
The following JWT payload passes a parameter called `org` with the value `test_org` to the API endpoint:
Example fixed parameters
```
{
"type": "PIPES:READ",
"resource": "requests_per_day",
"fixed_params": {
"org": "test_org"
}
}
```
This is particularly useful when you want to pass dynamic values to an API endpoint that are set by your backend and must be safe from user tampering. A good example is multi-tenant applications that require row-level security, where you need to filter data based on a user or tenant ID.
The value for the `org` parameter is always the one specified in the `fixed_params`. Even if you specify a new value in the URL when requesting the endpoint, Tinybird always uses the one specified in the JWT.
## JWT filters (https://www.tinybird.co/docs/forward/core-concepts/jwt#jwt-filters)
Filters allow you to pass WHERE clauses to Data Source queries. The filter must be a valid `WHERE` clause. Tinybird automatically appends it to the Data Source at query time.
Consider the following data source with an `org_id` column:
example.datasource
```
SCHEMA >
`timestamp` DateTime `json:$.timestamp`,
`org_id` String `json:$.org_id`,
`action` LowCardinality(String) `json:$.action`,
`version` LowCardinality(String) `json:$.version`,
`payload` String `json:$.payload`
ENGINE MergeTree
ENGINE_SORTING_KEY org_id, timestamp
```
The following JWT payload passes a filter for the data source:
Example filter in a scope
```
{
"type": "DATASOURCES:READ",
"resource": "events",
"filter": "org_id = 'testing'"
}
```
This is particularly useful when you want to pass dynamic filters to queries. A good example is multi-tenant applications that require row-level security, where you need to filter data based on a user or tenant ID.
## JWT example (https://www.tinybird.co/docs/forward/core-concepts/jwt#jwt-example)
Consider the following payload with all [required and optional fields](https://www.tinybird.co/docs/forward/core-concepts/jwt#jwt-payload):
Example payload
```
{
"workspace_id": "workspaces_id",
"name": "frontend_jwt",
"exp": 123123123123,
"scopes": [
{
"type": "PIPES:READ",
"resource": "requests_per_day",
"fixed_params": {
"org_id": "testing"
}
},
{
"type": "DATASOURCES:READ",
"resource": "events",
"filter": "org_id = 'testing'"
}
],
"limits": {
"rps": 10
}
}
```
Use the workspace admin token as your signing key (`TINYBIRD_SIGNING_KEY`):
Example workspace admin token
```
p.eyJ1IjogIjA1ZDhiYmI0LTdlYjctND...
```
With the payload and admin token, the signed JWT payload would look like this:
Example JWT
```
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ3b3Jrc3BhY2V...
```
## JWT limitations (https://www.tinybird.co/docs/forward/core-concepts/jwt#jwt-limitations)
The following limitations apply to JWTs:
- You can't refresh JWTs individually from inside Tinybird because Tinybird doesn't store them. Refresh them from your app, or globally invalidate all JWTs by refreshing your Admin Token.
- If you refresh your admin token, all the tokens are invalidated.
- If your token expires or is invalidated, you get a 403 HTTP status code from Tinybird when requesting the API endpoint.
## Create a JWT in production (https://www.tinybird.co/docs/forward/core-concepts/jwt#create-a-jwt-in-production)
There is wide support for creating JWTs in many programming languages and frameworks. Any library that supports JWTs should work with Tinybird.
- JavaScript (Next.js)
- Python
Create a JWT in Python using pyjwt
```
import jwt
import datetime
import os
TINYBIRD_SIGNING_KEY = os.getenv('TINYBIRD_SIGNING_KEY')
def generate_jwt():
expiration_time = datetime.datetime.utcnow() + datetime.timedelta(hours=3)
payload = {
"workspace_id": "workspaces_id",
"name": "frontend_jwt",
"exp": expiration_time,
"scopes": [
{
"type": "PIPES:READ",
"resource": "requests_per_day",
"fixed_params": {
"org_id": "testing"
}
},
{
"type": "DATASOURCES:READ",
"resource": "events",
"filter": "org_id = 'testing'"
},
]
}
return jwt.encode(payload, TINYBIRD_SIGNING_KEY, algorithm='HS256')
```
## Create a JWT token with Tinybird (https://www.tinybird.co/docs/forward/core-concepts/jwt#create-a-jwt-token-with-tinybird)
If you don't want to self-sign JWTs, Tinybird can create them through the Token API, the Tinybird CLI, or the TypeScript and Python SDKs. These methods require a workspace token with the `ADMIN` or `TOKENS` scope.
- API
- Tinybird CLI
- TypeScript SDK
- Python SDK
Create a JWT with the Tinybird CLI
```
tb token create jwt my_jwt --ttl 1h --scope PIPES:READ --resource my_pipe --fixed-params "column_name=value" --scope DATASOURCES:READ --resource my_datasource --filter "column_name='value'"
```
## Error handling (https://www.tinybird.co/docs/forward/core-concepts/jwt#error-handling)
There are many reasons why a request might return a `403` status code. When a `403` is received, check the following:
1. Confirm the JWT is valid and hasn't expired. The expiration time is in the `exp` field in the JWT's payload.
2. The generated JWTs can only read Tinybird API endpoints or query data sources. Confirm you're not trying to use the JWT to access other APIs.
3. Confirm the JWT has a scope to read the endpoint or data source you are trying to read.
4. If you generated the JWT outside of Tinybird, without using the API or the CLI, make sure you are using the **workspace** `admin token`, not your personal one.
## Rate limits for JWTs (https://www.tinybird.co/docs/forward/core-concepts/jwt#rate-limits-for-jwts)
Check the [limits page](/docs/forward/pricing/limits) for limits on ingestion, queries, API Endpoints, and more.
When you specify a `limits.rps` field in the payload of the JWT, Tinybird uses the name specified in the payload of the JWT to track the number of requests being done. If the number of requests goes beyond the limit, Tinybird starts rejecting new requests and returns an "HTTP 429 Too Many Requests" error.
The following example shows the tracking of all requests done by `frontend_jwt`. Once you reach 10 requests per second, Tinybird would start rejecting requests:
Example payload with global rate limit
```
{
"workspace_id": "workspaces_id",
"name": "frontend_jwt",
"exp": 123123123123,
"scopes": [
{
"type": "PIPES:READ",
"resource": "requests_per_day",
"fixed_params": {
"org_id": "testing"
}
},
{
"type": "DATASOURCES:READ",
"resource": "events",
"filter": "org_id = 'testing'"
}
],
"limits": {
"rps": 10
}
}
```
If `rps <= 0`, Tinybird ignores the limit and assumes there is no limit.
As the `name` field doesn't have to be unique, all the tokens generated using the `name=frontend_jwt` would be under the same umbrella. This can be useful if you want to have a global limit in one of your apps or components.
If you want to limit for each specific user, you can generate a JWT using the following payload. In this case, you would specify a unique name so the limits only apply to each user:
Example of a payload with isolated rate limit
```
{
"workspace_id": "workspaces_id",
"name": "frontend_jwt_user_",
"exp": 123123123123,
"scopes": [
{
"type": "PIPES:READ",
"resource": "requests_per_day",
"fixed_params": {
"org_id": "testing"
}
},
{
"type": "DATASOURCES:READ",
"resource": "events",
"filter": "org_id = 'testing'"
}
],
"limits": {
"rps": 10
}
}
```
## Next steps (https://www.tinybird.co/docs/forward/core-concepts/jwt#next-steps)
- Learn about [workspaces](/docs/forward/core-concepts/workspaces).
- Learn about [endpoints](/docs/forward/core-concepts/api-endpoints).
- Using [Clerk.com](https://clerk.com/) in your app? [Let Clerk.com create your Tinybird JWTs automatically](https://clerk.com/blog/tinybird-and-clerk).
- Using [Auth0](https://auth0.com/) in your app? [Let Auth0 create your Tinybird JWTs automatically](https://www.tinybird.co/templates/auth0-jwt).
---
URL: https://www.tinybird.co/docs/forward/core-concepts/deployments
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Deployments \| Tinybird"
theme-color: "#171612"
description: "Deploy your data project to Tinybird."
inkeep:version: "forward"
---
# Deployments in Tinybird (https://www.tinybird.co/docs/forward/core-concepts/deployments#deployments-in-tinybird)
Changing state in data infrastructure can be complex. Each state transition must ensure data integrity and consistency.
Tinybird deployments simplify this process by providing robust mechanisms for managing state changes, allowing you to validate and push updates seamlessly while minimizing the risk of data conflicts or loss.
## What is a deployment? (https://www.tinybird.co/docs/forward/core-concepts/deployments#what-is-a-deployment)
Deployments are versions of your project resources and data running on local or cloud infrastructure.
## Types of deployments (https://www.tinybird.co/docs/forward/core-concepts/deployments#types-of-deployments)
There are two types of deployments:
- Staging deployments: Deployments you can use to validate your changes. You access them using the `--staging` flag.
- Live deployments: Deployments that make your changes available to your users.
## Deployment status (https://www.tinybird.co/docs/forward/core-concepts/deployments#deployment-status)
Deployments have the following statuses:
- `In progress`: The deployment is in progress. Use `--wait` to wait for it to finish.
- `Live`: The deployment is active and has been promoted from staging.
- `Staging`: The deployment is active in staging. Use `--staging` to access it.
- `Failed`: The deployment failed. Try `tb deploy --check` to debug the issue.
- `Deleted`: The deployment was deleted as a result of creating new deployments.
## Deployment methods (https://www.tinybird.co/docs/forward/core-concepts/deployments#deployment-methods)
The following deployment methods are available:
- [Manual deployment](/docs/forward/development-workflow/manual-deployment).
- [CI/CD](/docs/forward/development-workflow/cicd).
If your project was initialized with `tb init --dev-mode local` or `tb init --dev-mode branch`, `tb deploy` without explicit environment flags deploys to your cloud main workspace.
## Staging deployments (https://www.tinybird.co/docs/forward/core-concepts/deployments#staging-deployments)
You can write data to, and read data from, a staging deployment before promoting it to live. This is useful when you've made schema changes that might be incompatible with the current live deployment, like adding new columns.
Automatic data source changes that Tinybird applies with `ALTER`, such as adding a column or changing a data source TTL, are applied only when the deployment is promoted to live. Those changes aren't available in staging deployments.
Use staging deployments to validate a deployment before it goes live. For longer-lived development environments that shouldn't block the deployment process, use [Cloud Branches](/docs/forward/core-concepts/branches) instead.
### Writing to staging deployments (https://www.tinybird.co/docs/forward/core-concepts/deployments#writing-to-staging-deployments)
You can use the [Events API](/docs/forward/ingest-data/events-api) to write directly to staging deployments through the `__tb__min_deployment` parameter, which indicates the target deployment ID. For example:
```
curl \
-H "Authorization: Bearer " \
-d '{"date": "2020-04-05 00:05:38", "city": "Chicago", "new_column": "value"}' \
'https:///v0/events?name=events_test&__tb__min_deployment=5'
```
In the example, if the ID of your current live deployment is 4 and you're creating deployment 5, Tinybird ingests the data into staging deployment 5 only. You can:
1. Make schema changes in a staging deployment.
2. Ingest data compatible with the new schema.
3. Validate the changes work as expected.
4. Promote the deployment to live when ready.
Without the parameter, data would be rejected if it doesn't match the schema of the current live deployment.
If you discard the staging deployment instead of promoting it, the live deployment doesn't contain data written only to that staging deployment.
To get the deployment ID, run `tb deployment ls`.
### Reading from staging deployments (https://www.tinybird.co/docs/forward/core-concepts/deployments#reading-from-staging-deployments)
You can query data from a staging deployment using [pipe endpoints](/docs/forward/core-concepts/api-endpoints). To access a staging endpoint, add the `__tb__deployment` parameter to your API request:
```
curl \
-H "Authorization: Bearer " \
'https:///v0/pipes/my_endpoint?__tb__deployment=5'
```
You can:
1. Test your endpoints with the new schema changes.
2. Validate query results before promoting to live.
3. Ensure your app works correctly with the updated data structure.
To get the deployment ID, run `tb deployment ls`.
### Continuous operation (https://www.tinybird.co/docs/forward/core-concepts/deployments#continuous-operation)
Once the deployment is promoted to live, you can continue using the same API calls. In the previous example, calls using the `__tb__min_deployment=5` or `__tb__deployment=5` parameters continue working without interruption. The parameters ensure compatibility both before and after promotion.
For more details on the Events API parameters, see the [Events API Reference](/docs/api-reference/events-api).
## On-demand compute for deployment populates (https://www.tinybird.co/docs/forward/core-concepts/deployments#on-demand-compute-for-deployment-populates)
When a deployment needs to populate Data Sources, such as when creating Materialized Views or evolving schemas, Tinybird can run the populate on dedicated on-demand compute instead of your main workspace infrastructure. This keeps the populate isolated from production query and ingestion workloads.
Tinybird automatically uses on-demand compute for large deployment populates. For pricing, regional rates, and example calculations, see [On-demand CPUs](/docs/forward/pricing#on-demand-cpus).
## Next steps (https://www.tinybird.co/docs/forward/core-concepts/deployments#next-steps)
- See how to [deploy your project manually](/docs/forward/development-workflow/manual-deployment).
- See how to [deploy your project using CI/CD](/docs/forward/development-workflow/cicd).
---
URL: https://www.tinybird.co/docs/forward/core-concepts/data-sources
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Data sources \| Tinybird"
theme-color: "#171612"
description: "Data sources contain all the data you bring into Tinybird, acting like tables in a database."
inkeep:version: "forward"
---
# Data sources (https://www.tinybird.co/docs/forward/core-concepts/data-sources#data-sources)
When you send data to Tinybird, it's stored in a data source. You then write SQL queries to publish API [endpoints](/docs/forward/core-concepts/api-endpoints), use the [ClickHouse® interface](/docs/forward/query-data/clickhouse-interface), or use [MCP](/docs/forward/query-data/mcp) to query the data.
For example, if your event data lives in a Kafka topic, you can create a data source that connects directly to [Kafka](/docs/forward/ingest-data/connectors/kafka) and writes the events to Tinybird. Similarly, you can [send events](/docs/forward/ingest-data/events-api) or data [from a file](/docs/forward/ingest-data/local-file).
There are also intermediate data sources that are the result of [materialization](/docs/forward/core-concepts/materialized-views) or a [copy pipe](/docs/forward/core-concepts/copy-pipes).
Data sources can be defined with `.datasource` files, in `.ts` files with the TypeScript SDK, or in `.py` files using the Python SDK.
- Tinybird CLI
- TypeScript SDK
- Python SDK
sample.datasource
```
SCHEMA >
`timestamp` DateTime `json:$.timestamp`,
`session_id` String `json:$.session_id`,
`action` LowCardinality(String) `json:$.action`,
`version` LowCardinality(String) `json:$.version`,
`payload` String `json:$.payload`
ENGINE "MergeTree"
ENGINE_SORTING_KEY "session_id, timestamp"
ENGINE_TTL "timestamp + toIntervalDay(60)"
```
See all syntax options in the [Datafiles reference](/docs/forward/dev-reference/datafiles/datasource-files), [TypeScript SDK reference](/docs/forward/dev-reference/typescript-sdk-resources), and [Python SDK reference](/docs/forward/dev-reference/python-sdk-resources).
## Create Data Sources (https://www.tinybird.co/docs/forward/core-concepts/data-sources#create-data-sources)
To create a new Data Source, define it in your project and then deploy it.
If you need help generating a `.datasource` file, run `tb datasource create`. The CLI asks which type of Data Source you want to create:
- **Blank**: Generates a `.datasource` file with example columns you can edit.
- **Local file**: Creates a `.datasource` file based on the schema of a local file.
- **Remote URL**: Creates a `.datasource` file based on the schema of a file at a remote URL.
- **Kafka**: Creates a Data Source for a [Kafka connection](/docs/forward/ingest-data/connectors/kafka). If you don't have a Kafka connection yet, create it first; the schema is built from the topic you select.
- **Amazon S3**: Creates a Data Source for an [S3 connection](/docs/forward/ingest-data/connectors/s3). You need an existing connection because the schema is built from the file in the bucket you choose.
- **GCS**: Creates a Data Source for a [Google Cloud Storage connection](/docs/forward/ingest-data/connectors/gcs). You need an existing connection because the schema is built from the file in the bucket you choose.
Run `tb datasource create -h` anytime to see this list in the command help.
To convert `.datasource` files to SDK definitions, use `tinybird migrate` for TypeScript projects or `tinybird migrate` for Python projects.
## Delete Data Sources (https://www.tinybird.co/docs/forward/core-concepts/data-sources#delete-data-sources)
To delete a data source in Tinybird, remove its corresponding `.datasource` file or definition from your project and deploy your changes.
- Tinybird CLI
- TypeScript SDK
- Python SDK
```
tb deploy --allow-destructive-operations
```
The Tinybird CLI and SDK CLIs require the `--allow-destructive-operations` flag to confirm the removal.
This operation permanently removes the Data Source and all its data from your Workspace. Review dependencies such as Pipes or Materialized Views that might rely on the Data Source before deleting it.
## Shared Data Source (https://www.tinybird.co/docs/forward/core-concepts/data-sources#shared-data-source)
Workspace administrators can share a Data Source with another Workspace they have access to in the same Organization.
To share a Data Source, add the destination Workspace names to the Data Source definition and deploy your changes. Use `SHARED_WITH` in `.datasource` files, `sharedWith` in the TypeScript SDK, or `shared_with` in the Python SDK.
- Tinybird CLI
- TypeScript SDK
- Python SDK
origin_datasource.datasource
```
# ... data source definition ...
SHARED_WITH >
,
```
```
tb deploy
```
You can use the shared Data Source to create Pipes in the target Workspace. Users that have access to a shared Data Source can access the `tinybird.datasources_ops_log` and the `tinybird.kafka_ops_log` Service Data Sources.
### Limitations (https://www.tinybird.co/docs/forward/core-concepts/data-sources#limitations)
The following limitations apply to shared Data Sources:
- Shared Data Sources are read-only.
- You can't share a shared Data Source, only the original Workspace can share it.
- You can't check the quarantine of a shared Data Source.
- You can't create a Materialized View from a shared Data Source unless you're migrating from Classic and you already have them.
### Working locally with Shared Data Sources (https://www.tinybird.co/docs/forward/core-concepts/data-sources#working-locally-with-shared-data-sources)
When one Workspace shares a Data Source with another Workspace, deploy order matters. For example, say Workspace A shares a Data Source with Workspace B, and Workspace B uses that Data Source in an endpoint. If you start with a fresh Tinybird Local environment without either Workspace, deploy in this order:
- Tinybird CLI
- TypeScript SDK
- Python SDK
```
# Workspace B: first deploy creates the workspace, but fails because the shared Data Source is not available yet.
tb deploy
# Workspace A: deploy the Data Source that is shared with Workspace B.
tb deploy
# Workspace B: deploy again after the shared Data Source is available.
tb deploy
```
`tb build` for datafile projects, or `tinybird build` for SDK projects, hides all this complexity and creates the necessary workspaces and Data Sources to verify that a workspace is valid.
### Keeping .datasource files up-to-date (https://www.tinybird.co/docs/forward/core-concepts/data-sources#keeping-datasource-files-up-to-date)
For datafile projects, run `tb [--cloud] pull --only-vendored` to update the `.datasource` files of Data Sources shared with your Workspace. The command places them in `vendor//datasources`.
You can only deploy your project if the files in `vendor/` are up-to-date. If they aren't, your the deployment fails and prompts you to run the preceding command.
If you're using the SDKs, run the corresponding `tinybird migrate` command to convert `.datasource` files to `.ts` or `.py` definitions.
## Quarantine Data Sources (https://www.tinybird.co/docs/forward/core-concepts/data-sources#quarantine-data-sources)
Every Data Source you create in your Workspace has an associated quarantine Data Source that stores rows that don't fit the schema. If a row doesn't match the Data Source schema, Tinybird writes it to the quarantine Data Source instead of failing the whole ingest process.
The quarantine Data Source keeps the columns from the original row and adds metadata columns that explain why the row was quarantined:
- `c__error_column` (`Array(String)`): The columns that contain invalid values.
- `c__error` (`Array(String)`): The ingestion errors that caused the row to be quarantined.
- `c__import_id` (`Nullable(String)`): The import job identifier, when the row was imported through a job.
- `insertion_date` (`DateTime`): The timestamp when Tinybird ingested the row.
Quarantine Data Sources are recreated when you deploy a new version of the original Data Source. If you need to inspect, query, or recover quarantined rows, do it before deploying schema changes to the original Data Source.
Use quarantine Data Sources to inspect schema mismatches, recover failed rows, and validate schema changes during ingest workflows. For step-by-step examples, see the [Quarantine Data Sources guide](/docs/forward/guides/quarantine).
## Data operations (https://www.tinybird.co/docs/forward/core-concepts/data-sources#data-operations)
Data Sources are append-only by default. When you need to update or remove data, use explicit data operations instead of treating Data Sources like mutable OLTP tables.
- **Append data** from events, files, or connectors. See [Ingest Data](/docs/forward/ingest-data).
- **Replace data** when you need to reingest a full Data Source or a partitioned slice. See [Replace and delete data](/docs/forward/guides/replace-and-delete-data) and [Replace data from a file](/docs/forward/ingest-data/local-file#replace-data-from-a-file).
- **Delete data** for selective removals or compliance workflows. See [Delete data selectively](/docs/forward/guides/replace-and-delete-data#delete-data-selectively), [GDPR-compliant data deletion](/docs/forward/guides/gdpr-compliant-data-deletion), and [delete limits](/docs/forward/pricing/limits#delete-limits).
- **Deduplicate data** at query time, with table engines, or by rebuilding derived Data Sources depending on your workload. See [Deduplication strategies](/docs/forward/guides/deduplication-strategies). This is often the best pattern when you need update-like behavior in analytical workloads.
For one-off corrections, use replace or delete operations directly. If updates and deletes are frequent, model the workload as append-only events and deduplicate the latest state at query or processing time. This is usually more efficient and easier to operate at scale.
---
URL: https://www.tinybird.co/docs/forward/core-concepts/copy-pipes
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Copy pipes \| Tinybird"
theme-color: "#171612"
description: "Copy pipes capture the result of a pipe at a moment in time and write the result into a target data source. They can be run on a schedule, or executed on demand."
inkeep:version: "forward"
---
# Copy pipes (https://www.tinybird.co/docs/forward/core-concepts/copy-pipes#copy-pipes)
Copy pipes capture the result of a pipe at a moment in time and write the result into a target data source. They can be run on a schedule, or executed on demand.
Copy pipes should not be confused with [materialized views](/docs/forward/core-concepts/materialized-views). While materialized views continuously update as new events are inserted, copy pipes generate a single snapshot at a specific point in time.
## Create a copy pipe (https://www.tinybird.co/docs/forward/core-concepts/copy-pipes#create-a-copy-pipe)
Copy pipes are defined with SQL nodes and a target Data Source. Use `TYPE COPY` in `.pipe` files, `defineCopyPipe` in the TypeScript SDK, or `define_copy_pipe` in the Python SDK.
Define the queries that filter and transform the data as needed. The final result is written into the target Data Source. Before pushing a copy pipe to your workspace, make sure that the target Data Source already exists and has a schema that matches the query output.
The following example creates a copy pipe scheduled to run every hour and write the query result into the `sales_hour_copy` Data Source.
Copy Pipes can use Tinybird's [templating language](/docs/forward/core-concepts/templating-language) in SQL nodes. Scheduled Copy Pipes must provide defaults for parameters because there is no request-time value.
- Tinybird CLI
- TypeScript SDK
- Python SDK
datasources/teams.datasource
```
SCHEMA >
`starting_date` DateTime,
`country` String,
`sales` Float64
ENGINE "MergeTree"
ENGINE_SORTING_KEY "starting_date, country"
```
datasources/sales_hour_copy.datasource
```
SCHEMA >
`day` DateTime,
`country` String,
`total_sales` Float64
ENGINE "MergeTree"
ENGINE_SORTING_KEY "day, country"
```
pipes/sales_hour_copy.pipe
```
NODE daily_sales
SQL >
%
SELECT toStartOfDay(starting_date) day, country, sum(sales) as total_sales
FROM teams
WHERE
day BETWEEN toStartOfDay({{DateTime(job_timestamp)}}) - interval 1 day AND toStartOfDay({{DateTime(job_timestamp)}})
and country = {{ String(country, 'US')}}
GROUP BY day, country
TYPE COPY
TARGET_DATASOURCE sales_hour_copy
COPY_SCHEDULE 0 * * * *
COPY_MODE append
```
See all syntax options in the [Copy pipes reference](/docs/forward/dev-reference/datafiles/pipe-files#copy-pipes), [TypeScript SDK reference](/docs/forward/dev-reference/typescript-sdk-resources#copy-pipes), and [Python SDK reference](/docs/forward/dev-reference/python-sdk-resources#copy-pipes).
## Schedule a copy pipe (https://www.tinybird.co/docs/forward/core-concepts/copy-pipes#schedule-a-copy-pipe)
You can schedule copy pipes to run at a specific time using a cron expression. To schedule a copy pipe, configure `COPY_SCHEDULE` with a cron expression. On-demand copy pipes are defined by configuring `COPY_SCHEDULE` with the value `@on-demand`.
Copy Pipes scheduled with a cron expression have a maximum timeout defined by the lower of the limit enforced by your plan and half the interval of your cron expression. For example, a Copy Pipe scheduled to run every 5 minutes (300 seconds) has a maximum timeout of 2.5 minutes (150 seconds), even if your plan permits an hour. This prevents your Copy Pipes from overlapping.
All schedules use the UTC time zone. When you configure a schedule that runs at a specific time, convert the desired time from your local time zone to UTC.
## Run, pause, or resume a copy pipe (https://www.tinybird.co/docs/forward/core-concepts/copy-pipes#run-pause-or-resume-a-copy-pipe)
Use the `tb copy` command to run, pause, or resume a copy pipe. See [tb copy](/docs/forward/dev-reference/commands/tb-copy).
You can query `tinybird.jobs_log` to see any running jobs, as well as any jobs that have finished during the last 48 hours.
### Run Copy Pipes in Tinybird Local (https://www.tinybird.co/docs/forward/core-concepts/copy-pipes#run-copy-pipes-in-tinybird-local)
When running Copy Pipes locally, you might encounter this error:
```
Error: Failed creating copy job: Error while running job: There was a problem while copying data: [Error] Cannot schedule a task: no free thread (timeout=0) (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK)
```
This occurs when the local ClickHouse® instance runs out of available threads. To fix this, limit the number of threads the copy job can use by adding the `max_threads` parameter at the top of your SQL query:
```
NODE daily_sales
SQL >
%
{{max_threads(1)}}
SELECT toStartOfDay(starting_date) day, country, sum(sales) as total_sales
FROM teams
WHERE
day BETWEEN toStartOfDay({{DateTime(job_timestamp)}}) - interval 1 day AND toStartOfDay({{DateTime(job_timestamp)}})
and country = {{ String(country, 'US')}}
GROUP BY day, country
TYPE COPY
TARGET_DATASOURCE sales_hour_copy
COPY_SCHEDULE 0 * * * *
```
The `max_threads(1)` parameter ensures the copy job uses only one thread, preventing the scheduling error in local environments with limited resources.
## On-demand compute for copy jobs (https://www.tinybird.co/docs/forward/core-concepts/copy-pipes#on-demand-compute-for-copy-jobs)
When running a copy job, you can isolate execution on dedicated on-demand compute, separate from your main workspace infrastructure. This prevents copy jobs from impacting production query performance.
Unlike deployment populates, on-demand compute for copy jobs is not automatically activated based on data size. You must pass the flag explicitly:
```
tb copy run --on-demand-compute
```
For pricing, regional rates, and example calculations, see [On-demand CPUs](/docs/forward/pricing#on-demand-cpus).
## Next steps (https://www.tinybird.co/docs/forward/core-concepts/copy-pipes#next-steps)
- Learn about [Materialized views](/docs/forward/core-concepts/materialized-views).
- Publish your data with [API Endpoints](/docs/forward/core-concepts/api-endpoints).
---
URL: https://www.tinybird.co/docs/forward/core-concepts/connections
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Connections \| Tinybird"
theme-color: "#171612"
description: "Connections store the credentials and settings Tinybird uses to ingest data from external services."
inkeep:version: "forward"
---
# Connections (https://www.tinybird.co/docs/forward/core-concepts/connections#connections)
Connections store the credentials and settings Tinybird uses to connect to external services such as Kafka, Amazon S3, and Google Cloud Storage. Data Sources reference connections to stream or batch import data into Tinybird.
Connections are managed by Tinybird and are deployed as part of your project. They handle the integration details for:
- Authentication and secure connections
- Schema detection and mapping
- Incremental updates and change data capture
- Error handling and monitoring
- Scheduling and orchestration
Connections are independent from Data Sources. Deleting a Data Source that uses a connection does not delete the connection, so you can reuse the same connection in other Data Sources.
## Define a connection (https://www.tinybird.co/docs/forward/core-concepts/connections#define-a-connection)
You can define connections with `.connection` files, in `.ts` files with the TypeScript SDK, or in `.py` files with the Python SDK. Use `tb_secret()` in datafiles, or `secret()` in the SDKs, to avoid storing credentials in plain text. See [Templating language](/docs/forward/core-concepts/templating-language#secrets) for how secrets are rendered across environments.
- Tinybird CLI
- TypeScript SDK
- Python SDK
connections/events_kafka.connection
```
TYPE kafka
KAFKA_BOOTSTRAP_SERVERS {{ tb_secret("KAFKA_BOOTSTRAP_SERVERS", "localhost:9092") }}
KAFKA_SECURITY_PROTOCOL SASL_SSL
KAFKA_SASL_MECHANISM PLAIN
KAFKA_KEY {{ tb_secret("KAFKA_KEY") }}
KAFKA_SECRET {{ tb_secret("KAFKA_SECRET") }}
```
See all syntax options in the [Connection files reference](/docs/forward/dev-reference/datafiles/connection-files), [TypeScript SDK reference](/docs/forward/dev-reference/typescript-sdk-resources#connections), and [Python SDK reference](/docs/forward/dev-reference/python-sdk-resources#connections).
## Create a connection (https://www.tinybird.co/docs/forward/core-concepts/connections#create-a-connection)
To create a connection, add the `.connection` file to the project, or the connection definition to `.ts` or `.py` files and deploy.
For datafile projects, you can scaffold a new connection with `tb connection create`:
```
tb connection create
```
Where `` is one of the available connection types:
- `kafka` - Create a [Kafka connection](/docs/forward/ingest-data/connectors/kafka)
- `s3` - Create an [Amazon S3 connection](/docs/forward/ingest-data/connectors/s3)
- `gcs` - Create a [Google Cloud Storage connection](/docs/forward/ingest-data/connectors/gcs)
The Tinybird CLI guides you through the configuration process and prompts you for the required credentials and settings. Once created, the CLI saves the Connection as a `.connection` file in your project's `connections/` folder.
To convert existing `.connection` files to SDK definitions, use `tinybird migrate` for TypeScript projects or `tinybird migrate` for Python projects.
## Use a connection in a Data Source (https://www.tinybird.co/docs/forward/core-concepts/connections#use-a-connection-in-a-data-source)
After you define a connection, reference it from a Data Source. Use `KAFKA_CONNECTION_NAME`, `S3_CONNECTION_NAME`, or `GCS_CONNECTION_NAME` in `.datasource` files, or pass the connection object in the TypeScript and Python SDKs.
- Tinybird CLI
- TypeScript SDK
- Python SDK
datasources/events.datasource
```
SCHEMA >
`timestamp` DateTime `json:$.timestamp`,
`payload` String `json:$.payload`
ENGINE "MergeTree"
ENGINE_SORTING_KEY "timestamp"
KAFKA_CONNECTION_NAME events_kafka
KAFKA_TOPIC events
KAFKA_GROUP_ID {{ tb_secret("KAFKA_GROUP_ID") }}
```
## Delete a connection (https://www.tinybird.co/docs/forward/core-concepts/connections#delete-a-connection)
To delete a connection, remove its `.connection` file or SDK definition, make sure no Data Sources reference it, and deploy your changes.
- Tinybird CLI
- TypeScript SDK
- Python SDK
```
tb deploy --allow-destructive-operations
```
The Tinybird CLI and SDK CLIs require the `--allow-destructive-operations` flag to confirm the removal.
Before deleting a connection, make sure no Data Sources use it. If a Data Source references the connection, the deployment fails. Remove the connection reference from all Data Sources first.
## Available connection types (https://www.tinybird.co/docs/forward/core-concepts/connections#available-connection-types)
Tinybird supports the following connection types:
- [Kafka](/docs/forward/ingest-data/connectors/kafka)
- [Amazon S3](/docs/forward/ingest-data/connectors/s3)
- [Google Cloud Storage](/docs/forward/ingest-data/connectors/gcs)
---
URL: https://www.tinybird.co/docs/forward/core-concepts/branches
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Branches \| Tinybird"
theme-color: "#171612"
description: "Create branches from your production workspace to develop and test your project before deploying to production."
inkeep:version: "forward"
---
# Tinybird Cloud Branches (https://www.tinybird.co/docs/forward/core-concepts/branches#tinybird-cloud-branches)
Tinybird Cloud Branches let you develop and test your project in ephemeral cloud environments without affecting your main workspace. They give you a full Tinybird project for development: its own ClickHouse® database, token namespace, API surface, and resource definitions.
A branch is a copy of your Workspace resources, such as Data Sources, Pipes, Endpoints, Tokens, and Connections. When you create a branch manually with `tb branch create`, it copies the workspace structure, not the workspace data. When `tb build` auto-creates a branch with `dev_mode=branch`, it attaches the latest production partition by default, as configured in `branch_data_mode`. This gives you an isolated environment where you can change schemas, queries, and APIs safely. When you need a branch without production data, create it manually without `--last-partition`.
Branches are designed for short-lived development and validation loops. Use them to test schema changes against real production values, run preview environments for pull requests, let coding agents iterate safely, or validate connector-based pipelines before deploying. You can truncate tables, redefine schemas, create endpoints, or start connector ingestion in a branch without changing production.
## When to use Cloud Branches (https://www.tinybird.co/docs/forward/core-concepts/branches#when-to-use-cloud-branches)
Branches are the default way to develop your project, optionally testing against real production data. You should use them in the following scenarios:
- You want to test your changes with real production data.
- You work with preview environments in your CI/CD pipeline before deploying to production.
- You don't use Docker and you want to test your changes without affecting production.
## Data in branches (https://www.tinybird.co/docs/forward/core-concepts/branches#data-in-branches)
When you create a branch with `--last-partition`, Tinybird attaches the latest production partition without copying the underlying data. The branch gets an isolated snapshot backed by immutable ClickHouse parts, so writes, deletes, and schema changes in the branch stay in the branch. Connector data is isolated too: Kafka starts stopped with its own consumer group, and S3, GCS, and DynamoDB connectors can import samples for validation.
## Branches in the development workflow (https://www.tinybird.co/docs/forward/core-concepts/branches#branches-in-the-development-workflow)
Branches are not promoted to production. You use a branch to validate your project, then deploy the project to your main Tinybird Cloud Workspace with `tb deploy`, the SDK deployment flow, or CI/CD.
If your Tinybird project is in a Git repository, you usually don't need to create branches manually. Check out a Git branch and run `tb build`; Tinybird automatically creates or uses a Cloud Branch with the same name as your Git branch and attaches the latest production partition by default.
```
git checkout -b preview_1
tb build
```
## Manually create a branch (https://www.tinybird.co/docs/forward/core-concepts/branches#manually-create-a-branch)
You can also create a branch manually with the Tinybird CLI.
```
tb branch create preview_1
```
If you want to use production data, use the `--last-partition` flag to bring the last partition of production data into the branch.
```
tb branch create preview_1 --last-partition
```
## Start a branch (https://www.tinybird.co/docs/forward/core-concepts/branches#start-a-branch)
After you create a branch manually, start a development session against it:
```
tb --branch=preview_1 dev
```
If the branch was created automatically with `tb build`, start the development session by passing the same branch name. `tb dev` without `--branch` starts Tinybird Local.
```
tb --branch=preview_1 dev
```
Keep this terminal running while you are working in the branch.
`tb dev` watches for changes in your project and rebuilds it automatically.
```
» Building project...
✓ datasources/user_actions.datasource created
✓ endpoints/user_actions_line_chart.pipe created
✓ Rebuild completed in 0.2s
```
Stop the process and the branch with `Ctrl+C`.
```
» Received shutdown signal, stopping...
✓ Branch 'preview_1' session stopped
```
## Make changes to your project (https://www.tinybird.co/docs/forward/core-concepts/branches#make-changes-to-your-project)
While `tb --branch=preview_1 dev` is running, you can edit files in your project and see the changes automatically applied in your branch.
### Using your editor (https://www.tinybird.co/docs/forward/core-concepts/branches#using-your-editor)
Open your editor and start editing your project. The branch detects changes and rebuilds your project automatically:
```
» Building project...
✓ datasources/user_actions.datasource updated
✓ Rebuild completed in 0.3s
```
### Using Tinybird UI (https://www.tinybird.co/docs/forward/core-concepts/branches#using-tinybird-ui)
You can also use Tinybird UI to edit your project. Run the following command to open the Tinybird UI pointing to your branch.
```
tb --branch=preview_1 open
```
`tb dev` exposes your project as API, so you can edit it directly in the browser and see changes applied automatically.
## Test with connector data (https://www.tinybird.co/docs/forward/core-concepts/branches#test-with-connector-data)
When your project uses connector data sources (Kafka, S3, GCS, or DynamoDB), you can test them in branches using the `--with-connections` flag and dedicated CLI commands. Validate schema changes, test pipelines, and verify data transformations against real production data without affecting your production environment.
### Enable connectors in branches (https://www.tinybird.co/docs/forward/core-concepts/branches#enable-connectors-in-branches)
Use `--with-connections` when building or starting a branch:
```
tb --branch=preview_1 dev --with-connections
```
Or when building directly:
```
tb --branch=preview_1 build --with-connections
```
This creates data linkers for your connector data sources (S3, Kafka, GCS, DynamoDB) in the branch.
### Kafka: Pause and resume ingestion (https://www.tinybird.co/docs/forward/core-concepts/branches#kafka-pause-and-resume-ingestion)
By default, Kafka ingestion is stopped in branches. Use `tb datasource start` to begin ingesting and `tb datasource stop` to pause.
Each time you start ingestion, a new consumer group is created with a unique ID for the branch. This means ingestion starts from the latest offset, not from where it previously left off. Consumer group IDs are unique and don't collide with production or other branches. This is by design. Branches are ephemeral testing environments, not production replicas.
Because each start creates a new consumer group, previous consumer groups become orphaned. Depending on your Kafka cluster's consumer group TTL (`offsets.retention.minutes`), these orphan groups may persist until they expire. Keep this in mind if you start and stop ingestion frequently across multiple branches.
```
tb --branch=preview_1 datasource start my_kafka_datasource
# ... observe data flowing in, test your pipelines ...
tb --branch=preview_1 datasource stop my_kafka_datasource
```
### S3/GCS: Import sample data (https://www.tinybird.co/docs/forward/core-concepts/branches#s3gcs-import-sample-data)
Instead of syncing all files from the bucket (as in production), you can import a small sample to validate your schemas and pipelines. The sample import runs as a separate job. It doesn't affect production sync state or offsets.
You can trigger a sample import using the CLI or the API:
Using the CLI
```
tb --branch=preview_1 datasource sample my_datasource --wait
```
Using the API
```
curl -X POST "https://api.tinybird.co/v0/datasources/my_datasource/sample" \
-H "Authorization: Bearer "
```
The response includes a `job_id` to track progress via `GET /v0/jobs/{job_id}`. You can import up to 10 files per request.
This is useful for:
- Validating that your schema matches the data format.
- Testing downstream pipes and endpoints.
- Verifying data transformations.
### DynamoDB: import sample data (https://www.tinybird.co/docs/forward/core-concepts/branches#dynamodb-import-sample-data)
DynamoDB data sources start empty in branches: building or deploying sets up the connection but doesn't trigger a PITR export. Use `tb datasource sample` to load a bounded subset of the table on demand:
```
tb --branch=preview_1 datasource sample my_datasource --wait
```
By default, the sample imports up to 1,500 rows by scanning the table directly, which is faster and cheaper than a full export. Use `--rows` or `--max-bytes` to adjust the cap, or `--full-export` to trigger a full PITR export instead. See [Sample data in branches and Local](/docs/forward/ingest-data/connectors/dynamodb#sample-data-in-branches-and-local) for all options and limits.
Connectors in branches are meant for testing and validation, not for replicating production workloads.
## Alternative: Tinybird Local workflow (https://www.tinybird.co/docs/forward/core-concepts/branches#alternative-tinybird-local-workflow)
If you don't need a shared Cloud environment or production data, you can use [Tinybird Local](/docs/forward/core-concepts/tinybird-local) instead. Tinybird Local runs on your machine and is useful for fast, isolated development and CI checks before you deploy to Tinybird Cloud. For the Local workflow, see [Local development](/docs/forward/development-workflow/local-development).
---
URL: https://www.tinybird.co/docs/forward/core-concepts/api-endpoints
Last update: 2026-06-02T11:48:14.000Z
Content:
---
title: "Endpoints \| Tinybird"
theme-color: "#171612"
description: "Endpoints make it easy to use the results of your queries in applications."
inkeep:version: "forward"
---
# API Endpoints (https://www.tinybird.co/docs/forward/core-concepts/api-endpoints#api-endpoints)
Endpoints are a type of pipe that you can call from other applications. You ingest your data into your Data Sources, build SQL logic inside a pipe, and then publish the result of your query as a REST API endpoint.
## Create an endpoint (https://www.tinybird.co/docs/forward/core-concepts/api-endpoints#create-an-endpoint)
You define Endpoints in `.pipe` files with the `TYPE ENDPOINT` directive, with `defineEndpoint` in the TypeScript SDK, or with `define_endpoint` in the Python SDK.
- Tinybird CLI
- TypeScript SDK
- Python SDK
pipes/top_pages.pipe
```
DESCRIPTION >
Get the most visited pages
TOKEN dashboard READ
NODE aggregated
SQL >
%
SELECT pathname, count() AS views
FROM page_views
WHERE timestamp >= {{DateTime(start_date)}}
AND timestamp <= {{DateTime(end_date)}}
GROUP BY pathname
ORDER BY views DESC
LIMIT {{Int32(limit, 10)}}
TYPE ENDPOINT
```
See all syntax options in the [Endpoint pipes reference](/docs/forward/dev-reference/datafiles/pipe-files#endpoint-pipes), [TypeScript SDK reference](/docs/forward/dev-reference/typescript-sdk-resources#endpoints), and [Python SDK reference](/docs/forward/dev-reference/python-sdk-resources#endpoints).
You can list your endpoints and their URLs and tokens using the `tb endpoint` command. See [tb endpoint](/docs/forward/dev-reference/commands/tb-endpoint).
## Response formats (https://www.tinybird.co/docs/forward/core-concepts/api-endpoints#response-formats)
API Endpoints can return results in different formats. Choose the response format by appending the format extension to the Endpoint URL.
For example:
```
https://api.tinybird.co/v0/pipes/top_pages.json
https://api.tinybird.co/v0/pipes/top_pages.csv
https://api.tinybird.co/v0/pipes/top_pages.ndjson
https://api.tinybird.co/v0/pipes/top_pages.parquet
https://api.tinybird.co/v0/pipes/top_pages.prometheus
```
Supported formats include:
- `.json`: JSON response. This is the default format.
- `.csv`: CSV response.
- `.csvwithnames`: CSV response with column names.
- `.ndjson`: Newline-delimited JSON response.
- `.parquet`: Parquet response.
- `.prometheus`: Prometheus metrics response. Use this when you want observability tools to scrape an Endpoint.
See the [Pipe API reference](/docs/api-reference/pipe-api/api-endpoints#response-formats) for request parameters and response details.
## Query parameters (https://www.tinybird.co/docs/forward/core-concepts/api-endpoints#query-parameters)
Endpoints can receive query parameters to make the published query dynamic at request time. Use parameters for filters, date ranges, pagination, sorting, tenant IDs, and limits.
Endpoints use Tinybird's [templating language](/docs/forward/core-concepts/templating-language) to validate parameter types and render SQL safely before the query runs.
For parameter syntax, defaults, helper functions, pagination, and advanced patterns, see [Query parameters](/docs/forward/query-data/query-parameters).
## Errors and retries (https://www.tinybird.co/docs/forward/core-concepts/api-endpoints#errors-and-retries)
For request behavior, HTTP error codes, and retry guidance, see [Endpoints in Query Data](/docs/forward/query-data/api-endpoints#errors-and-retries).
---
URL: https://www.tinybird.co/docs/forward/copy-export-data/s3-sink
Last update: 2026-07-31T11:10:11.000Z
Content:
---
title: "S3 Sink \| Tinybird"
theme-color: "#171612"
description: "Offload data to S3 on a batch-based schedule using Tinybird's fully managed S3 Sink Connector."
inkeep:version: "forward"
---
# S3 Sink (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#s3-sink)
Set up an S3 Sink to export data from Tinybird to any S3 bucket in CSV, NDJSON, or Parquet format. Offload data on a batch-based schedule using Tinybird's fully managed Connector.
Setting up the S3 Sink requires:
1. Configuring AWS [permissions](https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#aws-permissions) using [IAM roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html).
2. Creating a connection file in Tinybird.
3. Creating a Sink pipe that uses this connection.
The S3 Sink feature is available for Developer and Enterprise plans. See [Plans](/docs/forward/pricing).
## Environment considerations (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#environment-considerations)
Before setting up the S3 Sink, understand how it works in different environments.
### Cloud environment (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#cloud-environment)
In the Tinybird Cloud environment, Tinybird uses its own AWS account to assume the IAM role you create and write to your S3 bucket.
### Local environment (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#local-environment)
When using the S3 Sink in the Tinybird Local environment, which runs in a container, you need to pass your local AWS credentials to the container. These credentials must have the [permissions described in the AWS permissions section](https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#aws-permissions), including access to S3 operations like `PutObject`, `ListBucket`, etc. This allows Tinybird Local to assume the IAM role you specify in your connection.
To pass your AWS credentials, use the `--use-aws-creds` flag when starting Tinybird Local:
```
tb local start --use-aws-creds
» Starting Tinybird Local...
✓ AWS credentials found and will be passed to Tinybird Local (region: us-east-1)
* Waiting for Tinybird Local to be ready...
✓ Tinybird Local is ready!
```
If you're using a specific AWS profile, you can specify it using the `AWS_PROFILE` environment variable:
```
AWS_PROFILE=my-profile tb local start --use-aws-creds
```
When using the S3 Sink in the `--local` environment, scheduled sink operations are not supported. You can only run on-demand sinks using `tb sink run `. For scheduled sink operations, use the Cloud environment.
## AWS permissions (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#aws-permissions)
The S3 Sink requires an IAM Role with specific permissions to write objects to your Amazon S3 bucket:
- `s3:GetObject`
- `s3:PutObject`
- `s3:PutObjectAcl`
- `s3:ListBucket`
- `s3:GetBucketLocation`
**One IAM role can access multiple buckets**: Update the access policy to include multiple buckets by adding their ARNs to the `Resource` array. Reuse the same IAM role across multiple S3 Sink Connections for different buckets.
You need to create both an access policy and a trust policy in AWS:
- AWS Access Policy
- AWS Trust Policy
```
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:PutObjectAcl"
],
"Resource": "arn:aws:s3:::{bucket-name}/*"
},
{
"Effect": "Allow",
"Action": [
"s3:GetBucketLocation",
"s3:ListBucket"
],
"Resource": "arn:aws:s3:::{bucket-name}"
}
]
}
```
## Set up the sink (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#set-up-the-sink)
1
### Create an S3 connection (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#create-an-s3-connection)
You can create an S3 connection in Tinybird using either the guided CLI process or by manually creating a connection file.
#### Option 1: Use the guided CLI process (recommended) (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#option-1-use-the-guided-cli-process-recommended)
The Tinybird CLI provides a guided process that helps you set up the required AWS permissions and creates the connection file automatically:
```
tb connection create s3
```
When prompted, you need to:
1. Enter a name for your connection.
2. Specify whether you use this Connection for sinking or ingesting data.
3. Enter the S3 bucket name.
4. Enter the AWS region where your bucket is located.
5. Copy the displayed AWS IAM policy to your clipboard. You need it to set up permissions in AWS.
6. Copy the displayed AWS IAM role trust policy for your Local environment, then enter the ARN of the role you create.
7. Copy the displayed AWS IAM role trust policy for your Cloud environment, then enter the ARN of the role you create.
8. Store the ARN values securely using [tb secret](/docs/forward/dev-reference/commands/tb-secret) to use different roles for each environment.
#### Option 2: Define the Connection manually (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#option-2-define-the-connection-manually)
You can also define the S3 Connection manually in your project. The recommended authentication method is an IAM role with secrets for the role ARN.
- Tinybird CLI
- TypeScript SDK
- Python SDK
connections/s3sample.connection
```
TYPE s3
S3_REGION ""
S3_ARN {{ tb_secret("AWS_ROLE_ARN") }}
```
When creating your Connection manually, set up the required AWS IAM role with appropriate permissions. See the [AWS permissions](https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#aws-permissions) section for details on the required access policy and trust policy configurations.
Use IAM role authentication instead of HMAC authentication. It uses temporary credentials and follows AWS security best practices.
See [Connection files](/docs/forward/dev-reference/datafiles/connection-files) for more details on how to create a connection file and manage secrets.
Create separate connections for each environment: Local and Cloud.
For example, you can create:
- `my-s3-local` for your Local environment
- `my-s3-cloud` for your Cloud environment
2
### Create a Sink pipe (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#create-a-sink-pipe)
To create a Sink pipe, filter the data you want to export to your bucket in the SQL section as in any other pipe. Then, specify the pipe as a Sink and add the needed configuration.
- Tinybird CLI
- TypeScript SDK
- Python SDK
s3_export.pipe
```
NODE node_0
SQL >
SELECT *
FROM events
WHERE status = 'processed'
TYPE sink
EXPORT_CONNECTION_NAME "s3sample"
EXPORT_BUCKET_URI "s3://tinybird-sinks"
EXPORT_FILE_TEMPLATE "daily_prices" # Supports partitioning
EXPORT_SCHEDULE "*/5 * * * *"
EXPORT_FORMAT "csv" # Optional
EXPORT_COMPRESSION "gz" # Optional
EXPORT_STRATEGY "create_new" # Optional
```
3
### Deploy the Sink pipe (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#deploy-the-sink-pipe)
After defining your S3 data source and connection, test it by running a deploy check:
- Tinybird CLI
- TypeScript SDK
- Python SDK
```
tb --cloud deploy --check
```
This runs the connection locally and checks if the connection is valid. To see the connection details, run `tb --cloud connection ls`.
When ready, push the datafile to your Workspace using `tb deploy` to create the Sink pipe:
- Tinybird CLI
- TypeScript SDK
- Python SDK
```
tb --cloud deploy
```
This creates the Sink pipe in your workspace and makes it available for execution.
## Connection file settings (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#connection-file-settings)
The S3 Connector uses the following settings in `.connection` files:
| Instruction | Required | Description |
| --- | --- | --- |
| `S3_REGION` | Yes | Region of the S3 bucket. |
| `S3_ARN` | No* | ARN of the IAM role with the required permissions. Required for IAM Role authentication. |
| `S3_ACCESS_KEY` | No* | AWS access key for HMAC authentication. Store as a [Tinybird secret](/docs/forward/dev-reference/commands/tb-secret). |
| `S3_SECRET` | No* | AWS secret key for HMAC authentication. Store as a [Tinybird secret](/docs/forward/dev-reference/commands/tb-secret). |
*Either `S3_ARN` (for IAM Role authentication) or both `S3_ACCESS_KEY` and `S3_SECRET` (for HMAC authentication) are required.
## Pipe settings (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#pipe-settings)
The S3 Sink pipe uses the following settings in .pipe files:
| Key | Type | Description |
| --- | --- | --- |
| `EXPORT_CONNECTION_NAME` | string | Required. The connection name to the destination service. This the connection created in Step 1. |
| `EXPORT_BUCKET_URI` | string | Required. The path to the destination bucket. Example: `s3://tinybird-export` |
| `EXPORT_FILE_TEMPLATE` | string | Required. The target filename. Can use parameters to dynamically name and partition the files. See [File template](https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#file-template). Example: `daily_prices_{customer_id}` |
| `EXPORT_SCHEDULE` | string | Required. A crontab expression that sets the frequency of the Sink operation or the @on-demand string. |
| `EXPORT_FORMAT` | string | Optional. The output format of the file. Values: CSV, NDJSON, Parquet. Default value: CSV |
| `EXPORT_COMPRESSION` | string | Optional. Accepted values: `none`, `gz` for gzip, `br` for brotli, `xz` for LZMA, `zst` for zstd. Default: `none` |
| `EXPORT_STRATEGY` | string | Optional. Defines how to handle existing files. Values: `create_new` (default), `replace`. See [Write strategies](https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#write-strategies). |
### Supported regions (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#supported-regions)
The Tinybird S3 Sink feature only supports exporting data to the following AWS regions:
- `us-east-*`
- `us-west-*`
- `eu-central-*`
- `eu-west-*`
- `eu-south-*`
- `eu-north-*`
### Scheduling considerations (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#scheduling-considerations)
The schedule applied to a Sink pipe doesn't guarantee that the underlying job executes immediately at the configured time. The job is placed into a job queue when the configured time elapses. It is possible that, if the queue is busy, the job could be delayed and executed after the scheduled time.
To reduce the chances of a busy queue affecting your Sink pipe execution schedule, distribute the jobs over a wider period of time rather than grouping them close together.
### Write strategies (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#write-strategies)
The `EXPORT_STRATEGY` parameter determines how Tinybird handles existing files in your S3 bucket:
- **`create_new`** (default): Creates new files without overwriting existing ones. If a file with the same name already exists, Tinybird appends a suffix to make the filename unique.
- **`replace`**: Overwrites existing files with the same name. Use this when you want to replace previous exports entirely.
### Query parameters (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#query-parameters)
You can add [query parameters](/docs/forward/query-data/query-parameters) to your Sink pipes, the same way you do in API Endpoints or Copy pipes.
- For on-demand executions, you can set parameters when you trigger the Sink pipe to whatever values you wish.
- For scheduled executions, the Sink pipe uses the default parameter values.
## Execute the Sink pipe (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#execute-the-sink-pipe)
### On-demand execution (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#on-demand-execution)
You can trigger your Sink pipe manually using:
```
tb sink run
```
When triggering a Sink pipe you have the option of overriding several of its settings, like format or compression. Refer to the [Sink pipes API spec](/docs/api-reference/sink-pipes-api) for the full list of parameters.
### Scheduled execution (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#scheduled-execution)
If you configured a schedule with `EXPORT_SCHEDULE`, the Sink pipe runs automatically according to the cron expression.
Once the Sink pipe is triggered, it creates a standard Tinybird job that can be followed via the `v0/jobs` API or using `tb job ls --kind=sink`.
## File template (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#file-template)
Partition the result into different files to organize your data and get smaller files. The file template defines the partitioning based on the values of columns in the result set.
### Partition by column (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#partition-by-column)
Add a template variable like `{COLUMN_NAME}` to the filename. For instance, consider the following query schema and result for an export:
| customer_id | invoice_id | amount |
| --- | --- | --- |
| ACME | INV20230608 | 23.45 |
| ACME | 12345INV | 12.3 |
| GLOBEX | INV-ABC-789 | 35.34 |
| OSCORP | INVOICE2023-06-08 | 57 |
| ACME | INV-XYZ-98765 | 23.16 |
| OSCORP | INV210608-001 | 62.23 |
| GLOBEX | 987INV654 | 36.23 |
With the given file template `invoice_summary_{customer_id}.csv` you'd get 3 files:
`invoice_summary_ACME.csv`
| customer_id | invoice_id | amount |
| --- | --- | --- |
| ACME | INV20230608 | 23.45 |
| ACME | 12345INV | 12.3 |
| ACME | INV-XYZ-98765 | 23.16 |
`invoice_summary_OSCORP.csv`
| customer_id | invoice_id | amount |
| --- | --- | --- |
| OSCORP | INVOICE2023-06-08 | 57 |
| OSCORP | INV210608-001 | 62.23 |
`invoice_summary_GLOBEX.csv`
| customer_id | invoice_id | amount |
| --- | --- | --- |
| GLOBEX | INV-ABC-789 | 35.34 |
| GLOBEX | 987INV654 | 36.23 |
### Values format (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#values-format)
For DateTime columns, partitioning by the column can create as many files as seconds because each second is a different value. In an hour, that's potentially 3600 files.
To help partition in a sensible way, you can add a format string to the column name using the following placeholders:
| Placeholder | Description | Example |
| --- | --- | --- |
| %Y | Year | 2023 |
| %m | Month as an integer number (01-12) | 06 |
| %d | Day of the month, zero-padded (01-31) | 07 |
| %H | Hour in 24 h format from 00 through 23 | 14 |
| %i | Minute (00-59) | 45 |
For instance, for a result like this:
| timestamp | invoice_id | amount |
| --- | --- | --- |
| 2023-07-07 09:07:05 | INV20230608 | 23.45 |
| 2023-07-07 09:07:01 | 12345INV | 12.3 |
| 2023-07-07 09:06:45 | INV-ABC-789 | 35.34 |
| 2023-07-07 09:05:35 | INVOICE2023-06-08 | 57 |
| 2023-07-06 23:14:05 | INV-XYZ-98765 | 23.16 |
| 2023-07-06 23:14:02 | INV210608-001 | 62.23 |
| 2023-07-06 23:10:55 | 987INV654 | 36.23 |
All 7 events have different times in the column timestamp. Using a file template like `invoices_{timestamp}` would create 7 different files.
If you were interested in writing one file per hour, you could use a file template like `invoices_{timestamp, '%Y%m%d-%H'}`. You'd then get only two files for that dataset:
`invoices_20230707-09.csv`
| timestamp | invoice_id | amount |
| --- | --- | --- |
| 2023-07-07 09:07:05 | INV20230608 | 23.45 |
| 2023-07-07 09:07:01 | 12345INV | 12.3 |
| 2023-07-07 09:06:45 | INV-ABC-789 | 35.34 |
| 2023-07-07 09:05:35 | INVOICE2023-06-08 | 57 |
`invoices_20230706-23.csv`
| timestamp | invoice_id | amount |
| --- | --- | --- |
| 2023-07-06 23:14:05 | INV-XYZ-98765 | 23.16 |
| 2023-07-06 23:14:02 | INV210608-001 | 62.23 |
| 2023-07-06 23:10:55 | 987INV654 | 36.23 |
### By number of files (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#by-number-of-files)
You also have the option to write the result into X files. Instead of using a column name, use an integer between brackets.
Example: `invoice_summary.{8}.csv`
This is convenient to reduce the file size of the result, especially when the files are meant to be consumed by other services, like Snowflake where uploading big files is discouraged.
The results are written in random order. This means that the final result rows would be written in X files, but you can't count the specific order of the result.
There are a maximum of 16 files.
### Combining different partitions (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#combining-different-partitions)
It's possible to add more than one partitioning parameter in the file template. This is useful, for instance, when you do a daily dump of data, but want to export one file per hour.
Setting the file template as `invoices/dt={timestamp, '%Y-%m-%d'}/H{timestamp, '%H}.csv` would create the following file structure in different days and executions:
```
Invoices
├── dt=2023-07-07
│ └── H23.csv
│ └── H22.csv
│ └── H21.csv
│ └── ...
├── dt=2023-07-06
│ └── H23.csv
│ └── H22.csv
```
You can also mix column names and number of files. For instance, setting the file template as `invoices/{customer_id}/dump_{4}.csv` would create the following file structure in different days and executions:
```
Invoices
├── ACME
│ └── dump_0.csv
│ └── dump_1.csv
│ └── dump_2.csv
│ └── dump_3.csv
├── OSCORP
│ └── dump_0.csv
│ └── dump_1.csv
│ └── dump_2.csv
│ └── dump_3.csv
```
Avoid excessive partitioning. The write process creates as many files as combinations of the partitioning column values for a given result set.
## Supported file types (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#supported-file-types)
The S3 Sink supports exporting data in the following file formats:
| File type | Accepted extensions | Compression formats supported |
| --- | --- | --- |
| CSV | `.csv`, `.csv.gz` | `gzip` |
| NDJSON | `.ndjson`, `.ndjson.gz`, `.jsonl`, `.jsonl.gz`, `.json`, `.json.gz` | `gzip` |
| Parquet | `.parquet`, `.parquet.gz` | `snappy`, `gzip`, `lzo`, `brotli`, `lz4`, `zstd` |
You can optionally configure the export format using the `EXPORT_FORMAT` parameter (defaults to CSV) and compression using the `EXPORT_COMPRESSION` parameter in your Sink pipe configuration.
## Observability (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#observability)
Sink pipes operations are logged in the [tinybird.jobs_log](/docs/forward/monitoring/service-datasources#tinybird-jobs-log) Service Data Source. You can filter by `job_type = 'sink'` to see only Sink pipe executions.
For more detailed Sink-specific information, you can also use [tinybird.sinks_ops_log](/docs/forward/monitoring/service-datasources#tinybird-sinks-ops-log).
Data Transfer incurred by Sink pipes is tracked in [tinybird.data_transfer](/docs/forward/monitoring/service-datasources#tinybird-data-transfer) Service Data Source.
## Limits & quotas (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#limits-quotas)
Check the [limits page](/docs/forward/pricing/limits) for limits on ingestion, queries, API Endpoints, and more.
## Billing (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#billing)
Tinybird bills Sink pipes based on Data Transfer. When a Sink pipe executes, it uses your plan's included compute resources to run the query, then writes the result to a bucket as Data Transfer. If Tinybird compresses the resulting files, it accounts for the compressed size.
### Data transfer (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#data-transfer)
Data Transfer depends on your environment. There are two scenarios:
- The destination bucket is in the **same** cloud provider and region as your Tinybird Workspace: $0.01 / GB
- The destination bucket is in a **different** cloud provider or region as your Tinybird Workspace: $0.10 / GB
## Next steps (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#next-steps)
- Get familiar with the [Service Data Source](/docs/forward/monitoring/service-datasources) and see what's going on in your account
- Deep dive on Tinybird's [pipes concept](/docs/forward/core-concepts/pipes)
---
URL: https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink
Last update: 2026-08-07T12:15:37.000Z
Content:
---
title: "Kafka Sink \| Tinybird"
theme-color: "#171612"
description: "Push events to Kafka on a batch-based schedule using Tinybird's fully managed Kafka Sink Connector."
inkeep:version: "forward"
---
# Kafka Sink (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#kafka-sink)
Set up a Kafka Sink to export your data from Tinybird to any Kafka topic. Push events to Kafka on a batch-based schedule using Tinybird's fully managed Connector.
Setting up the Kafka Sink requires:
1. Creating a connection file in Tinybird with your Kafka configuration.
2. Creating a Sink pipe that uses this connection.
Kafka Sinks are available on the Developer and Enterprise plans. See [Plans](/docs/forward/pricing).
## Environment considerations (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#environment-considerations)
Before setting up the Kafka Sink, understand how it works in different environments.
### Cloud environment (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#cloud-environment)
In the Tinybird Cloud environment, Tinybird connects directly to your Kafka cluster using the connection credentials you provide.
### Local environment (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#local-environment)
When using the Kafka Sink in the Tinybird Local environment, the connection is made from within the container to your Kafka cluster. Ensure your Kafka cluster is accessible from the container network.
In the `--local` environment, you can only run on-demand Kafka Sinks using `tb sink run `. Use the Cloud environment for scheduled Sink operations.
## Set up the sink (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#set-up-the-sink)
1
### Create a Kafka connection (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#create-a-kafka-connection)
You can create a Kafka connection in Tinybird using either the guided CLI process or by manually creating a connection file.
#### Option 1: Use the guided CLI process (recommended) (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#option-1-use-the-guided-cli-process-recommended)
The Tinybird CLI provides a guided process that helps you set up the Kafka connection:
```
tb connection create kafka
```
When prompted, provide:
1. Enter a name for your connection.
2. Provide the Kafka bootstrap servers (comma-separated list).
3. Configure authentication settings (SASL/SSL if required).
4. Optionally configure additional Kafka client properties.
#### Option 2: Define the Connection manually (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#option-2-define-the-connection-manually)
You can also define the Kafka Connection manually in your project. Store credentials as secrets instead of writing them in plain text.
- Tinybird CLI
- TypeScript SDK
- Python SDK
connections/kafka_sample.connection
```
TYPE kafka
KAFKA_BOOTSTRAP_SERVERS "bootstrap_servers:port"
KAFKA_SECURITY_PROTOCOL SASL_SSL
KAFKA_SASL_MECHANISM PLAIN
KAFKA_KEY {{ tb_secret("KAFKA_KEY") }}
KAFKA_SECRET {{ tb_secret("KAFKA_SECRET") }}
```
See [Connection files](/docs/forward/dev-reference/datafiles/connection-files) for more details on how to create a connection file and manage secrets.
You need to create separate Connections for each environment you're working with, Local and Cloud.
For example, you can create:
- `kafka-local` for your Local environment
- `kafka-cloud` for your Cloud environment
2
### Create a Sink pipe (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#create-a-sink-pipe)
To create a Sink pipe, filter the data you want to export to your Kafka topic in the SQL section as in any other pipe. Then, specify the pipe as a Sink and add the needed configuration.
- Tinybird CLI
- TypeScript SDK
- Python SDK
kafka_export.pipe
```
NODE node_0
SQL >
SELECT
customer_id,
event_type,
status,
amount
FROM events
WHERE status = 'completed'
TYPE sink
EXPORT_CONNECTION_NAME "kafka_connection"
EXPORT_KAFKA_TOPIC "events_topic"
EXPORT_SCHEDULE "*/5 * * * *"
```
3
### Deploy the Sink pipe (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#deploy-the-sink-pipe)
After defining your Sink pipe and connection, test the Kafka connection:
```
tb connection data
```
This command tests the Kafka connection configuration by attempting to consume from a topic (it's designed for the Kafka source connector). While it validates connection details, for Kafka Sinks, the connection is used to produce data to Kafka, not consume from it. To see the connection details, run `tb --cloud connection ls`.
When ready, push the datafile to your Workspace using `tb deploy` to create the Sink pipe:
- Tinybird CLI
- TypeScript SDK
- Python SDK
```
tb --cloud deploy
```
This creates the Sink pipe in your workspace and makes it available for execution.
## Connection file settings (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#connection-file-settings)
The Kafka connector uses the following settings in `.connection` files:
| Instruction | Required | Description |
| --- | --- | --- |
| `KAFKA_BOOTSTRAP_SERVERS` | Yes | Comma-separated list of one or more Kafka brokers, including Port numbers. |
| `KAFKA_KEY` | Yes | Key used to authenticate with Kafka. Sometimes called Key, Client Key, or Username depending on the Kafka distribution. |
| `KAFKA_SECRET` | Yes | Secret used to authenticate with Kafka. Sometimes called Secret, Secret Key, or Password depending on the Kafka distribution. |
| `KAFKA_SECURITY_PROTOCOL` | No | Security protocol for the connection. Accepted values are `PLAINTEXT` and `SASL_SSL`. Default value is `SASL_SSL`. |
| `KAFKA_SASL_MECHANISM` | No | SASL mechanism to use for authentication. Supported values are `PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512`. Default value is `PLAIN`. |
| `KAFKA_SCHEMA_REGISTRY_URL` | No | URL of the Kafka schema registry. Used for `avro` and `json_with_schema` deserialization of keys and values. If the registry requires Basic Auth, include the username and password in the URL user information component. |
| `KAFKA_SSL_CA_PEM` | No | Content of the CA certificate in PEM format for SSL connections. |
## Pipe file settings (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#pipe-file-settings)
The Kafka Sink pipe uses the following settings in .pipe files:
| Key | Type | Description |
| --- | --- | --- |
| `EXPORT_CONNECTION_NAME` | string | Required. The connection name to the destination service. This is the connection created in Step 1. |
| `EXPORT_KAFKA_TOPIC` | string | Required. The Kafka topic where events are published. |
| `EXPORT_SCHEDULE` | string | Required. A crontab expression that sets the frequency of the Sink operation or the @on-demand string. |
### Scheduling considerations (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#scheduling-considerations)
The schedule applied to a Sink pipe doesn't guarantee that the underlying job executes immediately at the configured time. The job is placed into a job queue when the configured time elapses. It is possible that, if the queue is busy, the job could be delayed and executed after the scheduled time.
To reduce the chances of a busy queue affecting your Sink pipe execution schedule, distribute the jobs over a wider period of time rather than grouping them close together.
### Query parameters (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#query-parameters)
You can add [query parameters](/docs/forward/query-data/query-parameters) to your Sink pipes, the same way you do in API Endpoints or Copy pipes.
- For on-demand executions, you can set parameters when you trigger the Sink pipe to whatever values you wish.
- For scheduled executions, the default values for the parameters are used when the Sink pipe runs.
## Execute the Sink pipe (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#execute-the-sink-pipe)
### On-demand execution (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#on-demand-execution)
You can trigger your Sink pipe manually using:
```
tb sink run
```
When triggering a Sink pipe you have the option of overriding several of its settings, like topic or format. Refer to the [Sink pipes API spec](/docs/api-reference/sink-pipes-api) for the full list of parameters.
### Scheduled execution (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#scheduled-execution)
If you configured a schedule with `EXPORT_SCHEDULE`, the Sink pipe runs automatically according to the cron expression.
Once the Sink pipe is triggered, it creates a standard Tinybird job that can be followed via the `v0/jobs` API or using `tb job ls --kind=sink`.
## Observability (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#observability)
Sink pipes operations are logged in the [tinybird.jobs_log](/docs/forward/monitoring/service-datasources#tinybird-jobs-log) Service Data Source. You can filter by `job_type = 'sink'` to see only Sink pipe executions.
For more detailed Sink-specific information, you can also use [tinybird.sinks_ops_log](/docs/forward/monitoring/service-datasources#tinybird-sinks-ops-log).
## Limits & quotas (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#limits-quotas)
Check the [limits page](/docs/forward/pricing/limits) for limits on ingestion, queries, API Endpoints, and more.
## Billing (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#billing)
When a Sink pipe executes, it uses your plan's included compute resources to run the query, then publishes the result to Kafka.
### Enterprise customers (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#enterprise-customers)
Tinybird includes Data Transfer allowances for Enterprise customers. Contact your Customer Success team or [Tinybird support](/docs/forward/support) to discuss your specific requirements.
## Performance (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#performance)
### Throughput considerations (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#throughput-considerations)
Kafka Sink performance depends on:
- Query result size
- Network latency to Kafka cluster
- Kafka cluster capacity
- Message serialization overhead
**Monitor Sink performance:**
```
SELECT
pipe_name,
job_type,
status,
elapsed_time,
rows_written,
bytes_written
FROM tinybird.jobs_log
WHERE job_type = 'sink'
AND timestamp > now() - INTERVAL 24 hour
ORDER BY timestamp DESC
```
### Optimization tips (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#optimization-tips)
1. **Batch size**: Larger batches improve throughput but increase latency
2. **Schedule frequency**: Balance between freshness and resource usage
3. **Query optimization**: Optimize your Sink pipe queries for performance
4. **Message format**: Use efficient serialization (JSON is typically fastest)
## Use cases and examples (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#use-cases-and-examples)
### Real-time event streaming (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#real-time-event-streaming)
Stream processed events to downstream systems:
event_stream.pipe
```
NODE node_0
SQL >
SELECT
user_id,
event_type,
timestamp,
metadata
FROM processed_events
WHERE timestamp > now() - INTERVAL 1 hour
AND status = 'processed'
TYPE sink
EXPORT_CONNECTION_NAME "kafka_connection"
EXPORT_KAFKA_TOPIC "downstream_events"
EXPORT_SCHEDULE "*/5 * * * *"
```
### Data synchronization (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#data-synchronization)
Sync data changes to external systems:
user_sync.pipe
```
NODE node_0
SQL >
SELECT
user_id,
email,
updated_at,
status
FROM users
WHERE updated_at > now() - INTERVAL 1 hour
TYPE sink
EXPORT_CONNECTION_NAME "kafka_connection"
EXPORT_KAFKA_TOPIC "user_updates"
EXPORT_SCHEDULE "0 * * * *"
```
### Analytics data export (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#analytics-data-export)
Export aggregated analytics to Kafka for further processing:
analytics_export.pipe
```
NODE node_0
SQL >
SELECT
toStartOfHour(timestamp) as hour,
event_type,
count() as event_count,
uniq(user_id) as unique_users
FROM events
WHERE timestamp > now() - INTERVAL 1 hour
GROUP BY hour, event_type
TYPE sink
EXPORT_CONNECTION_NAME "kafka_connection"
EXPORT_KAFKA_TOPIC "hourly_analytics"
EXPORT_SCHEDULE "0 * * * *"
```
## Troubleshooting (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#troubleshooting)
### Issue: Sink not executing (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#issue-sink-not-executing)
**Symptoms:**
- No jobs in `jobs_log`
- Scheduled Sink not running
**Solutions:**
1. Verify schedule syntax is correct (cron format)
2. Check Sink pipe is deployed: `tb pipe ls`
3. Test connection: `tb connection data ` (validates connection details; note this command is for the Kafka source connector and tests consumption, while Sinks produce data)
4. Check for errors in `jobs_log`
### Issue: Messages not appearing in Kafka (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#issue-messages-not-appearing-in-kafka)
**Symptoms:**
- Sink executes successfully
- No messages in Kafka topic
**Solutions:**
1. Verify topic name is correct
2. Check Kafka connection credentials
3. Verify topic exists in Kafka cluster
4. Check Kafka producer permissions
5. Review `sinks_ops_log` for errors
### Issue: Slow Sink execution (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#issue-slow-sink-execution)
**Symptoms:**
- Long execution times
- Timeouts
**Solutions:**
1. Optimize Sink pipe query
2. Reduce query result size
3. Add filters to limit data
4. Check network latency to Kafka
5. Monitor Kafka cluster performance
### Monitoring Sink operations (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#monitoring-sink-operations)
**Check Sink execution status:**
```
SELECT
timestamp,
pipe_name,
status,
elapsed_time,
rows_written,
error
FROM tinybird.jobs_log
WHERE job_type = 'sink'
AND timestamp > now() - INTERVAL 24 hour
ORDER BY timestamp DESC
```
**Check Sink-specific metrics:**
```
SELECT
timestamp,
pipe_name,
topic,
messages_sent,
bytes_sent,
error
FROM tinybird.sinks_ops_log
WHERE timestamp > now() - INTERVAL 24 hour
ORDER BY timestamp DESC
```
## Next steps (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#next-steps)
- Get familiar with the [Service Data Source](/docs/forward/monitoring/service-datasources) and see what's going on in your account
- Deep dive on Tinybird's [pipes concept](/docs/forward/core-concepts/pipes)
- Review [Kafka connector documentation](/docs/forward/ingest-data/connectors/kafka) for Kafka setup
- Learn about [monitoring Sink operations](/docs/forward/monitoring/service-datasources#tinybird-sinks-ops-log)
---
URL: https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink
Last update: 2026-07-31T11:10:11.000Z
Content:
---
title: "GCS Sink \| Tinybird"
theme-color: "#171612"
description: "Offload data to Google Cloud Storage on a batch-based schedule using Tinybird's fully managed GCS Sink Connector."
inkeep:version: "forward"
---
# GCS Sink (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#gcs-sink)
Set up a GCS Sink to export data from Tinybird to any GCS bucket in CSV, NDJSON, or Parquet format. Offload data on a batch-based schedule using Tinybird's fully managed Connector.
Setting up the GCS Sink requires:
1. Configuring a [Service Account](https://cloud.google.com/iam/docs/service-accounts-create) with these [permissions](https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#gcs-permissions) in GCP.
2. Creating a connection file in Tinybird.
3. Creating a Sink pipe that uses this connection
The GCS Sink feature is available for Developer and Enterprise plans. See [Plans](/docs/forward/pricing).
## Environment considerations (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#environment-considerations)
Before setting up the GCS Sink, understand how it works in different environments.
### Cloud environment (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#cloud-environment)
In the Tinybird Cloud environment, Tinybird uses the Service Account credentials you provide to write to your GCS bucket.
### Local environment (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#local-environment)
When using the GCS Sink in the Tinybird Local environment, which runs in a container, you need to pass your local GCP credentials to the container. These credentials must have the [permissions described in the GCS permissions section](https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#gcs-permissions), including access to GCS operations like `storage.objects.create`, `storage.objects.get`, etc.
When using the GCS Sink in the `--local` environment, scheduled sink operations are not supported. You can only run on-demand sinks using `tb sink run `. For scheduled sink operations, use the Cloud environment.
## Set up the sink (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#set-up-the-sink)
1
### Create a GCS connection (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#create-a-gcs-connection)
You can create a GCS connection in Tinybird using either the guided CLI process or by manually creating a connection file.
#### Option 1: Use the guided CLI process (recommended) (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#option-1-use-the-guided-cli-process-recommended)
The Tinybird CLI provides a guided process that helps you set up the required GCP permissions and creates the connection file automatically:
```
tb connection create gcs
```
When prompted, you need to:
1. Enter a name for your connection.
2. Enter the GCS bucket name.
3. Provide the service account credentials (JSON key file).
4. Store the credentials securely using [tb secret](/docs/forward/dev-reference/commands/tb-secret) to use different credentials for each environment.
#### Option 2: Define the Connection manually (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#option-2-define-the-connection-manually)
You can also define the GCS Connection manually in your project. Use a GCP Service Account with the credentials stored as a secret:
- Tinybird CLI
- TypeScript SDK
- Python SDK
connections/gcs_sample.connection
```
TYPE gcs
GCS_SERVICE_ACCOUNT_CREDENTIALS_JSON {{ tb_secret("GCS_KEY") }}
```
When creating your Connection manually, set up the required GCP Service Account with appropriate permissions. See the [GCS permissions](https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#gcs-permissions) section for details on the required role configurations.
Use Service Account authentication instead of HMAC authentication. It provides better integration with GCP IAM and more granular permission control.
See [Connection files](/docs/forward/dev-reference/datafiles/connection-files) for more details on how to create a connection file and manage secrets.
Create separate connections for each environment: Local and Cloud.
For example, you can create:
- `my-gcs-local` for your Local environment
- `my-gcs-cloud` for your Cloud environment
2
### Create a Sink pipe (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#create-a-sink-pipe)
To create a Sink pipe, filter the data you want to export to your bucket in the SQL section as in any other pipe. Then, specify the pipe as a Sink and add the needed configuration.
- Tinybird CLI
- TypeScript SDK
- Python SDK
gcs_export.pipe
```
NODE node_0
SQL >
SELECT *
FROM events
WHERE status = 'processed'
TYPE sink
EXPORT_CONNECTION_NAME "gcs_sample"
EXPORT_BUCKET_URI "gs://tinybird-sinks"
EXPORT_FILE_TEMPLATE "daily_prices" # Supports partitioning
EXPORT_SCHEDULE "*/5 * * * *"
EXPORT_FORMAT "csv" # Optional
EXPORT_COMPRESSION "gz" # Optional
EXPORT_STRATEGY "create_new" # Optional
```
3
### Deploy the Sink pipe (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#deploy-the-sink-pipe)
After defining your GCS data source and connection, test it by running a deploy check:
- Tinybird CLI
- TypeScript SDK
- Python SDK
```
tb --cloud deploy --check
```
This runs the connection locally and checks if the connection is valid. To see the connection details, run `tb --cloud connection ls`.
When ready, push the datafile to your Workspace using `tb deploy` to create the Sink pipe:
- Tinybird CLI
- TypeScript SDK
- Python SDK
```
tb --cloud deploy
```
This creates the Sink pipe in your workspace and makes it available for execution.
## Connection file settings (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#connection-file-settings)
The GCS connector uses the following settings in `.connection` files:
| Instruction | Required | Description |
| --- | --- | --- |
| `GCS_SERVICE_ACCOUNT_CREDENTIALS_JSON` | No* | Service Account Key in JSON format for Service Account authentication. Store it using [Tinybird Secrets](/docs/forward/dev-reference/commands/tb-secret). |
| `GCS_ACCESS_ID` | No* | HMAC access ID for HMAC authentication. Store as a [Tinybird secret](/docs/forward/dev-reference/commands/tb-secret). |
| `GCS_SECRET` | No* | HMAC secret key for HMAC authentication. Store as a [Tinybird secret](/docs/forward/dev-reference/commands/tb-secret). |
*Either `GCS_SERVICE_ACCOUNT_CREDENTIALS_JSON` (for Service Account authentication) or both `GCS_ACCESS_ID` and `GCS_SECRET` (for HMAC authentication) are required.
## Pipe file settings (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#pipe-file-settings)
The GCS Sink pipe uses the following settings in `.pipe` files:
| Key | Type | Description |
| --- | --- | --- |
| `EXPORT_CONNECTION_NAME` | string | Required. The connection name to the destination service. This is the connection created in Step 1. |
| `EXPORT_BUCKET_URI` | string | Required. The path to the destination bucket. Example: `gs://tinybird-export` |
| `EXPORT_FILE_TEMPLATE` | string | Required. The target filename. Can use parameters to dynamically name and partition the files. See [File template](https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#file-template). Example: `daily_prices_{customer_id}` |
| `EXPORT_SCHEDULE` | string | Required. A crontab expression that sets the frequency of the Sink operation or the @on-demand string. |
| `EXPORT_FORMAT` | string | Optional. The output format of the file. Values: CSV, NDJSON, Parquet. Default value: CSV |
| `EXPORT_COMPRESSION` | string | Optional. Accepted values: `none`, `gz` for gzip, `br` for brotli, `xz` for LZMA, `zst` for zstd. Default: `none` |
| `EXPORT_STRATEGY` | string | Optional. Defines how to handle existing files. Values: `create_new` (default), `replace`. See [Write strategies](https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#write-strategies). |
### Supported regions (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#supported-regions)
The Tinybird GCS Sink feature only supports exporting data to the following GCP regions:
- `us-*`
- `eu-*`
- `us-central-*`
- `us-east-*`
- `us-west-*`
- `europe-west-*`
- `northamerica-northeast-*`
### Scheduling considerations (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#scheduling-considerations)
The schedule applied to a Sink pipe doesn't guarantee that the underlying job executes immediately at the configured time. The job is placed into a job queue when the configured time elapses. It is possible that, if the queue is busy, the job could be delayed and executed after the scheduled time.
To reduce the chances of a busy queue affecting your Sink pipe execution schedule, distribute the jobs over a wider period of time rather than grouping them close together.
### Write strategies (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#write-strategies)
The `EXPORT_STRATEGY` parameter determines how Tinybird handles existing files in your GCS bucket:
- **`create_new`** (default): Creates new files without overwriting existing ones. If a file with the same name already exists, Tinybird appends a suffix to make the filename unique.
- **`replace`**: Overwrites existing files with the same name. Use this when you want to replace previous exports entirely.
### Query parameters (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#query-parameters)
You can add [query parameters](/docs/forward/query-data/query-parameters) to your Sink pipes, the same way you do in API Endpoints or Copy pipes.
- For on-demand executions, you can set parameters when you trigger the Sink pipe to whatever values you wish.
- For scheduled executions, the Sink pipe uses the default parameter values.
## Execute the Sink pipe (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#execute-the-sink-pipe)
### On-demand execution (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#on-demand-execution)
You can trigger your Sink pipe manually using:
```
tb sink run
```
When triggering a Sink pipe you have the option of overriding several of its settings, like format or compression. Refer to the [Sink pipes API spec](/docs/api-reference/sink-pipes-api) for the full list of parameters.
### Scheduled execution (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#scheduled-execution)
If you configured a schedule with `EXPORT_SCHEDULE`, the Sink pipe runs automatically according to the cron expression.
Once the Sink pipe is triggered, it creates a standard Tinybird job that can be followed via the `v0/jobs` API or using `tb job ls --kind=sink`.
## File template (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#file-template)
Partition the result into different files to organize your data and get smaller files. The file template defines the partitioning based on the values of columns in the result set.
### Partition by column (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#partition-by-column)
Add a template variable like `{COLUMN_NAME}` to the filename. For instance, consider the following query schema and result for an export:
| customer_id | invoice_id | amount |
| --- | --- | --- |
| ACME | INV20230608 | 23.45 |
| ACME | 12345INV | 12.3 |
| GLOBEX | INV-ABC-789 | 35.34 |
| OSCORP | INVOICE2023-06-08 | 57 |
| ACME | INV-XYZ-98765 | 23.16 |
| OSCORP | INV210608-001 | 62.23 |
| GLOBEX | 987INV654 | 36.23 |
With the given file template `invoice_summary_{customer_id}.csv` you'd get 3 files:
`invoice_summary_ACME.csv`
| customer_id | invoice_id | amount |
| --- | --- | --- |
| ACME | INV20230608 | 23.45 |
| ACME | 12345INV | 12.3 |
| ACME | INV-XYZ-98765 | 23.16 |
`invoice_summary_OSCORP.csv`
| customer_id | invoice_id | amount |
| --- | --- | --- |
| OSCORP | INVOICE2023-06-08 | 57 |
| OSCORP | INV210608-001 | 62.23 |
`invoice_summary_GLOBEX.csv`
| customer_id | invoice_id | amount |
| --- | --- | --- |
| GLOBEX | INV-ABC-789 | 35.34 |
| GLOBEX | 987INV654 | 36.23 |
### Values format (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#values-format)
For DateTime columns, partitioning by the column can create as many files as seconds because each second is a different value. In an hour, that's potentially 3600 files.
To help partition in a sensible way, you can add a format string to the column name using the following placeholders:
| Placeholder | Description | Example |
| --- | --- | --- |
| %Y | Year | 2023 |
| %m | Month as an integer number (01-12) | 06 |
| %d | Day of the month, zero-padded (01-31) | 07 |
| %H | Hour in 24 h format from 00 through 23 | 14 |
| %i | Minute (00-59) | 45 |
For instance, for a result like this:
| timestamp | invoice_id | amount |
| --- | --- | --- |
| 2023-07-07 09:07:05 | INV20230608 | 23.45 |
| 2023-07-07 09:07:01 | 12345INV | 12.3 |
| 2023-07-07 09:06:45 | INV-ABC-789 | 35.34 |
| 2023-07-07 09:05:35 | INVOICE2023-06-08 | 57 |
| 2023-07-06 23:14:05 | INV-XYZ-98765 | 23.16 |
| 2023-07-06 23:14:02 | INV210608-001 | 62.23 |
| 2023-07-06 23:10:55 | 987INV654 | 36.23 |
All 7 events have different times in the column timestamp. Using a file template like `invoices_{timestamp}` would create 7 different files.
If you were interested in writing one file per hour, you could use a file template like `invoices_{timestamp, '%Y%m%d-%H'}`. You'd then get only two files for that dataset:
`invoices_20230707-09.csv`
| timestamp | invoice_id | amount |
| --- | --- | --- |
| 2023-07-07 09:07:05 | INV20230608 | 23.45 |
| 2023-07-07 09:07:01 | 12345INV | 12.3 |
| 2023-07-07 09:06:45 | INV-ABC-789 | 35.34 |
| 2023-07-07 09:05:35 | INVOICE2023-06-08 | 57 |
`invoices_20230706-23.csv`
| timestamp | invoice_id | amount |
| --- | --- | --- |
| 2023-07-06 23:14:05 | INV-XYZ-98765 | 23.16 |
| 2023-07-06 23:14:02 | INV210608-001 | 62.23 |
| 2023-07-06 23:10:55 | 987INV654 | 36.23 |
### By number of files (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#by-number-of-files)
You also have the option to write the result into X files. Instead of using a column name, use an integer between brackets.
Example: `invoice_summary.{8}.csv`
This is convenient to reduce the file size of the result, especially when the files are meant to be consumed by other services where uploading big files is discouraged.
The results are written in random order. This means that the final result rows would be written in X files, but you can't count the specific order of the result.
There are a maximum of 16 files.
### Combining different partitions (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#combining-different-partitions)
It's possible to add more than one partitioning parameter in the file template. This is useful, for instance, when you do a daily dump of data, but want to export one file per hour.
Setting the file template as `invoices/dt={timestamp, '%Y-%m-%d'}/H{timestamp, '%H}.csv` would create the following file structure in different days and executions:
```
Invoices
├── dt=2023-07-07
│ └── H23.csv
│ └── H22.csv
│ └── H21.csv
│ └── ...
├── dt=2023-07-06
│ └── H23.csv
│ └── H22.csv
```
You can also mix column names and number of files. For instance, setting the file template as `invoices/{customer_id}/dump_{4}.csv` would create the following file structure in different days and executions:
```
Invoices
├── ACME
│ └── dump_0.csv
│ └── dump_1.csv
│ └── dump_2.csv
│ └── dump_3.csv
├── OSCORP
│ └── dump_0.csv
│ └── dump_1.csv
│ └── dump_2.csv
│ └── dump_3.csv
```
Avoid excessive partitioning. The write process creates as many files as combinations of the partitioning column values for a given result set.
## Supported file types (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#supported-file-types)
The GCS Sink supports exporting data in the following file formats:
| File type | Accepted extensions | Compression formats supported |
| --- | --- | --- |
| CSV | `.csv`, `.csv.gz` | `gzip` |
| NDJSON | `.ndjson`, `.ndjson.gz`, `.jsonl`, `.jsonl.gz`, `.json`, `.json.gz` | `gzip` |
| Parquet | `.parquet`, `.parquet.gz` | `snappy`, `gzip`, `lzo`, `brotli`, `lz4`, `zstd` |
You can optionally configure the export format using the `EXPORT_FORMAT` parameter (defaults to CSV) and compression using the `EXPORT_COMPRESSION` parameter in your Sink pipe configuration.
## GCS permissions (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#gcs-permissions)
The GCS Connector requires a Service Account with specific permissions to access objects in your GCS bucket. Following the principle of least privilege, the minimum required roles are:
- **Storage Object Creator**, `roles/storage.objectCreator` - Allows users to create objects
- **Storage Object Viewer** (`roles/storage.objectViewer`) - Grants access to view objects and their metadata, and list objects in a bucket
- **Storage Bucket Viewer** (`roles/storage.legacyBucketReader`) - Allows users to list objects in a bucket and view bucket metadata
Create a Service Account in GCP:
1. In the GCP console, create or use an existing Service Account.
2. Assign the following roles to the service account for the specific bucket or project:
- `roles/storage.legacyBucketReader`
- `roles/storage.objectCreator`
- `roles/storage.objectViewer`
3. Generate a JSON key file and download it.
4. Store the key as a Tinybird secret.
Alternatively, you can use the broader **Storage Object Admin** (`roles/storage.objectAdmin`) role which includes all the necessary permissions, but it grants additional permissions beyond what's required for the sink operation.
## Observability (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#observability)
Sink pipes operations are logged in the [tinybird.jobs_log](/docs/forward/monitoring/service-datasources#tinybird-jobs-log) Service Data Source. You can filter by `job_type = 'sink'` to see only Sink pipe executions.
For more detailed Sink-specific information, you can also use [tinybird.sinks_ops_log](/docs/forward/monitoring/service-datasources#tinybird-sinks-ops-log).
Data Transfer incurred by Sink pipes is tracked in [tinybird.data_transfer](/docs/forward/monitoring/service-datasources#tinybird-data-transfer) Service Data Source.
## Limits & quotas (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#limits-quotas)
Check the [limits page](/docs/forward/pricing/limits) for limits on ingestion, queries, API Endpoints, and more.
## Billing (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#billing)
Tinybird bills Sink pipes based on Data Transfer. When a Sink pipe executes, it uses your plan's included compute resources to run the query, then writes the result to a bucket as Data Transfer. If Tinybird compresses the resulting files, it accounts for the compressed size.
### Data transfer (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#data-transfer)
Data Transfer depends on your environment. There are two scenarios:
- The destination bucket is in the **same** cloud provider and region as your Tinybird Workspace: $0.01 / GB
- The destination bucket is in a **different** cloud provider or region as your Tinybird Workspace: $0.10 / GB
You must include the **Storage Bucket Viewer** permission in your Service Account configuration. This permission allows Tinybird to determine your bucket's region and apply the correct billing rate. Without this permission, Tinybird can't detect the bucket region and charges the higher cross-region rate of $0.10 / GB regardless of your bucket's actual location.
## Next steps (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#next-steps)
- Get familiar with the [Service Data Source](/docs/forward/monitoring/service-datasources) and see what's going on in your account
- Deep dive on Tinybird's [pipes concept](/docs/forward/core-concepts/pipes)
---
URL: https://www.tinybird.co/docs/forward/guides/test-your-project
Last update: 2026-05-29T14:27:44.000Z
Content:
---
title: "Test your project \| Tinybird"
theme-color: "#171612"
description: "Set up local checks, fixtures, ingest API tests, and test suites for your Tinybird project."
inkeep:version: "forward"
---
# Test your project (https://www.tinybird.co/docs/forward/guides/test-your-project#test-your-project)
Test your data project before deploying to Tinybird Cloud. Use this guide to validate deployments, load fixture data, call local ingest APIs, and run YAML test suites.
You can validate and test with these workflows:
- Check the deployment before creating it.
- Create fixture files in your project.
- Ingest sample data through local APIs.
- Run and update YAML tests with `tb test`.
## Check deployment (https://www.tinybird.co/docs/forward/guides/test-your-project#check-deployment)
After finishing development, run `tb deploy --check` to validate the deployment before creating it.
```
tb deploy --check
```
## Connection checks (https://www.tinybird.co/docs/forward/guides/test-your-project#connection-checks)
When you run `tb deploy --check` (local) or `tb --cloud deploy --check` (cloud), Tinybird validates [external connections](/docs/forward/core-concepts/connections) to S3, Kafka, GCS, and databases referenced via [table functions](/docs/forward/ingest-data/table-functions).
For local checks:
- Use [tb secret set](/docs/forward/dev-reference/commands/tb-secret) to store connection secrets.
- If you use [S3 locally](/docs/forward/ingest-data/connectors/s3#local-environment), start Tinybird Local with `tb local start --use-aws-creds`.
## Fixture files (https://www.tinybird.co/docs/forward/guides/test-your-project#fixture-files)
Fixtures are sample files stored in the `fixtures/` folder.
```
my-app/
├─ datasources/
│ ├─ user_actions.datasource
│ └─ ...
├─ fixtures/
│ ├─ user_actions.ndjson
│ └─ ...
```
Create fixtures manually. For example:
```
cat > fixtures/user_actions.ndjson <<'EOF'
{"timestamp":"2026-01-01 00:00:00","action":"CLICKED","user_id":"u_1"}
{"timestamp":"2026-01-01 00:01:00","action":"VIEWED","user_id":"u_2"}
EOF
```
When fixture files match the data source name (for example, `fixtures/user_actions.ndjson` for `datasources/user_actions.datasource`), `tb build` loads them automatically.
If you use custom fixture filenames, append them manually:
```
tb datasource append user_actions --file fixtures/custom_user_actions.ndjson
```
## Call the ingest APIs (https://www.tinybird.co/docs/forward/guides/test-your-project#call-the-ingest-apis)
You can also test by calling local ingest APIs:
- Send events to the local [Events API](/docs/api-reference/events-api).
- Ingest files through the [Data Sources API](/docs/api-reference/datasource-api).
Get a token with `tb token ls`, then call your local endpoint:
- cURL (Events API)
- Local file (Data sources API)
- Remote file (Data sources API)
```
curl \
-X POST 'http://localhost:7181/v0/events?name=' \
-H "Authorization: Bearer " \
-d $''
```
## Create a test suite (https://www.tinybird.co/docs/forward/guides/test-your-project#create-a-test-suite)
Create test YAML files manually in the `tests/` folder.
Example `tests/user_action_insights_widget.yaml`:
```
- name: user_action_insights_widget_clicked
description: Returns user actions filtered by CLICKED
expected_http_status: 200
parameters: action=CLICKED
expected_result: |
{"action":"CLICKED", "user_id":"u_1"}
```
Then run tests:
```
tb test run
```
To refresh expected results for a test file or resource:
```
tb test update user_action_insights_widget
```
Copy Pipes write into target data sources instead of returning rows directly. To test them:
1. Create a YAML test file for the copy pipe in `tests/`.
2. Run `tb copy run ` to populate the target data source.
3. Run `tb test update ` to capture expected results.
## Next steps (https://www.tinybird.co/docs/forward/guides/test-your-project#next-steps)
- Make changes to your data sources and test them. See [Evolve data sources](/docs/forward/guides/evolve-data-source).
- Learn more about [deployments](/docs/forward/core-concepts/deployments).
- Learn about datafiles, like `.datasource` and `.pipe` files. See [Datafiles](/docs/forward/dev-reference/datafiles).
---
URL: https://www.tinybird.co/docs/forward/guides/sorting-key-pagination
Last update: 2026-08-10T12:50:33.000Z
Content:
---
title: "Paginate Endpoint results with a cursor \| Tinybird"
theme-color: "#171612"
description: "Use cursor pagination to page through Endpoint results."
inkeep:version: "forward"
---
# Paginate Endpoint results with a cursor (https://www.tinybird.co/docs/forward/guides/sorting-key-pagination#paginate-endpoint-results-with-a-cursor)
Cursor pagination, also called keyset pagination, uses values from the last row of one page to retrieve the next page. Use it to process a large result set sequentially without the increasing query cost of a large `OFFSET`.
This guide creates a Data Source and an API Endpoint that paginate tenant activity by timestamp and activity ID.
## Before you start (https://www.tinybird.co/docs/forward/guides/sorting-key-pagination#before-you-start)
Before you create the resources, ensure you have:
- A Tinybird project
- The Tinybird CLI installed
- A dataset with columns that define a stable, unique order
## Step 1: Create the Data Source (https://www.tinybird.co/docs/forward/guides/sorting-key-pagination#step-1-create-the-data-source)
Create a `tenant_activity_log.datasource` file:
datasources/tenant_activity_log.datasource
```
DESCRIPTION >
Fake tenant activity data for testing keyset pagination
SCHEMA >
`tenant_id` String `json:$.tenant_id`,
`occurred_at` DateTime64(3) `json:$.occurred_at`,
`activity_id` UInt64 `json:$.activity_id`,
`activity_type` LowCardinality(String) `json:$.activity_type`,
`details` String `json:$.details`
ENGINE "MergeTree"
ENGINE_SORTING_KEY "tenant_id, occurred_at, activity_id"
```
The sorting key matches how the Endpoint filters and orders the data:
- `tenant_id` identifies one tenant and appears first in the sorting key.
- `occurred_at` orders the tenant's activity chronologically.
- `activity_id` provides a unique tie-breaker when multiple activities have the same timestamp.
## Step 2: Create the API Endpoint (https://www.tinybird.co/docs/forward/guides/sorting-key-pagination#step-2-create-the-api-endpoint)
Create a `tenant_activity_page.pipe` file:
pipes/tenant_activity_page.pipe
```
TOKEN tenant_activity_page_read READ
NODE paginated_activity
SQL >
%
SELECT
tenant_id,
occurred_at,
activity_id,
activity_type,
details
FROM tenant_activity_log
WHERE tenant_id = {{String(tenant_id, required=True)}}
AND (occurred_at, activity_id) < (
{{DateTime64(cursor_occurred_at)}},
{{UInt64(cursor_activity_id)}}
)
ORDER BY
occurred_at DESC,
activity_id DESC
LIMIT {{UInt16(page_size, 5)}}
TYPE ENDPOINT
```
---
URL: https://www.tinybird.co/docs/forward/guides/share-endpoint-documentation
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Share API endpoints documentation \| Tinybird"
theme-color: "#171612"
description: "In this guide you'll learn how to share your Tinybird API endpoint documentation with development teams."
inkeep:version: "forward"
---
# Share Tinybird API endpoint documentation (https://www.tinybird.co/docs/forward/guides/share-endpoint-documentation#share-tinybird-api-endpoint-documentation)
Learn how to share your Tinybird API endpoint documentation with development teams.
## The Tinybird API endpoint page (https://www.tinybird.co/docs/forward/guides/share-endpoint-documentation#the-tinybird-api-endpoint-page)
When you publish an API endpoint, Tinybird generates a documentation page for you that is ready to share and compatible with OpenAPI 3.0. It contains your API endpoint description, information about the dynamic parameters you can use when querying this endpoint, and code snippets for quickly integrating your API in third-party applications.
To share your published API Endpoint, select **Create Chart** > **Share this API endpoint**:
## Use Static Tokens to define API endpoint subsets (https://www.tinybird.co/docs/forward/guides/share-endpoint-documentation#use-static-tokens-to-define-api-endpoint-subsets)
Tinybird authentication is based on [Tokens](/docs/forward/core-concepts/tokens), which contain different scopes for specific resources. For example, use a Token to read from one or more Endpoints or to get write permissions for a particular Data Source.
In the URLs generated for sharing a public Endpoint page, a Token parameter appears after the Endpoint ID. This page is accessible only if the Token provided in the URL has read permissions for it:
```
https://api.tinybird.co/endpoint/t_bdcad2252e794c6573e21e7e?token=
```
For security, Tinybird automatically generates a read-only Token when sharing a public Endpoint page for the first time. If you don't explicitly use it, your Admin Token is never exposed.
### The API endpoints list page (https://www.tinybird.co/docs/forward/guides/share-endpoint-documentation#the-api-endpoints-list-page)
You can also render the Endpoint information for a given Token.
```
https://app.tinybird.co///endpoints?token=
```
Enter the previous URL, with your Token and the provider and region where the API endpoint is published, into the browser to retrieve a list that shows all API endpoints that the token can read from.
When integrating your Endpoint in your apps, manage dedicated Tokens. Create a Token for every app environment so you can track requests to your Endpoints by app and choose which Endpoints are accessible to them.
Once you do that, you can share auto-generated documentation with ease, without compromising your data privacy and security.
API endpoint docs pages include a read token by default. In the "Share this API endpoint" modal, you can also see public URLs for every token with read permissions for your pipe.
## Browse your API docs (https://www.tinybird.co/docs/forward/guides/share-endpoint-documentation#browse-your-api-docs)
All endpoint documentation is compatible with OpenAPI 3.0 and accessible through the API. If you use a token with permissions for more than one API endpoint, the OpenAPI documentation contains information about all the API endpoints at once.
---
URL: https://www.tinybird.co/docs/forward/guides/replace-and-delete-data
Last update: 2026-06-29T08:18:10.000Z
Content:
---
title: "Replace and delete data \| Tinybird"
theme-color: "#171612"
description: "Update & delete operations are common in transactional databases over operational data, but sometimes you also need to make these changes on your analytical data in Tinybird."
inkeep:version: "forward"
---
# Replace and delete data in your Tinybird Data Sources (https://www.tinybird.co/docs/forward/guides/replace-and-delete-data#replace-and-delete-data-in-your-tinybird-data-sources)
Update and delete operations are common in transactional databases over operational data, but sometimes you also need to make these changes on your analytical data in Tinybird.
Sometimes, you need to delete or replace some of your data in Tinybird. Perhaps there was a bug in your app, a transient error in your operational database, or simply an evolution of requirements due to product or regulatory changes.
It's **not safe** to replace data in the partitions where you are actively ingesting data. You may lose the data inserted during the process.
Tinybird works well with append-only workloads but also fully supports replacing and deleting data. It abstracts away the tricky complexities of data replication, partition management and mutations rewriting, allowing you to focus on your data engineering flows and not the internals of real-time analytical databases.
This guide shows you using different examples, how to selectively delete or update data in Tinybird using the REST API. You can then adapt these processes for your own needs.
All API operations on this page require a Token with the `ADMIN` [scope](/docs/forward/core-concepts/static-tokens#other-tokens). In the code snippets, replace `` by your actual token.
## Delete data selectively (https://www.tinybird.co/docs/forward/guides/replace-and-delete-data#delete-data-selectively)
To delete data that's within a condition, send a POST request to the [Data Sources /delete API](/docs/api-reference/datasource-api#post--v0-datasources-(.+)-delete), providing the name of one of your Data Sources in Tinybird and a `delete_condition` parameter, which is an SQL expression filter.
Delete operations don't automatically cascade to downstream Materialized Views. You may need to perform separate delete operations on Materialized Views.
ClickHouse®'s lightweight delete feature is now available in beta. Use **synchronous** mode to wait for the deletion to complete, or **asynchronous** mode to poll a job and monitor its progress. See [Lightweight delete](https://www.tinybird.co/docs/forward/guides/replace-and-delete-data#lightweight-delete) or the [Data Sources API reference](/docs/api-reference/datasource-api#delete-rows-from-a-data-source).
Imagine you have a Data Source called `events` and you want to remove all the transactions for November 2019. You'd send a POST request like this:
- Tinybird CLI
- API
Delete data selectively
```
tb datasource delete events --sql-condition "toDate(date) >= '2019-11-01' and toDate(date) <= '2019-11-30'"
```
Once you make the request, you can see that the `POST` request to the delete API Endpoint is asynchronous. It returns a [job response](/docs/api-reference/jobs-api#jobs-api-getting-information-about-jobs), indicating an ID for the job, the status of the job, the `delete_condition`, and some other metadata. Although the delete operation runs asynchronously, the operation waits synchronously for all the mutations to be rewritten and delete the data replicas. Queries reading data either see the state before the operation or after it's complete.
```
{
"id": "64e5f541-xxxx-xxxx-xxxx-00524051861b",
"job_id": "64e5f541-xxxx-xxxx-xxxx-00524051861b",
"job_url": "https:///v0/jobs/64e5f541-xxxx-xxxx-xxxx-00524051861b",
"job": {
"kind": "delete_data",
"id": "64e5f541-xxxx-xxxx-xxxx-00524051861b",
"job_id": "64e5f541-xxxx-xxxx-xxxx-00524051861b",
"status": "waiting",
"created_at": "2023-04-11 13:52:32.423207",
"updated_at": "2023-04-11 13:52:32.423213",
"started_at": null,
"is_cancellable": true,
"datasource": {
"id": "t_c45d5ae6781b41278fcee365f5bxxxxx",
"name": "shopping_data"
},
"delete_condition": "event = 'search'"
},
"status": "waiting",
"delete_id": "64e5f541-xxxx-xxxx-xxxx-00524051861b"
}
```
You can periodically poll the `job_url` with the given ID to check the status of the deletion process. When the status is `done` your job deleted the data matching the SQL expression filter and all your Pipes and API Endpoints continue running with the remaining data in the Data Source.
### Lightweight delete (https://www.tinybird.co/docs/forward/guides/replace-and-delete-data#lightweight-delete)
The lightweight delete endpoint is currently in beta.
The `POST /v1/datasources/{name}/delete` endpoint uses ClickHouse®'s lightweight delete algorithm: matched rows are flagged in place, and queries filter them out through that mask at read time.
It takes a `delete_condition` (the SQL `WHERE` clause that matches the rows to delete), plus an optional `partition` parameter to scope the operation. You control how the call behaves with the `wait` parameter: synchronous by default, asynchronous when you pass `wait=false`. The full parameter reference is in the [Data Sources API reference](/docs/api-reference/datasource-api#delete-rows-from-a-data-source).
#### Synchronous (default) (https://www.tinybird.co/docs/forward/guides/replace-and-delete-data#synchronous-default)
A synchronous call holds the connection open until the underlying mutation finishes. The response includes a `rows_affected` property and a partition-level `mutation` block in the response body. ClickHouse® doesn't expose the affected row count for lightweight deletes, so Tinybird computes `rows_affected` by counting the rows that match `delete_condition` immediately before issuing the delete. Treat it as a best-effort number, not a strict guarantee. Reach for this mode when you want an inline acknowledgement of the deletion. Typical use cases include cleanup scripts, GDPR erasure requests, and one-off fixes.
- Tinybird CLI
- API
Synchronous lightweight delete
```
tb datasource delete events \
--sql-condition "toDate(date) >= '2019-11-01' and toDate(date) <= '2019-11-30'" \
--lightweight-delete
```
Synchronous response
```
{
"rows_affected": 1287,
"query_id": "5ad8c2e0-xxxx-xxxx-xxxx-7a3a6b2db1f0",
"delete_condition": "toDate(date) >= '2019-11-01' and toDate(date) <= '2019-11-30'",
"mutation": {
"is_done": true,
"partition_ids": ["2019-11-01", "2019-11-02"],
"partitions_changed": 2,
"partitions_done": 2,
"partitions_in_progress": 0
}
}
```
If the mutation hasn't finished within `wait_max_seconds` (default `60`), the response is still `200 OK` but with `mutation.is_done = false`. Query the `system.mutations` table to track the remaining progress until the delete completes.
#### Asynchronous (https://www.tinybird.co/docs/forward/guides/replace-and-delete-data#asynchronous)
With `wait=false`, the call returns immediately with a job envelope. Reach for this mode when the delete is large enough that you'd rather not hold the connection open, or when you already have a worker polling jobs.
- Tinybird CLI
- API
Asynchronous lightweight delete
```
tb datasource delete events \
--sql-condition "toDate(date) >= '2019-11-01' and toDate(date) <= '2019-11-30'" \
--lightweight-delete \
--no-wait
```
Asynchronous response
```
{
"id": "64e5f541-xxxx-xxxx-xxxx-00524051861b",
"job_id": "64e5f541-xxxx-xxxx-xxxx-00524051861b",
"job_url": "https:///v0/jobs/64e5f541-xxxx-xxxx-xxxx-00524051861b",
"job": {
"kind": "delete_data",
"delete_strategy": "lightweight",
"id": "64e5f541-xxxx-xxxx-xxxx-00524051861b",
"status": "waiting",
"datasource": {
"id": "t_c45d5ae6781b41278fcee365f5bxxxxx",
"name": "events"
},
"delete_condition": "toDate(date) >= '2019-11-01' and toDate(date) <= '2019-11-30'"
},
"status": "waiting",
"delete_id": "64e5f541-xxxx-xxxx-xxxx-00524051861b"
}
```
Poll the `job_url` through the [Jobs API](/docs/api-reference/jobs-api#jobs-api-getting-information-about-jobs). Once `status` is `done`, the job payload carries the same `rows_affected` and `mutation` fields as the synchronous response. Asynchronous lightweight deletes share the delete-job concurrency limit with the v0 endpoint.
The lightweight delete endpoint doesn't support deleting from Quarantine Data Sources or using subqueries inside `delete_condition`. Use the `POST /v0/datasources/{name}/delete` endpoint for those cases.
### Truncate a Data Source (https://www.tinybird.co/docs/forward/guides/replace-and-delete-data#truncate-a-data-source)
Sometimes you want to delete all data contained in a Data Source. You can perform this action from the UI and API.
Using the API, the [truncate](/docs/api-reference/datasource-api#post--v0-datasources-(.+)-truncate) endpoint deletes all rows in a Data Source as shown in this example:
- Tinybird CLI
- API
Truncate a Data Source
```
tb datasource truncate
```
You can also truncate a Data Source directly from the UI:
Deleting available via CLI or API, but truncating it to delete all the data is available via the UI.
## Replace data selectively (https://www.tinybird.co/docs/forward/guides/replace-and-delete-data#replace-data-selectively)
The ability to update data is often not the top priority when designing analytical databases, but there are always scenarios where you need to update or replace your analytical data. For example, you might have reconciliation processes over your transactions that affect your original data. Or maybe your ingestion process was simply faulty, and you ingested inaccurate data for a period of time.
In Tinybird, you can specify a condition to replace only part of the data during the ingestion process. For instance, if you want to reingest a CSV with the data for November 2019 and update your Data Source accordingly. To update the data, you pass the `replace_condition` parameter with the `toDate(date) >= '2019-11-01' and toDate(date) <= '2019-11-30'` condition.
When using `mode=replace` with an S3 URL via the API, you must use a pre-signed URL. Unlike `mode=append`, `mode=replace` is a multi-step process that passes the URL to a background worker with no access to your S3 Connector credentials. Generate a pre-signed URL programmatically using the AWS SDK or CLI before passing it to the API.
- Tinybird CLI
- API
Replace data selectively
```
tb datasource replace events \
https://storage.googleapis.com/tinybird-assets/datasets/guides/events_1M_november2019_1.csv \
--sql-condition "toDate(date) >= '2019-11-01' and toDate(date) <= '2019-11-30'"
```
The response to the previous API call looks like this:
Response after replacing data
```
{
"id": "a83fcb35-8d01-47b9-842c-a288d87679d0",
"job_id": "a83fcb35-8d01-47b9-842c-a288d87679d0",
"job_url": "https:///v0/jobs/a83fcb35-8d01-47b9-842c-a288d87679d0",
"job": {
"kind": "import",
"id": "a83fcb35-8d01-47b9-842c-a288d87679d0",
"job_id": "a83fcb35-8d01-47b9-842c-a288d87679d0",
"import_id": "a83fcb35-8d01-47b9-842c-a288d87679d0",
"status": "waiting",
"statistics": null,
"datasource": { ... },
"quarantine_rows": 0,
"invalid_lines": 0
},
"status": "waiting",
"import_id": "a83fcb35-8d01-47b9-842c-a288d87679d0"
}
```
As in the case of the selective deletion, selective replacement also runs as an asynchronous request, so [check the status of the job](/docs/api-reference/jobs-api#jobs-api-getting-information-about-jobs) periodically. You can see the status of the job by using the `job_url` returned in the previous response.
### About the replace condition (https://www.tinybird.co/docs/forward/guides/replace-and-delete-data#about-the-replace-condition)
Conditional replaces apply over partitions and the match condition selects partitions needed for the operation. The records remaining after the match condition determine the partitions involved.
Always include the partition key in the replace condition to maintain consistency.
The replace condition filters the new data that's appended, meaning it excludes rows not matching the condition. The condition is also applied for the selected partitions in the Data Source, removing rows that don't match the condition in these partitions. Rows that don't match the condition and may be present in other partitions remain.
See the [example](https://www.tinybird.co/docs/forward/guides/replace-and-delete-data#example) that follows for a better understanding of selectively replacing data in a Data Source.
### Linked Materialized Views (https://www.tinybird.co/docs/forward/guides/replace-and-delete-data#linked-materialized-views)
If you have several connected Materialized Views, then selective replaces proceed in a cascading fashion. For example, if Data Source A materializes data to Data Source B and from there to Data Source C, then when you replace data in Data Source A, Data Sources B and C automatically update accordingly. All three Data Sources need to have compatible partition keys since replaces processed by partition.
Remember: The provided Token must have the `ADMIN` [scope](/docs/forward/core-concepts/static-tokens#other-tokens).
### Example (https://www.tinybird.co/docs/forward/guides/replace-and-delete-data#example)
For this example, consider this Data Source:
characters.datasource
```
SCHEMA >
`age` Int32 `json:$.age`,
`name` String `json:$.name`,
`profession` String `json:$.profession`
ENGINE "MergeTree"
ENGINE_SORTING_KEY "name"
ENGINE_PARTITION_KEY "profession"
```
Its partition key is `ENGINE_PARTITION_KEY "profession"`. If you wanted to replace the last two rows with new data, you can send this request with the replace condition `replace_condition=(profession='Jedi')`:
- Tinybird CLI
- API
Replace with partition in condition
```
echo "50,Mace Windu,Jedi" > jedi.csv
tb datasource replace characters jedi.csv --sql-condition "profession='Jedi'"
```
Since the replace condition column matches the partition key, the result is:
However, consider what happens if you create the Data Source with `ENGINE_PARTITION_KEY "name"`:
characters.datasource
```
SCHEMA >
`age` Int32 `json:$.age`,
`name` String `json:$.name`,
`profession` String `json:$.profession`
ENGINE "MergeTree"
ENGINE_SORTING_KEY "name"
ENGINE_PARTITION_KEY "name"
```
If you were to run the same replace request, the result probably doesn't make sense:
Why were the existed rows not removed? Because the `replace` process uses the payload rows to identify which partitions to work on. The Data Source is now partitioned by name and not profession, so the process didn't delete the other "Jedi" rows. They're in different partitions because they have different names.
The rule of thumb is this: **Always make sure the replace condition uses the partition key as the filter field**.
## Replace a Data Source completely (https://www.tinybird.co/docs/forward/guides/replace-and-delete-data#replace-a-data-source-completely)
To replace a complete Data Source, make an API call similar to the previous example, without providing a `replace_condition`:
- Tinybird CLI
- API
Replace Data Source completely
```
tb datasource replace events https://storage.googleapis.com/tinybird-assets/datasets/guides/events_1M_november2019_1.csv
```
The example request is replacing a Data Source with the data found in a given URL pointing to a CSV file.
Schemas must be identical. When replacing data either selectively or entirely, the schema of the new inbound data must match that of the original Data Source. Rows not containing the same schema go to quarantine.
---
URL: https://www.tinybird.co/docs/forward/guides/reliable-scheduling-with-trigger
Last update: 2026-05-29T14:27:44.000Z
Content:
---
title: "Reliable scheduling with Trigger.dev \| Tinybird"
theme-color: "#171612"
description: "Learn how to create complex, reliable scheduling with Trigger.dev"
inkeep:version: "forward"
---
# Reliable scheduling with Trigger.dev (https://www.tinybird.co/docs/forward/guides/reliable-scheduling-with-trigger#reliable-scheduling-with-trigger-dev)
[Trigger.dev](https://trigger.dev/) is an open source background job platform. With Trigger.dev you can easily create, schedule, and manage background jobs using code.
Read on to learn how to create complex, reliable scheduling with Trigger.dev.
## Before you start (https://www.tinybird.co/docs/forward/guides/reliable-scheduling-with-trigger#before-you-start)
Before you start, ensure:
- You have a [Trigger.dev account](https://trigger.dev/).
- You have a [Tinybird workspace](https://www.tinybird.co/).
## Create your first trigger task (https://www.tinybird.co/docs/forward/guides/reliable-scheduling-with-trigger#create-your-first-trigger-task)
The [tinybird-trigger-tasks package](https://www.npmjs.com/package/@sdairs/tinybird-trigger-tasks) implements tasks for Tinybird copy pipes and the Query API. You can [find the source code in the @sdairs/tinybird-trigger repo](https://github.com/sdairs/tinybird-trigger).
1. Create a working directory, and run `npx trigger.dev@latest init` to connect the project to Trigger.dev.
2. Inside the `trigger` directory, install the npm package with `npm install @sdairs/tinybird-trigger-tasks`.
3. Create a new file called `myTask.ts` and add the following code:
```
import { task } from "@trigger.dev/sdk/v3";
import { tinybirdCopyTask } from "@sdairs/tinybird-trigger-tasks";
export const exampleExecutor = task({
id: "example-executor",
run: async (payload, { ctx }) => {
console.log("Example executor task is running");
// Run a copy job
const copyResult = await tinybirdCopyTask.triggerAndWait({ pipeId: });
console.log(copyResult);
},
});
```
1. Create a new pipe using the following SQL:
```
SELECT number + 1 AS value
FROM numbers(100)
```
1. Name the pipe `my_copy` and deploy the changes.
2. Update `myTask.ts`, replacing `` with the name of your pipe, `my_copy` in this case.
3. Create a `.env` file in your directory root.
4. Go to your Tinybird workspace and copy the Admin Token, then add it to the `.env` file as follows:
```
TINYBIRD_TOKEN=p.eyJ...
```
1. Run `npx trigger.dev@latest dev` to push the task to Trigger.dev.
2. Go to your Trigger.dev dashboard, and perform a test run to trigger the task and the copy pipe.
3. Go to your Tinybird workspace and check the copy pipe results.
## See also (https://www.tinybird.co/docs/forward/guides/reliable-scheduling-with-trigger#see-also)
- [Trigger.dev quick start](https://trigger.dev/docs/quick-start)
- [tinybird-trigger repo](https://github.com/sdairs/tinybird-trigger)
- [YouTube: Using Trigger.dev with Tinybird for code-first background job execution](https://www.youtube.com/watch?v=0TcQfcMrGNw)
---
URL: https://www.tinybird.co/docs/forward/guides/redpanda-setup
Last update: 2026-06-02T11:48:14.000Z
Content:
---
title: "Redpanda setup guide \| Tinybird"
theme-color: "#171612"
description: "Guide to setting up Tinybird's Kafka connector with Redpanda, including authentication and network configuration."
inkeep:version: "forward"
---
# Redpanda setup guide (https://www.tinybird.co/docs/forward/guides/redpanda-setup#redpanda-setup-guide)
This guide walks you through setting up Tinybird's Kafka connector with Redpanda, a Kafka-compatible streaming data platform.
## Prerequisites (https://www.tinybird.co/docs/forward/guides/redpanda-setup#prerequisites)
- A Redpanda cluster (self-hosted or Redpanda Cloud)
- Redpanda credentials (if authentication is turned on)
- Network access to your Redpanda cluster
- A Tinybird workspace
## Redpanda compatibility (https://www.tinybird.co/docs/forward/guides/redpanda-setup#redpanda-compatibility)
Redpanda is fully compatible with the Apache Kafka protocol, so you can use Tinybird's Kafka connector with Redpanda without any special configuration. The connector works with Redpanda the same way it works with Apache Kafka.
## Step 1: Get your Redpanda connection details (https://www.tinybird.co/docs/forward/guides/redpanda-setup#step-1-get-your-redpanda-connection-details)
### Bootstrap servers (https://www.tinybird.co/docs/forward/guides/redpanda-setup#bootstrap-servers)
1. For **Redpanda Cloud**: Get the bootstrap server from your Redpanda Cloud console
2. For **self-hosted Redpanda**: Use your Redpanda broker addresses
The bootstrap server format is: `:`
Common ports:
- `9092` for PLAINTEXT
- `9093` for SASL_SSL
- `9094` for TLS
### Authentication (https://www.tinybird.co/docs/forward/guides/redpanda-setup#authentication)
Redpanda supports multiple authentication methods:
- **SASL/PLAIN** - Username and password
- **SASL/SCRAM-SHA-256** or **SCRAM-SHA-512** - SCRAM authentication
- **TLS/mTLS** - Certificate-based authentication
- **PLAINTEXT** - No authentication (local development only)
## Step 2: Create the Kafka connection (https://www.tinybird.co/docs/forward/guides/redpanda-setup#step-2-create-the-kafka-connection)
You can create the Kafka connection using the CLI wizard or by manually creating a connection file.
### Option 1: Use the CLI wizard (recommended) (https://www.tinybird.co/docs/forward/guides/redpanda-setup#option-1-use-the-cli-wizard-recommended)
Run the following command to create a connection interactively:
```
tb connection create kafka
```
The wizard prompts you for:
1. Connection name
2. Bootstrap server
3. Kafka key (username for SASL/PLAIN or SCRAM)
4. Kafka secret (password for SASL/PLAIN or SCRAM)
After the wizard completes, you can edit the generated `.connection` file to add additional settings like `KAFKA_SASL_MECHANISM` if needed (defaults to PLAIN).
### Option 2: Manually create a connection file (https://www.tinybird.co/docs/forward/guides/redpanda-setup#option-2-manually-create-a-connection-file)
#### SASL/PLAIN authentication (https://www.tinybird.co/docs/forward/guides/redpanda-setup#saslplain-authentication)
connections/redpanda.connection
```
TYPE kafka
KAFKA_BOOTSTRAP_SERVERS
KAFKA_SECURITY_PROTOCOL SASL_SSL
KAFKA_SASL_MECHANISM PLAIN
KAFKA_KEY {{ tb_secret("REDPANDA_USERNAME") }}
KAFKA_SECRET {{ tb_secret("REDPANDA_PASSWORD") }}
```
Set the secrets:
```
tb [--cloud] secret set REDPANDA_USERNAME
tb [--cloud] secret set REDPANDA_PASSWORD
```
### SASL/SCRAM authentication (https://www.tinybird.co/docs/forward/guides/redpanda-setup#saslscram-authentication)
connections/redpanda_scram.connection
```
TYPE kafka
KAFKA_BOOTSTRAP_SERVERS
KAFKA_SECURITY_PROTOCOL SASL_SSL
KAFKA_SASL_MECHANISM SCRAM-SHA-256
KAFKA_KEY {{ tb_secret("REDPANDA_USERNAME") }}
KAFKA_SECRET {{ tb_secret("REDPANDA_PASSWORD") }}
```
### PLAINTEXT (local development only) (https://www.tinybird.co/docs/forward/guides/redpanda-setup#plaintext-local-development-only)
connections/redpanda_local.connection
```
TYPE kafka
KAFKA_BOOTSTRAP_SERVERS localhost:9092
KAFKA_SECURITY_PROTOCOL PLAINTEXT
```
PLAINTEXT connections should only be used for local development. Production Redpanda clusters should use SASL_SSL.
## Step 3: SSL/TLS certificate (if required) (https://www.tinybird.co/docs/forward/guides/redpanda-setup#step-3-ssltls-certificate-if-required)
If your Redpanda cluster uses self-signed certificates or a private CA, provide the CA certificate:
```
tb [--cloud] secret set --multiline REDPANDA_SSL_CA_PEM
```
Add to your connection file:
```
KAFKA_SSL_CA_PEM >
{{ tb_secret("REDPANDA_SSL_CA_PEM") }}
```
## Step 4: Create the Kafka Data Source (https://www.tinybird.co/docs/forward/guides/redpanda-setup#step-4-create-the-kafka-data-source)
Now that your connection is configured, create a Kafka Data Source. See [Create a Kafka data source](/docs/forward/ingest-data/connectors/kafka#create-a-kafka-data-source) in the main Kafka connector guide for detailed instructions on:
- Using `tb datasource create --kafka` for a guided setup
- Manually creating `.datasource` files
- Defining schemas with JSONPath expressions
- Configuring Kafka-specific settings
## Step 5: Test the connection (https://www.tinybird.co/docs/forward/guides/redpanda-setup#step-5-test-the-connection)
Test your connection and preview data:
```
tb connection data redpanda
```
This command prompts you to select a topic and consumer group ID, then returns preview data. This verifies that Tinybird can connect to your Redpanda cluster, authenticate, and consume messages.
## Redpanda Cloud specific setup (https://www.tinybird.co/docs/forward/guides/redpanda-setup#redpanda-cloud-specific-setup)
If you're using Redpanda Cloud:
1. **Get connection details** from the Redpanda Cloud console
2. **Use SASL/PLAIN** authentication with your Redpanda Cloud credentials
3. **Use the provided bootstrap server** address
4. **No SSL certificate needed** - Redpanda Cloud uses public CA certificates
## Common Redpanda issues (https://www.tinybird.co/docs/forward/guides/redpanda-setup#common-redpanda-issues)
### Issue: Authentication failed (https://www.tinybird.co/docs/forward/guides/redpanda-setup#issue-authentication-failed)
**Solutions:**
1. Verify username and password are correct
2. Check SASL mechanism matches your Redpanda configuration
3. Ensure the user has read permissions on the topic
4. Verify security protocol matches (SASL_SSL vs PLAINTEXT)
### Issue: Connection timeout (https://www.tinybird.co/docs/forward/guides/redpanda-setup#issue-connection-timeout)
**Solutions:**
1. Verify bootstrap server address is correct
2. Check network connectivity to Redpanda cluster
3. Verify firewall rules allow outbound connections
4. For self-hosted Redpanda, ensure the cluster is accessible
### Issue: SSL certificate validation (https://www.tinybird.co/docs/forward/guides/redpanda-setup#issue-ssl-certificate-validation)
**Solutions:**
1. Provide CA certificate if using self-signed certificates
2. Verify certificate is in PEM format
3. Check certificate hasn't expired
## Redpanda vs Kafka differences (https://www.tinybird.co/docs/forward/guides/redpanda-setup#redpanda-vs-kafka-differences)
While Redpanda is Kafka-compatible, there are some differences to be aware of:
1. **Performance**: Redpanda is optimized for lower latency and higher throughput
2. **Resource usage**: Redpanda typically uses less CPU and memory
3. **Configuration**: Some Kafka-specific configurations may not apply
4. **Schema Registry**: Redpanda doesn't include Schema Registry by default (use Confluent Schema Registry separately if needed)
## Best practices (https://www.tinybird.co/docs/forward/guides/redpanda-setup#best-practices)
1. **Use SASL_SSL** for production Redpanda clusters
2. **Monitor consumer lag** using `kafka_ops_log`
3. **Use unique consumer group IDs** for each Data Source
4. **Set up alerts** for high lag or errors
5. **Test connections** before deploying to production
## Related documentation (https://www.tinybird.co/docs/forward/guides/redpanda-setup#related-documentation)
- [Kafka connector documentation](/docs/forward/ingest-data/connectors/kafka) - Main setup and configuration guide
- [Monitor Kafka connectors](/docs/forward/monitoring/kafka-clickhouse-monitoring) - Set up monitoring and alerts
- [Troubleshooting guide](/docs/forward/ingest-data/connectors/kafka/troubleshooting) - Common issues and solutions
---
URL: https://www.tinybird.co/docs/forward/guides/quarantine
Last update: 2026-08-05T14:53:06.000Z
Content:
---
title: "Quarantine Data Sources \| Tinybird"
theme-color: "#171612"
description: "Quarantine data sources store data that doesn't fit the schema."
inkeep:version: "forward"
---
# Quarantine Data Sources (https://www.tinybird.co/docs/forward/guides/quarantine#quarantine-data-sources)
Every data source in your workspace has an associated quarantine data source that stores data that doesn't fit the schema. If you send rows that don't match the data source schema, they're automatically sent to the quarantine table so that the ingest process doesn't fail.
By convention, quarantine data sources follow the naming pattern `{datasource_name}_quarantine`. You can review quarantined rows at any time or perform operations on them using Pipes. This is a useful source of information when fixing issues in the origin source or applying changes during ingest.
Data in quarantine data sources is retained for 1 month, after which it's automatically deleted.
## Review quarantined data (https://www.tinybird.co/docs/forward/guides/quarantine#review-quarantined-data)
To check your quarantine data sources, run the `tb sql` command. For example:
```
tb sql "select * from _quarantine limit 10"
```
A sample output of the `tb sql` command is the following:
```
──────────────────────────────────────────────────────────────────
c__error_column: ['abslevel']
c__error: ["value '' on column 'abslevel' is not Float32"]
c__import_id: 01JKQPWT8GVXAN5GJ1VBD4XM27
day: 2014-07-30
station: Embassament de Siurana (Cornudella de Montsant)
volume: 11.57
insertion_date: 2025-02-10 10:36:20
──────────────────────────────────────────────────────────────────
```
The quarantine data source schema contains the columns of the original row and the following columns with information about the issues that caused the quarantine:
- `c__error_column` Array(String) contains an array of all the columns that contain an invalid value.
- `c__error` Array(String) contains an array of all the errors that caused the ingestion to fail and led to the row being stored in the Quarantine Data Source. Use this column with `c__error_column` to identify which columns have problems and their specific errors.
- `c__import_id` Nullable(String) contains the job's identifier in case the column was imported through a job.
- `insertion_date` (DateTime) contains the timestamp in which the ingestion was done.
## Fixing quarantined data example (https://www.tinybird.co/docs/forward/guides/quarantine#fixing-quarantined-data-example)
Using the Electric Vehicle Population Data example:
```
tb datasource create --url "https://data.wa.gov/api/views/f6w7-q2d2/rows.csv?accessType=DOWNLOAD" --name rows
tb build
tb datasource append rows "https://data.wa.gov/api/views/f6w7-q2d2/rows.csv?accessType=DOWNLOAD"
```
You should get the following quarantine error: `Error appending fixtures for 'rows': There was an error with file contents: 564 rows in quarantine.`
Inspecting the `rows_quarantine` data source:
```
tb sql "SELECT DISTINCT c__error FROM rows_quarantine"
# ────────────────────────────────────────────────────────────────────────────────────────────
# c__error: ["value '' on column 'postal_code' is not Int64", "value '' on column 'legislative_district' is not Int16", # "value '' on column 'c_2020_census_tract' is not Int64"]
# ────────────────────────────────────────────────────────────────────────────────────────────
# c__error: ["value '' on column 'electric_range' is not Int32", "value '' on column 'base_msrp' is not Int64"]
# ────────────────────────────────────────────────────────────────────────────────────────────
# c__error: ["value '' on column 'legislative_district' is not Int16"]
# ────────────────────────────────────────────────────────────────────────────────────────────
```
The problem is that some columns should be `Nullable` or have a `DEFAULT` value. Add a `DEFAULT` value of 0 for them.
Edit the `datasources/rows.datasource` file.
datasources/rows.datasource
```
DESCRIPTION >
Generated from https://data.wa.gov/api/views/f6w7-q2d2/rows.csv?accessType=DOWNLOAD
SCHEMA >
`vin__1_10_` String,
`county` String,
`city` String,
`state` String,
`postal_code` Int64 DEFAULT 0,
`model_year` Int32,
`make` String,
`model` String,
`electric_vehicle_type` String,
`clean_alternative_fuel_vehicle__cafv__eligibility` String,
`electric_range` Int32 DEFAULT 0,
`base_msrp` Int64 DEFAULT 0,
`legislative_district` Int16 DEFAULT 0,
`dol_vehicle_id` Int64,
`vehicle_location` String,
`electric_utility` String,
`c_2020_census_tract` Int64 DEFAULT 0
```
The dev server rebuilds the edited resources.
```
tb build
tb datasource append rows "https://data.wa.gov/api/views/f6w7-q2d2/rows.csv?accessType=DOWNLOAD"
```
No errors now, you're good to continue developing.
## Recover data from quarantine (https://www.tinybird.co/docs/forward/guides/quarantine#recover-data-from-quarantine)
If you have data in quarantine, the first thing to do is understand what caused the error and fix it. See [Fixing quarantined data example](https://www.tinybird.co/docs/forward/guides/quarantine#fixing-quarantined-data-example).
Once the schema issues are fixed, you can recover the quarantined data back to your main data source. The approach involves deploying the schema fix and a Copy Pipe that reads from quarantine at the same time, then triggering the copy to recover the data.
### Step 1: Fix the Data Source schema (https://www.tinybird.co/docs/forward/guides/quarantine#step-1-fix-the-data-source-schema)
Edit the Data Source definition so the quarantined rows become valid. Most quarantine issues come down to a wrong data type or a missing column, which you fix by changing the type, making the column nullable, adding a `DEFAULT` value, or adding and dropping columns.
Make only the minimum changes required to fix the schema, and leave anything unrelated for a later deployment.
Continuing the previous example, `datasources/rows.datasource` now has `DEFAULT 0` on the columns that were quarantining rows. See [Evolve data sources](/docs/forward/guides/evolve-data-source) for how Tinybird applies each type of change.
### Step 2: Add a Copy Pipe that reads from quarantine (https://www.tinybird.co/docs/forward/guides/quarantine#step-2-add-a-copy-pipe-that-reads-from-quarantine)
Add a Copy Pipe that selects the quarantined rows, casts them to the fixed schema, and appends them to the Data Source. Because quarantine columns are `Nullable(String)`, cast every column to its target type, and don't select the `c__error`, `c__error_column`, `c__import_id`, and `insertion_date` columns.
The Copy Pipe has no schedule, so it only runs when you trigger it.
- Tinybird CLI
- TypeScript SDK
- Python SDK
pipes/recover_rows_quarantine.pipe
```
DESCRIPTION Recovers quarantined rows into the rows data source
NODE recover
SQL >
SELECT
coalesce(vin__1_10_, '') AS vin__1_10_,
coalesce(county, '') AS county,
coalesce(city, '') AS city,
coalesce(state, '') AS state,
coalesce(toInt64OrNull(postal_code), 0) AS postal_code,
coalesce(toInt32OrNull(model_year), 0) AS model_year,
coalesce(make, '') AS make,
coalesce(model, '') AS model,
coalesce(electric_vehicle_type, '') AS electric_vehicle_type,
coalesce(clean_alternative_fuel_vehicle__cafv__eligibility, '') AS clean_alternative_fuel_vehicle__cafv__eligibility,
coalesce(toInt32OrNull(electric_range), 0) AS electric_range,
coalesce(toInt64OrNull(base_msrp), 0) AS base_msrp,
coalesce(toInt16OrNull(legislative_district), 0) AS legislative_district,
coalesce(toInt64OrNull(dol_vehicle_id), 0) AS dol_vehicle_id,
coalesce(vehicle_location, '') AS vehicle_location,
coalesce(electric_utility, '') AS electric_utility,
coalesce(toInt64OrNull(c_2020_census_tract), 0) AS c_2020_census_tract
FROM rows_quarantine
TYPE COPY
TARGET_DATASOURCE rows
COPY_SCHEDULE @on-demand
COPY_MODE append
```
### Step 3: Deploy the schema fix and the Copy Pipe together (https://www.tinybird.co/docs/forward/guides/quarantine#step-3-deploy-the-schema-fix-and-the-copy-pipe-together)
Validate the deployment first:
```
tb --cloud deploy --check
```
Then deploy. The schema fix and the Copy Pipe go live in the same deployment:
```
tb --cloud deploy
```
Wait until the deployment is live before you run the copy. Changes applied with `ALTER` aren't available in staging deployments, so a copy triggered before promotion doesn't write to the fixed schema. `tb deploy` promotes the deployment for you.
### Step 4: Trigger the copy (https://www.tinybird.co/docs/forward/guides/quarantine#step-4-trigger-the-copy)
Run the Copy Pipe and wait for it to finish:
```
tb --cloud copy run recover_rows_quarantine --wait
```
Check that the rows landed in the Data Source:
```
tb --cloud sql "SELECT count() FROM rows"
```
For large recoveries, run the copy on a dedicated instance to avoid competing with your production workloads:
```
tb --cloud copy run recover_rows_quarantine --wait --on-demand-compute
```
See [on-demand compute for copy jobs](/docs/forward/core-concepts/copy-pipes#on-demand-compute-for-copy-jobs).
### Step 5: Remove the Copy Pipe (https://www.tinybird.co/docs/forward/guides/quarantine#step-5-remove-the-copy-pipe)
Once the rows are recovered, delete the Copy Pipe and deploy again:
```
tb --cloud deploy
```
This deployment doesn't change the Data Source schema, so it doesn't affect the Quarantine Data Source or its remaining rows.
---
URL: https://www.tinybird.co/docs/forward/guides/postgres-cdc-with-redpanda-connect
Last update: 2026-05-29T14:27:44.000Z
Content:
---
title: "Postgres CDC with Redpanda Connect \| Tinybird"
theme-color: "#171612"
description: "Learn how to ingest data from a Postgres database using Redpanda Connect and the Postgres CDC input."
inkeep:version: "forward"
---
# PostgreSQL CDC with Redpanda Connect (https://www.tinybird.co/docs/forward/guides/postgres-cdc-with-redpanda-connect#postgresql-cdc-with-redpanda-connect)
[Redpanda Connect](https://www.redpanda.com/connect) is an ecosystem of high-performance streaming connectors that serves as a simplified and powerful alternative to Kafka Connect.
Tinybird is the ideal complement to Postgres for handling OLAP workloads. The following guide shows you how to use Redpanda Connect to ingest data from a Postgres database into Tinybird.
## Before you start (https://www.tinybird.co/docs/forward/guides/postgres-cdc-with-redpanda-connect#before-you-start)
Before you connect Postgres to Redpanda, ensure:
- You have a Redpanda cluster and Redpanda Connect installed with version 4.43.0 or higher. The following instructions use Redpanda Serverless, but you can use Redpanda Cloud Dedicated or self-hosted.
- You have a PostgreSQL database with logical replication enabled.
## Connect Postgres to Redpanda (https://www.tinybird.co/docs/forward/guides/postgres-cdc-with-redpanda-connect#connect-postgres-to-redpanda)
1. In the Redpanda Cloud console, select **Connect** from the navigation menu, then select **Create pipeline**.
2. Add the pipeline configuration. You need the following information:
- Postgres connection string (`dsn`)
- Redpanda brokers (`seed_brokers`)
- SASL mechanism (`mechanism`)
- Username (`username`)
- Password (`password`)
Use the following YAML template:
```
input:
label: "postgres_cdc"
postgres_cdc:
dsn: <>
include_transaction_markers: false
slot_name: test_slot_native_decoder
snapshot_batch_size: 100000
stream_snapshot: true
temporary_slot: true
schema: public
tables:
- <>
output:
redpanda:
seed_brokers:
- ${REDPANDA_BROKERS}
topic: <>
tls:
enabled: false
sasl:
- mechanism: SCRAM-SHA-512
password: <>
username: <>
```
See the Redpanda Connect docs for more information on the `redpanda` output and `postgres_cdc` input.
1. Start the Redpanda Connect pipeline
Select **Create** to save and create the pipeline. This takes you back to the pipeline screen, where you can find your new pipeline. Open the new pipeline to view the logs and confirm that the pipeline is running.
Select the Topics page from the navigation menu and confirm that the topic exists and that messages are being produced.
1. Connect Redpanda to Tinybird using the Kafka Connector
Create a new data source with the Kafka Connector. See [Kafka](/docs/forward/ingest-data/connectors/kafka) for more information.
Redpanda Connect continuosly consumes changes from Postgres and pushes them to your Redpanda topic. Tinybird consumes the changes from Redpanda in real time, making them available to query with minimal latency.
## See also (https://www.tinybird.co/docs/forward/guides/postgres-cdc-with-redpanda-connect#see-also)
- `redpanda` output
- `postgres_cdc` input
---
URL: https://www.tinybird.co/docs/forward/guides/multitenant-real-time-apis-with-clerk-and-tinybird
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Multi-Tenant Real-Time APIs With Clerk \| Tinybird"
theme-color: "#171612"
description: "In this guide, you'll learn how to build a multi-tenant real-time API with Clerk and Tinybird."
inkeep:version: "forward"
---
# Multi-tenant real-time APIs with Clerk and Tinybird (https://www.tinybird.co/docs/forward/guides/multitenant-real-time-apis-with-clerk-and-tinybird#multi-tenant-real-time-apis-with-clerk-and-tinybird)
Learn how to build a multi-tenant real-time API with Clerk and Tinybird.
You can view the [live demo](https://clerk-tinybird.vercel.app/) or browse the [GitHub repo (clerk-tinybird)](https://github.com/tinybirdco/clerk-tinybird).
## Prerequisites (https://www.tinybird.co/docs/forward/guides/multitenant-real-time-apis-with-clerk-and-tinybird#prerequisites)
This guide assumes that you have a Tinybird account, and you are familiar with creating a Tinybird workspace and pushing resources to it.
You need a working familiarity with Clerk and Next.js.
## JWT template (https://www.tinybird.co/docs/forward/guides/multitenant-real-time-apis-with-clerk-and-tinybird#jwt-template)
Create a JWT template in Clerk, and use the generated JWT to access Tinybird pipe endpoints.
In Clerk go to `Configure` > `JWT Templates` and choose Tinybird.
Clerk JWT tokens Tinybird template
Modify the Tinybird JWT template with these claims:
```
{
"name": "frontend_jwt",
"limits": {
"rps": 10
},
"scopes": [
{
"type": "PIPES:READ",
"resource": "",
"fixed_params": {
"org": "{{org.slug}}",
"user": "{{user.id}}"
}
}
],
"workspace_id": ""
}
```
- Use your Tinybird admin token as signking key.
- Add as many scopes as needed, use fixed params to filter your Tinybird API endpoints.
- Configure `fixed_params` to match the parameter names and values in your Tinybird API endpoints.
From your app, request a token from Clerk using the `tinybird` template, where `tinybird` is the name you gave the template.
```
const authentication = await auth()
const { userId, sessionId, getToken } = authentication
const token = await getToken({ template: "tinybird" })
fetch('https://api.tinybird.co/v0/pipes/your_pipe.json', {
headers: {
Authorization: `Bearer ${token}`
}
})
```
Use this [demo project](https://www.tinybird.co/templates/clerk-jwt) to for a fully working example.
---
URL: https://www.tinybird.co/docs/forward/guides/monitoring-examples
Last update: 2026-05-29T14:27:44.000Z
Content:
---
title: "Monitoring examples \| Tinybird"
theme-color: "#171612"
description: "Practical examples for monitoring Tinybird workspaces, queries, jobs, and API Endpoints."
inkeep:version: "forward"
---
# Monitoring examples (https://www.tinybird.co/docs/forward/guides/monitoring-examples#monitoring-examples)
Use these examples to monitor Tinybird workspaces, queries, jobs, and Endpoints.
- Debug slow queries and failures with [system.query_log](/docs/forward/monitoring/query-log).
- Monitor Endpoint performance with [Endpoint performance](/docs/forward/monitoring/analyze-endpoints-performance).
- Track background jobs with [Workspace jobs](/docs/forward/monitoring/jobs).
- Monitor ingestion and platform operations with [Service Data Sources](/docs/forward/monitoring/service-datasources).
- Set up continuous checks with [Health checks](/docs/forward/monitoring/health-checks).
---
URL: https://www.tinybird.co/docs/forward/guides/migrate-from-shared-to-dedicated
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Migrate From Shared to Dedicated Infra \| Tinybird"
theme-color: "#171612"
description: "Migrate your Tinybird data from shared to dedicated infrastructure without disrupting live services."
inkeep:version: "forward"
---
# Migrate from shared to dedicated infrastructure (https://www.tinybird.co/docs/forward/guides/migrate-from-shared-to-dedicated#migrate-from-shared-to-dedicated-infrastructure)
This guide covers migrating data from shared to dedicated infrastructure without disrupting live services.
## Pre-migration checklist (https://www.tinybird.co/docs/forward/guides/migrate-from-shared-to-dedicated#pre-migration-checklist)
Before starting, verify the following:
- **Workspace setup**: Your dedicated Workspace is set up and accessible. You can create it yourself if:
- The creating user's email matches your company domain, for example, `@example.com`
- The Workspace is created in the region of your dedicated cluster
- **Access rights**: Your team can access both the old and new Workspaces
- **Downtime planning**: The process minimizes disruption but can fail. Plan accordingly
- **Customer Success**: Coordinate with your Customer Success Data Engineer so they can assist if needed
## Migration overview (https://www.tinybird.co/docs/forward/guides/migrate-from-shared-to-dedicated#migration-overview)
1. Replicate the project in the new Workspace
2. Recreate access Tokens
3. Export and load dimensional tables; duplicate any periodic update processes
4. Duplicate real-time ingestion, then backfill historical data for fact/event tables
5. Point your app to the new Endpoints and Tokens
6. Stop ingestion to the old Workspace
## Step 1: Replicate the Project (https://www.tinybird.co/docs/forward/guides/migrate-from-shared-to-dedicated#step-1-replicate-the-project)
- If your project resources (`.datasource`, `.pipe`, `.connection` files) are already in a Git repository, clone the repo locally and proceed to the deploy step
- If your resources are **not** in a Git repo, pull them from the existing Workspace first:
```
tb pull
```
- If you use **Connectors** (Kafka, S3, DynamoDB, Snowflake, etc.), ensure your `.connection` files use the same connection names as your current Workspace
- This is a good opportunity to clean up old versions and unnecessary resources before deploying
- Log in to the new Workspace, validate, then deploy:
```
tb login
tb deploy --check
tb deploy
```
## Step 2: Recreate Access Tokens (https://www.tinybird.co/docs/forward/guides/migrate-from-shared-to-dedicated#step-2-recreate-access-tokens)
Tokens cannot be migrated. Recreate them in the new Workspace with the same permissions. Use these Tokens to configure ingestion and consumption later.
See [Tokens](/docs/forward/core-concepts/tokens) and the [Tokens API reference](/docs/api-reference/token-api) for details.
If you use a Connector and can reingest all data from the original source, skip Steps 3 and 4.
## Step 3: Export dimensional tables (https://www.tinybird.co/docs/forward/guides/migrate-from-shared-to-dedicated#step-3-export-dimensional-tables)
Dimensional tables are smaller and updated periodically, making them simpler to migrate.
### If you have a periodic update process (https://www.tinybird.co/docs/forward/guides/migrate-from-shared-to-dedicated#if-you-have-a-periodic-update-process)
- Identify the Token used for updates in the source Workspace
- Create a matching Token in the destination Workspace
- Duplicate the update process, pointing to the new Workspace URL and Token
- **Same region, same cloud provider**: only the Token changes
- **Different region or cloud provider**: update both Token and URL
- Keep both processes running until migration is fully validated
### If you don't have a periodic update process (https://www.tinybird.co/docs/forward/guides/migrate-from-shared-to-dedicated#if-you-dont-have-a-periodic-update-process)
Export via Sinks, then ingest into the new Workspace. Repeat for every dimensional Data Source.
**Export the data:**
- Create a Pipe selecting all columns explicitly (avoid `SELECT *`)
- Create an on-demand Sink to write results to S3
- Split output files to stay within [ingestion limits](/docs/forward/ingest-data)
**Ingest the data** — choose one approach:
#### Option A: Direct URL ingestion (few files) (https://www.tinybird.co/docs/forward/guides/migrate-from-shared-to-dedicated#option-a-direct-url-ingestion-few-files)
```
curl \
-H "Authorization: Bearer " \
-X POST "https://api.tinybird.co/v0/datasources" \
-d "name=stocks" \
-d url='https://.../data.csv'
```
#### Option B: S3 connector (many files) (https://www.tinybird.co/docs/forward/guides/migrate-from-shared-to-dedicated#option-b-s3-connector-many-files)
- Configure an auxiliary Data Source to ingest from S3 on-demand with the filename pattern from your export
- Configure bucket access policies
- Monitor ingestion:
```
SELECT * FROM tinybird.sinks_ops_log ORDER BY timestamp DESC
```
- Copy data to the destination Data Source using a copy Pipe (select columns explicitly, match types)
## Step 4: Migrate fact and event tables (https://www.tinybird.co/docs/forward/guides/migrate-from-shared-to-dedicated#step-4-migrate-fact-and-event-tables)
Fact tables require maintaining ingestion continuity with no gaps or duplicates.
### 4a. Duplicate ingestion (https://www.tinybird.co/docs/forward/guides/migrate-from-shared-to-dedicated#4a-duplicate-ingestion)
Duplicate your ingestion process to also write to the new Workspace. Keep the original ingestion to the shared Workspace running. This creates two parallel active ingests: one continues writing to the shared Workspace, and one writes to the dedicated Workspace.
To configure the duplicate ingestion:
- **Same region, same cloud provider**: change only the Token
- **Different region or cloud provider**: change Token and URL. See [API base URLs](/docs/api-reference#regions-and-endpoints).
### 4b. Validate ingestion (https://www.tinybird.co/docs/forward/guides/migrate-from-shared-to-dedicated#4b-validate-ingestion)
Check via service Data Sources or the UI. Verify no data is in quarantine:
```
SELECT * FROM tinybird.datasources_ops_log
WHERE event_type = 'append'
ORDER BY timestamp DESC
```
### 4c. Identify the ingestion start point (https://www.tinybird.co/docs/forward/guides/migrate-from-shared-to-dedicated#4c-identify-the-ingestion-start-point)
Query the Data Source for the first ingested event (typically using a timestamp field). Use this timestamp as the cutoff for backfill.
### 4d. Export historical data (https://www.tinybird.co/docs/forward/guides/migrate-from-shared-to-dedicated#4d-export-historical-data)
- Create a Pipe querying the fact table up to the ingestion start point
- Create an on-demand Sink to export to S3
- Use **Parquet format** for large tables
- Partition into files < 1 GB
- Monitor the export:
```
SELECT * FROM tinybird.sinks_ops_log ORDER BY timestamp DESC
```
### 4e. Ingest backfill data (https://www.tinybird.co/docs/forward/guides/migrate-from-shared-to-dedicated#4e-ingest-backfill-data)
- Use the S3 connector to ingest into an auxiliary Data Source
- Copy to the destination Data Source using a copy Pipe (select columns explicitly, match types including `LowCardinality`, nullability, etc.)
- Monitor and validate as in Step 3
### 4f. Validate Endpoints (https://www.tinybird.co/docs/forward/guides/migrate-from-shared-to-dedicated#4f-validate-endpoints)
Confirm all Endpoints return accurate data. Materialized Views execute after the backfill copy, so data should be ready for consumption.
## Step 5: Switch over (https://www.tinybird.co/docs/forward/guides/migrate-from-shared-to-dedicated#step-5-switch-over)
- Point your applications to the new Workspace Endpoints
- **Same region, same cloud provider**: change only the access Token
- **Different region or cloud provider**: update both Token and URL
- Use the Tokens created in Step 2 with matching permissions
- Stop ingestion to the old Workspace once everything is confirmed working
## FAQ (https://www.tinybird.co/docs/forward/guides/migrate-from-shared-to-dedicated#faq)
**Memory limit or timeout errors occur during migration**
Contact Tinybird through your support or community Slack channel.
---
URL: https://www.tinybird.co/docs/forward/guides/migrate-from-postgres
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Migrate from PostgreSQL \| Tinybird"
theme-color: "#171612"
description: "Choose the right PostgreSQL migration path for Tinybird. Compare PostgreSQL table functions, CDC streams, and application writes."
inkeep:version: "forward"
---
# Migrate from PostgreSQL (https://www.tinybird.co/docs/forward/guides/migrate-from-postgres#migrate-from-postgresql)
PostgreSQL is usually where app data starts. Tinybird is where you move analytical workloads when PostgreSQL is doing too much: user-facing dashboards, high-concurrency API Endpoints, event analytics, operational reporting, or real-time aggregations over large tables.
There is no single migration path. The right choice depends on freshness requirements, table size, update patterns, operational constraints, and whether you already run Kafka-compatible infrastructure.
## Choose a migration path (https://www.tinybird.co/docs/forward/guides/migrate-from-postgres#choose-a-migration-path)
| Path | Use it for | Tradeoffs | Start here |
| --- | --- | --- | --- |
| PostgreSQL table function with Copy Pipes | Small tables, dimension tables, periodic imports, one-off backfills, prototypes. | Simple to operate. Pulls from PostgreSQL on a schedule, so freshness is bounded by the schedule. Large tables need chunking and indexed filters. Deletes need an explicit strategy. | [PostgreSQL table function](/docs/forward/ingest-data/table-functions/postgresql) |
| Change data capture to Kafka, then Tinybird Kafka connector | Real-time integrations, large event streams, production sync, low-latency analytics, updates and deletes. | More moving parts. Requires logical replication and a CDC tool such as Redpanda Connect, Debezium, Confluent, Estuary, or a managed equivalent. Best option when data must stay fresh. | [Postgres CDC with Redpanda Connect](/docs/forward/guides/postgres-cdc-with-redpanda-connect) |
| Application writes to Tinybird | New events generated by your app, migrations where PostgreSQL should stop being the analytics source. | Requires app changes. Avoids polling PostgreSQL and usually gives the cleanest event model for analytics. | [Events API](/docs/forward/ingest-data/events-api) |
| File export and batch import | Large historical backfills, archive imports, or migrations where database access is restricted. | Operationally simple, but not a live sync. You manage export, storage, and import cadence. | [Files](/docs/forward/ingest-data/local-file) |
Most migrations use more than one path:
1. Backfill historical data with the PostgreSQL table function or exported files.
2. Keep new data fresh with CDC or app writes.
3. Model updates and deletes in Tinybird with a `ReplacingMergeTree`, soft deletes, or a final materialized Data Source.
4. Move analytical queries from PostgreSQL to Tinybird Pipes and API Endpoints.
## Use the PostgreSQL table function for periodic imports (https://www.tinybird.co/docs/forward/guides/migrate-from-postgres#use-the-postgresql-table-function-for-periodic-imports)
Use the `postgresql()` table function when you want Tinybird to read directly from PostgreSQL and copy the result into a Data Source.
This is the simplest path when:
- The source table is small enough to copy within Copy Pipe limits.
- You can tolerate freshness measured in minutes or hours.
- The table is mostly append-only.
- The table has an indexed timestamp or monotonically increasing column for incremental filters.
- You are importing dimension tables that can be replaced on each run.
It is not the best default for high-volume transactional tables with constant updates and deletes. For those, use CDC.
### Example periodic import (https://www.tinybird.co/docs/forward/guides/migrate-from-postgres#example-periodic-import)
Create a target Data Source and a scheduled Copy Pipe that reads recent PostgreSQL rows.
- Tinybird CLI
- TypeScript SDK
- Python SDK
datasources/orders.datasource
```
SCHEMA >
`order_id` UInt64,
`customer_id` UInt64,
`status` String,
`amount` Float64,
`updated_at` DateTime
ENGINE "ReplacingMergeTree(updated_at)"
ENGINE_SORTING_KEY "order_id"
```
pipes/orders_from_postgres.pipe
```
NODE updated_orders
SQL >
%
SELECT
order_id,
customer_id,
status,
amount,
updated_at
FROM postgresql(
'postgres.example.com:5432',
'app',
'orders',
{{ tb_secret("PG_USERNAME") }},
{{ tb_secret("PG_PASSWORD") }}
)
WHERE updated_at >= now() - INTERVAL 10 MINUTE
TYPE copy
TARGET_DATASOURCE orders
COPY_MODE append
COPY_SCHEDULE */5 * * * *
```
Set credentials as secrets in each environment where the Copy Pipe runs:
```
tb --cloud secret set PG_USERNAME
tb --cloud secret set PG_PASSWORD
```
Deploy the project:
- Tinybird CLI
- TypeScript SDK
- Python SDK
```
tb --cloud deploy
```
For the full syntax, SDK examples, type mapping, and scheduling details, see [PostgreSQL table function](/docs/forward/ingest-data/table-functions/postgresql).
### Table function tradeoffs (https://www.tinybird.co/docs/forward/guides/migrate-from-postgres#table-function-tradeoffs)
The table function runs a query against PostgreSQL every time the Copy Pipe runs. Treat it like any other production query on your primary database.
- Add an index on the column used in the `WHERE` clause.
- Push filters down to PostgreSQL. Do not copy a large table every minute.
- Keep schedule frequency realistic for the source table size.
- Use `COPY_MODE replace` for small dimension tables.
- Use `COPY_MODE append` for event-like tables.
- Use `ReplacingMergeTree(updated_at)` when rows can be updated.
- Keep a strategy for deletes. Scheduled append jobs do not remove rows that disappeared from PostgreSQL.
For large historical imports, split the backfill into chunks by time or ID range. Each chunk should finish within PostgreSQL statement timeouts and Tinybird Copy Pipe limits.
## Use CDC for real-time PostgreSQL integration (https://www.tinybird.co/docs/forward/guides/migrate-from-postgres#use-cdc-for-real-time-postgresql-integration)
Use change data capture when PostgreSQL remains the source of truth and Tinybird needs to stay current with inserts, updates, and deletes.
The usual architecture is:
The CDC tool reads PostgreSQL logical replication or WAL changes and writes events to a Kafka-compatible topic. Tinybird consumes that topic with the [Kafka connector](/docs/forward/ingest-data/connectors/kafka).
Use CDC when:
- You need seconds-level freshness.
- You need to capture updates and deletes.
- The source table is large or changes continuously.
- You already use Kafka, Redpanda, Confluent, Debezium, Estuary, or another CDC platform.
- You want PostgreSQL query load to come from replication rather than repeated analytical scans.
CDC adds infrastructure. You need to operate or configure the CDC capture tool, Kafka-compatible storage, schemas, offsets, retries, and monitoring. The benefit is lower database impact and better freshness for production analytics.
### CDC event modeling in Tinybird (https://www.tinybird.co/docs/forward/guides/migrate-from-postgres#cdc-event-modeling-in-tinybird)
CDC events are not always the same shape as your final analytical table. Most tools emit metadata such as operation type, source timestamp, transaction position, and before or after values.
Common patterns in Tinybird:
- Store raw CDC events in a Kafka Data Source.
- Use Materialized Views to extract the current row shape into a query-friendly Data Source.
- Use `ReplacingMergeTree` with a version column for upserts.
- Represent deletes with a deleted flag or tombstone rows, then filter them from serving Pipes.
- Keep raw CDC events for replay and debugging.
For a practical setup with Redpanda Connect, see [Postgres CDC with Redpanda Connect](/docs/forward/guides/postgres-cdc-with-redpanda-connect). For the Tinybird side of the stream, see [Kafka connector](/docs/forward/ingest-data/connectors/kafka), [Confluent Cloud setup](/docs/forward/guides/confluent-cloud-setup), [AWS MSK setup](/docs/forward/guides/aws-msk-setup), and [Redpanda setup](/docs/forward/guides/redpanda-setup).
## Use app writes for new analytical events (https://www.tinybird.co/docs/forward/guides/migrate-from-postgres#use-app-writes-for-new-analytical-events)
If you control the app code and the data is event-like, consider writing new events directly to Tinybird instead of writing them to PostgreSQL first and extracting them later.
This works well for:
- Clickstream events.
- Product usage events.
- Logs and telemetry.
- Audit events.
- Analytics-only events that do not need PostgreSQL transactions.
The tradeoff is that you change the app write path. The benefit is a cleaner analytics model and less load on PostgreSQL.
Use the [Events API](/docs/forward/ingest-data/events-api), [TypeScript SDK quickstart](/docs/forward/quickstarts/typescript-sdk), or [Python SDK quickstart](/docs/forward/quickstarts/python-sdk) to write events directly.
## Backfill historical data (https://www.tinybird.co/docs/forward/guides/migrate-from-postgres#backfill-historical-data)
Even CDC migrations usually need a backfill. You have two common options:
- Use the PostgreSQL table function and run Copy Pipes over time or ID ranges.
- Export PostgreSQL data to files and import them into Tinybird.
For table function backfills, index the chunking column and check that PostgreSQL uses the index:
```
CREATE INDEX idx_orders_updated_at ON orders (updated_at);
VACUUM ANALYZE orders;
EXPLAIN
SELECT *
FROM orders
WHERE updated_at >= '2026-01-01 00:00:00'
AND updated_at < '2026-01-02 00:00:00';
```
Then run one Copy Pipe job per chunk. Start with small chunks, measure runtime, then increase the interval if both PostgreSQL and Tinybird have headroom.
When using CDC, confirm how your CDC tool handles initial snapshots. Some tools can snapshot existing rows and then continue streaming changes from the same consistent point. If your tool does that reliably, you may not need a separate table function backfill.
## Decide how to handle updates and deletes (https://www.tinybird.co/docs/forward/guides/migrate-from-postgres#decide-how-to-handle-updates-and-deletes)
PostgreSQL tables often have mutable rows. Tinybird Data Sources are optimized for analytical reads and append-heavy ingestion, so model mutability explicitly.
Use one of these patterns:
| Source behavior | Tinybird pattern | Notes |
| --- | --- | --- |
| Append-only events | `MergeTree` ordered by query filters | Best fit. Keep events immutable. |
| Updates by primary key | `ReplacingMergeTree(version_column)` sorted by primary key | Query with `FINAL` only when needed. Prefer materializing the latest state for serving paths. |
| Deletes | Soft delete flag or tombstone event | Filter deleted rows in serving Pipes, or materialize active rows into a final Data Source. |
| Late-arriving changes | Version column from source timestamp, WAL position, or CDC event timestamp | Make the version monotonic for each primary key. |
See [Deduplication strategies](/docs/forward/guides/deduplication-strategies) and [Materialized Views](/docs/forward/core-concepts/materialized-views) for the modeling details.
## Migration checklist (https://www.tinybird.co/docs/forward/guides/migrate-from-postgres#migration-checklist)
1. Identify the PostgreSQL tables used by analytical queries.
2. Classify each table as append-only, mutable dimension, or transactional current state.
3. Pick an ingestion path per table: table function, CDC, app writes, or file import.
4. Create Tinybird Data Sources with sorting keys based on your query filters.
5. Backfill historical data.
6. Start the live sync path.
7. Build Pipes and Endpoints for app queries.
8. Compare counts and key aggregates between PostgreSQL and Tinybird.
9. Move read traffic to Tinybird.
10. Remove old analytical queries from PostgreSQL when the new path is stable.
## See also (https://www.tinybird.co/docs/forward/guides/migrate-from-postgres#see-also)
- [PostgreSQL table function](/docs/forward/ingest-data/table-functions/postgresql)
- [Postgres CDC with Redpanda Connect](/docs/forward/guides/postgres-cdc-with-redpanda-connect)
- [Kafka connector](/docs/forward/ingest-data/connectors/kafka)
- [Events API](/docs/forward/ingest-data/events-api)
- [Evolve Data Sources](/docs/forward/guides/evolve-data-source)
---
URL: https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Migrate From Self-Hosted ClickHouse® \| Tinybird"
theme-color: "#171612"
description: "Learn how to migrate from self-hosted ClickHouse to Tinybird, from understanding the benefits, translating your SQL, deployment options, and integrating with your app."
inkeep:version: "forward"
---
# Migrate from ClickHouse self-hosted to Tinybird (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#migrate-from-clickhouse-self-hosted-to-tinybird)
If you're running ClickHouse self-hosted and looking for a more streamlined, production-ready analytics platform, use this guide to migrate to Tinybird. Learn how to translate your existing ClickHouse concepts, take advantage of Tinybird's unique features, and integrate the platform into your app architecture.
## Why migrate from ClickHouse self-hosted to Tinybird? (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#why-migrate-from-clickhouse-self-hosted-to-tinybird)
While ClickHouse is an excellent OLAP database, self-hosting comes with significant operational overhead. You're responsible for:
- **Infrastructure management**: Cluster operations, sharding, replication, and scaling
- **Security**: Authentication, authorization, and network security
- **Monitoring**: Performance tuning, query optimization, and observability
- **Maintenance**: Updates, backups, disaster recovery, and capacity planning
- **Development workflow**: Building APIs, handling authentication, and creating deployment pipelines
Tinybird abstracts away these operational complexities while providing additional capabilities that transform ClickHouse from a database into a complete analytics platform.
## Benefits of Tinybird over ClickHouse self-hosted (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#benefits-of-tinybird-over-clickhouse-self-hosted)
Tinybird offers several key advantages over managing ClickHouse yourself:
### Fully managed infrastructure (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#fully-managed-infrastructure)
Tinybird handles all the operational complexity of running ClickHouse at scale:
- **Automatic scaling**: Infrastructure scales transparently based on your workload
- **Zero cluster operations**: No need to manage shards, replicas, or cluster topology
- **Built-in monitoring**: Comprehensive observability with performance metrics and alerting
- **Managed updates**: Automatic updates and patches without downtime
### API-first architecture (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#api-first-architecture)
Transform SQL queries into production-grade REST APIs:
- **Instant API endpoints**: Turn any SQL query into a scalable HTTP API with authentication
- **Parameterized queries**: Build dynamic APIs with template variables and type validation
- **Built-in authentication**: Token-based and JWT authentication with fine-grained permissions
- **Rate limiting**: Built-in protection against abuse and cost control
### Real-time data ingestion (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#real-time-data-ingestion)
Simplified data ingestion with multiple options:
- **Events API**: High-throughput streaming ingestion for real-time events via HTTP
- **Native connectors**: Managed integrations for Kafka, S3, GCS, and more
- **Data Sources API**: Batch ingestion for CSV, Parquet, and NDJSON files
- **Schema evolution**: Handle schema changes without downtime
### Development workflow (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#development-workflow)
Modern development practices built in:
- **Local development**: Full local environment with Tinybird Local container
- **Version control**: Git-based workflow for all data assets
- **CI/CD integration**: Deploy changes through your existing pipelines
- **Testing and validation**: Built-in deployment validation and rollback capabilities
### BI and tool integration (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#bi-and-tool-integration)
Connect to your existing analytics tools:
- **ClickHouse HTTP interface**: Use existing ClickHouse drivers and tools
- **PostgreSQL compatibility**: Connect BI tools like Grafana, Metabase, and Tableau
- **Query API**: Direct SQL execution for existing applications
## ClickHouse to Tinybird concept translation (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#clickhouse-to-tinybird-concept-translation)
Tinybird has an opinionated approach to structuring analytics projects using two primary file types: `.datasource` files (tables) and `.pipe` files (queries, views, and materializations).
### Database → Workspace (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#database-workspace)
In ClickHouse, you organize tables within databases. In Tinybird:
- **Organization**: Top-level container for all your projects
- **Workspace**: Equivalent to a database, contains all your data sources and pipes
Learn more about [Tinybird Core Concepts](/docs/forward/core-concepts).
### Table → Data Source (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#table-data-source)
ClickHouse tables become Tinybird data sources, defined in `.datasource` files. Data sources in Tinybird define table schemas, engines, sorting keys, and partition keys. They may also contain references to [connectors](/docs/forward/core-concepts/connections) from which they receive data.
**ClickHouse `CREATE TABLE` statement:**
ClickHouse table creation
```
CREATE TABLE user_events
(
user_id UInt32 ,
event_type String,
timestamp DateTime,
properties JSON
)
ENGINE = MergeTree
ORDER BY (user_id, timestamp)
PARTITION BY toYYYYMM(timestamp)
```
**Tinybird `datasources/user_events.datasource`:**
user_events.datasource
```
DESCRIPTION >
User events tracking with properties for analytics
SCHEMA >
`user_id` UInt32 `json:$.user_id`,
`event_type` String `json:$.event_type`,
`timestamp` DateTime `json:$.timestamp`,
`properties` String `json:$.properties`
ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(timestamp)"
ENGINE_SORTING_KEY "user_id, timestamp"
# optional connector configuration
KAFKA_CONNECTION_NAME kafka_connection # The name of the .connection file
KAFKA_TOPIC topic_name
KAFKA_GROUP_ID {{ tb_secret("KAFKA_GROUP_ID") }}
```
Key differences:
- **JSONPath expressions**: Map JSON fields to columns automatically
- **Declarative schema**: Define structure without SQL DDL
- **Built-in description**: Document your data sources inline
- **Source connector definitions**: Define where the data comes from
Learn more about [data sources](/docs/forward/core-concepts/data-sources).
### Views and parametrized queries → Pipes (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#views-and-parametrized-queries-pipes)
ClickHouse views become Tinybird pipes, which can be used for to define API endpoints, materializations, internal queries, scheduled copies, sinks, or as building blocks for other pipes.
**ClickHouse parametrized view:**
ClickHouse parametrized view
```
CREATE VIEW user_activity_summary
AS SELECT
user_id,
event_type,
count() AS event_count,
min(timestamp) AS first_event,
max(timestamp) AS last_event
FROM user_events
WHERE timestamp >= {start_date:DateTime}
AND timestamp <= {end_date:DateTime}
AND user_id = {target_user:UInt32}
GROUP BY user_id, event_type
ORDER BY event_count DESC
```
**Tinybird `pipes/user_activity_summary.pipe`:**
user_activity_summary.pipe
```
DESCRIPTION >
User activity summary with date range and user filtering
NODE activity_summary
SQL >
%
SELECT
user_id,
event_type,
count() AS event_count,
min(timestamp) AS first_event,
max(timestamp) AS last_event
FROM user_events
WHERE timestamp >= {{DateTime(start_date)}}
AND timestamp <= {{DateTime(end_date)}}
AND user_id = {{UInt32(target_user)}}
GROUP BY user_id, event_type
ORDER BY event_count DESC
```
Key differences:
- **Template variables**: Use `%` token at the beginning of a query and `{{Type(variable)}}` syntax for parameters.
- **Advanced logic**: Support for `{% %}` [advanced templating functions](/docs/forward/query-data/query-parameters#advanced-templating-using-dynamic-parameters)
- **Node structure**: Organize complex queries into multiple nodes
- **Type safety**: Automatic type validation for parameters
Learn more about [pipes](/docs/forward/core-concepts/pipes).
### Materialized Views → Materialized Views (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#materialized-views-materialized-views)
ClickHouse incremental materialized views translate to Tinybird materialized views with target data sources.
**ClickHouse materialized view:**
ClickHouse materialized view
```
-- Target table
CREATE TABLE user_metrics_hourly
(
hour DateTime,
user_id UInt32,
event_count AggregateFunction(count),
unique_events AggregateFunction(uniq, String)
)
ENGINE = AggregatingMergeTree()
ORDER BY (hour, user_id);
-- Materialized view
CREATE MATERIALIZED VIEW user_metrics_hourly_mv
TO user_metrics_hourly
AS SELECT
toStartOfHour(timestamp) AS hour,
user_id,
countState() AS event_count,
uniqState(event_type) AS unique_events
FROM user_events
GROUP BY hour, user_id;
```
**Tinybird implementation:**
Target data source (`datasources/user_metrics_hourly.datasource`):
user_metrics_hourly.datasource
```
DESCRIPTION >
Hourly aggregated user metrics with state functions
SCHEMA >
`hour` DateTime,
`user_id` UInt32,
`event_count` AggregateFunction(count, UInt64),
`unique_events` AggregateFunction(uniq, String)
ENGINE "AggregatingMergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(hour)"
ENGINE_SORTING_KEY "hour, user_id"
```
Materialized view (`materializations/user_metrics_hourly.pipe`):
user_metrics_hourly.pipe
```
DESCRIPTION >
Incremental materialization of user metrics
NODE hourly_aggregation
SQL >
SELECT
toStartOfHour(timestamp) AS hour,
user_id,
countState() AS event_count,
uniqState(event_type) AS unique_events
FROM user_events
GROUP BY hour, user_id
TYPE MATERIALIZED
DATASOURCE user_metrics_hourly
```
Learn more about [materialized views](/docs/forward/core-concepts/materialized-views).
### Refreshable Materialized Views → Copy Pipes (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#refreshable-materialized-views-copy-pipes)
ClickHouse refreshable materialized views become Tinybird copy pipes with scheduling.
**ClickHouse refreshable materialized view:**
ClickHouse refreshable materialized view
```
CREATE MATERIALIZED VIEW daily_user_summary
REFRESH EVERY 1 HOUR
ENGINE = MergeTree()
ORDER BY (date, user_id)
AS SELECT
toDate(timestamp) AS date,
user_id,
count() AS total_events,
countDistinct(event_type) AS unique_event_types
FROM user_events
GROUP BY date, user_id;
```
**Tinybird implementation:**
Target Data Source (`datasources/daily_user_summary.datasource`):
daily_user_summary.datasource
```
DESCRIPTION >
Daily user activity summary
SCHEMA >
`date` Date,
`user_id` UInt32,
`total_events` UInt64,
`unique_event_types` UInt64
ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(date)"
ENGINE_SORTING_KEY "date, user_id"
```
Copy Pipe (`copies/daily_user_summary.pipe`):
daily_user_summary.pipe
```
DESCRIPTION >
Hourly copy job for daily user summaries
NODE daily_aggregation
SQL >
SELECT
toDate(timestamp) AS date,
user_id,
count() AS total_events,
countDistinct(event_type) AS unique_event_types
FROM user_events
GROUP BY date, user_id
TYPE COPY
TARGET_DATASOURCE daily_user_summary
COPY_SCHEDULE "0 * * * *" -- Every hour
```
Learn more about [copy pipes](/docs/forward/core-concepts/copy-pipes).
### Schema evolution (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#schema-evolution)
Evolving your schema in Tinybird is handled through deployments. Edit your `.datasource` or `.pipe` files and deploy changes:
- Tinybird CLI
- TypeScript SDK
- Python SDK
```
# Validate changes before deploying
tb deploy --check
# Deploy changes
tb deploy
```
Tinybird automatically handles backfills and schema migrations based on your changes.
Learn more about [Schema Evolution](/docs/forward/guides/evolve-data-source) and [Deployments](/docs/forward/core-concepts/deployments).
## Migrate data from ClickHouse (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#migrate-data-from-clickhouse)
After you translate schemas into Tinybird Data Sources, backfill historical data from ClickHouse. The usual path is:
1. Export each ClickHouse table, or each partition of a large table, to Parquet.
2. Create the target Data Source in Tinybird.
3. Import the Parquet files with `tb datasource append`, the Data Sources API, or an S3 connector.
4. Validate row counts and key aggregates before moving read traffic.
Prefer Parquet for backfills. It preserves column types better than CSV, compresses well, and avoids line-based parsing overhead for large tables.
### Export ClickHouse data to Parquet (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#export-clickhouse-data-to-parquet)
For small and medium tables, export the full table:
Export a ClickHouse table to Parquet
```
clickhouse-client \
--host \
--user \
--password \
--query "SELECT * FROM analytics.user_events FORMAT Parquet" \
> user_events.parquet
```
For large tables, export in chunks. Use the same partitioning or time column you use for queries:
Export one month of data
```
clickhouse-client \
--host \
--user \
--password \
--query "
SELECT *
FROM analytics.user_events
WHERE timestamp >= '2026-01-01 00:00:00'
AND timestamp < '2026-02-01 00:00:00'
FORMAT Parquet
" \
> user_events_2026_01.parquet
```
Chunking keeps each import job smaller, makes retries easier, and supports comparing counts per interval.
### Import local Parquet files (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#import-local-parquet-files)
Create the Tinybird Data Source first, then append each exported file:
Import a Parquet backfill file
```
tb --cloud datasource append user_events user_events_2026_01.parquet
```
You can also use the Data Sources API directly:
Import Parquet with the Data Sources API
```
curl \
-H "Authorization: Bearer " \
-X POST "https:///v0/datasources?mode=append&name=user_events" \
-F parquet=@user_events_2026_01.parquet
```
Use this path for one-off migrations and controlled backfills from your local machine or migration runner. See [Ingest data from files](/docs/forward/ingest-data/local-file).
### Import from S3 (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#import-from-s3)
For larger migrations, export Parquet files to S3 and ingest them from there. This avoids moving large files through a laptop and gives you a durable staging area for retries.
Use one of these options:
- Append individual S3 files with `tb datasource append` or the Data Sources API using a URL.
- Configure the [S3 Connector](/docs/forward/ingest-data/connectors/s3) when files continue landing in S3 and Tinybird should ingest them automatically.
For a one-off backfill with signed URLs:
Import a Parquet file from S3
```
tb --cloud datasource append user_events "https://.s3.amazonaws.com/backfills/user_events_2026_01.parquet?"
```
For recurring imports, define an S3 Connection and Data Source, then deploy it. Tinybird detects matching files in the bucket and ingests them into the Data Source. See [S3 connector](/docs/forward/ingest-data/connectors/s3).
### Validate the backfill (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#validate-the-backfill)
Compare counts and aggregates between ClickHouse and Tinybird for each chunk:
ClickHouse validation query
```
SELECT
toYYYYMM(timestamp) AS month,
count() AS rows,
uniqExact(user_id) AS users
FROM analytics.user_events
GROUP BY month
ORDER BY month
```
Tinybird validation query
```
tb --cloud sql "
SELECT
toYYYYMM(timestamp) AS month,
count() AS rows,
uniqExact(user_id) AS users
FROM user_events
GROUP BY month
ORDER BY month
"
```
When counts diverge, validate one chunk at a time. Check filters, time zones, nullable columns, and whether the source table contains duplicate or replacing rows that need `FINAL` in the ClickHouse export query.
## Integrate Tinybird into your app (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#integrate-tinybird-into-your-app)
### Data ingestion services (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#data-ingestion-services)
Tinybird provides multiple ways to ingest data. You do not need to set up ingestion infrastructure or use `INSERT` statements. Tinybird offers several native ingestion connectors and APIs.
#### Events API (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#events-api)
Best for real-time streaming data, and can be used to send data directly from the client to Tinybird. Send JSON or NDJSON events via HTTP POST:
Events API ingestion
```
curl -X POST "https:///v0/events?name=user_events" \
-H "Authorization: Bearer YOUR_TOKEN" \
-H "Content-Type: application/json" \
-d '{"user_id": 123, "event_type": "page_view", "timestamp": "2024-01-15 10:30:00", "properties": "{\"page\": \"/dashboard\"}"}'
```
Learn more about the [Events API](/docs/api-reference/events-api).
#### Data Sources API (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#data-sources-api)
For batch ingestion of CSV, Parquet, or NDJSON files:
Batch file ingestion
```
curl -X POST "https:///v0/datasources?mode=append&name=user_events" \
-H "Authorization: Bearer YOUR_TOKEN" \
-F "csv=@user_events.csv"
```
Learn more about the [Data Sources API](/docs/api-reference/datasource-api).
#### Connectors (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#connectors)
Managed integrations for popular data sources:
- **Kafka**: Real-time streaming with automatic schema detection
- **S3/GCS**: Automated file ingestion from cloud storage
Connectors handle authentication, schema mapping, and error handling automatically.
Learn more about [Connectors](/docs/forward/core-concepts/connections).
#### Table functions (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#table-functions)
Tinybird supports ClickHouse table functions for specialized data ingestion and transformation:
- **`url()`**: Load data directly from HTTP/HTTPS URLs
- **iceberg()**: Import data from Apache Iceberg tables
- **postgres()**: Import data from PostgreSQL databases
- **mysql()**: Import data from MySQL databases
Example using postgres() to import data from PostgreSQL:
pipes/import_from_postgres.pipe
```
DESCRIPTION >
Import user data from PostgreSQL database
NODE postgres_import
SQL >
SELECT
user_id,
email,
created_at,
last_login
FROM postgres(
'postgresql://username:password@hostname:5432/database',
'users'
)
WHERE created_at >= yesterday()
TYPE COPY
TARGET_DATASOURCE user_data
COPY_SCHEDULE "0 */6 * * *" -- Every 6 hours
```
This example creates a scheduled copy pipe that imports user data from a PostgreSQL table every 6 hours.
Learn more about [table functions](/docs/forward/ingest-data/table-functions).
### Query development (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#query-development)
Tinybird offers flexible options for querying data, from direct SQL to API endpoints.
#### Pipes vs Query API (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#pipes-vs-query-api)
**Pipes** are the default approach for production applications:
- **Type-safe parameters**: Automatic validation and conversion
- **Versioning**: Track changes and roll back if needed
- **Performance optimization**: Automatic query optimization and caching
- **Security**: Fine-grained access control with tokens
**Query API** for direct SQL execution:
- **Migration path**: Easy transition from existing ClickHouse applications
- **Ad-hoc analysis**: Quick data exploration and debugging
- **BI tools**: Direct integration with SQL-based tools
Example API endpoint from a pipe:
endpoints/user_activity.pipe
```
TOKEN analytics_api READ
DESCRIPTION >
User activity API endpoint with authentication
NODE user_metrics
SQL >
%
SELECT
user_id,
event_type,
count() AS event_count,
max(timestamp) AS last_activity
FROM user_events
WHERE user_id = {{UInt32(user_id)}}
AND timestamp >= {{DateTime(start_date, '2024-01-01 00:00:00')}}
GROUP BY user_id, event_type
ORDER BY event_count DESC
TYPE ENDPOINT
```
Access the endpoint:
```
curl "https:///v0/pipes/user_activity.json?user_id=123&start_date=2024-01-01" \
-H "Authorization: Bearer READ_TOKEN"
```
Learn more about [API Endpoints](/docs/forward/core-concepts/api-endpoints).
### BI integration using ClickHouse interface (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#bi-integration-using-clickhouse-interface)
Connect your existing BI tools using the [ClickHouse HTTP interface](/docs/forward/query-data/clickhouse-interface)
## Deployment options (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#deployment-options)
Tinybird offers flexible deployment options to meet your infrastructure and compliance requirements.
### Tinybird Cloud hosting (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#tinybird-cloud-hosting)
**Fully managed service** with global regions:
- **No infrastructure management**: Tinybird handles all operations
- **Global availability**: Multiple regions across AWS and GCP
- **Automatic scaling**: Infrastructure scales as needed based on selected plan
- **Enterprise features**: SSO, audit logs, advanced security controls
### Tinybird self-managed regions (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#tinybird-self-managed-regions)
**Bring Your Own Cloud** deployment for enhanced control:
- **Data sovereignty**: Keep data in your cloud environment
- **Network isolation**: Deploy within your VPC/VNet
- **Compliance**: Meet specific regulatory requirements
- **Custom configurations**: Tailored performance optimizations
**Benefits of self-managed regions:**
- Complete control over infrastructure and data location
- Integration with existing cloud commitments and reserved instances
- Direct connection to internal data sources without network egress
- Custom security policies and network configurations
**Deployment process:**
1. Register a self-managed region with Tinybird Cloud
2. Deploy Tinybird infrastructure in your cloud environment
3. Connect and manage through Tinybird CLI with `--region` flag
Self-managed regions are currently in beta. Contact Tinybird support for deployment assistance and requirements.
Learn more about [self-managed Regions](/docs/forward/core-concepts/self-managed).
## Open source projects using Tinybird (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#open-source-projects-using-tinybird)
Many successful open source projects have adopted Tinybird for their analytics needs:
- **[Ghost](https://ghost.org/)**: Publishing platform using Tinybird for member analytics and engagement metrics
- **[Papermark](https://www.papermark.io/)**: Document sharing platform with real-time analytics on document views and interactions
- **[Inbox Zero](https://www.getinboxzero.com/)**: Email management tool using Tinybird for email analytics and automation data
- **[LocalStack](https://localstack.cloud/)**: Cloud development platform using Tinybird for usage analytics and performance monitoring
- **[OpenStatus](https://www.openstatus.dev/)**: Website monitoring platform built on Tinybird for real-time uptime and performance analytics
- **[Dub](https://dub.co/)**: Link management platform using Tinybird for click analytics and link performance tracking
Each leverages Tinybird's real-time capabilities, API-first architecture, and managed infrastructure to focus on their core product instead of analytics infrastructure, while offering the flexibility between hosted and self-hosted deployments..
## Migration strategy (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#migration-strategy)
To successfully migrate from ClickHouse self-hosted to Tinybird:
### 1. Assessment and planning (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#1-assessment-and-planning)
- **Inventory your current setup**: Document tables, views, materialized views, and queries
- **Identify integration points**: Map how your app currently interacts with ClickHouse
- **Plan data migration**: Choose between full migration or gradual transition
### 2. Local development setup (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#2-local-development-setup)
```
# Install Tinybird CLI
curl https://tinybird.co | sh
# Authenticate
tb login
# Start local development environment
tb local start
```
### 3. Schema translation (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#3-schema-translation)
- Convert ClickHouse tables to `.datasource` files
- Transform views and materialized views to `.pipe` files
- Set up materialized views and copy pipes for aggregations
### 4. Data migration (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#4-data-migration)
- Export historical ClickHouse data to Parquet and import it into Tinybird. See [Migrate data from ClickHouse](https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#migrate-data-from-clickhouse).
- Use the Data Sources API or `tb datasource append` for one-off bulk imports.
- Use the S3 connector for large backfills or recurring file imports.
- Set up real-time ingestion with the Events API or connectors.
- Validate data consistency between systems.
### 5. Application integration (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#5-application-integration)
- Replace direct ClickHouse queries with Tinybird API endpoints or requests to the Query API
- Update authentication to use Tinybird tokens
- Configure BI tools to use Tinybird's ClickHouse interface
### 6. Production deployment (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#6-production-deployment)
- Deploy to Tinybird Cloud or set up self-managed region
- Configure monitoring and alerting via Tinybird service data sources
- Gradually shift traffic from ClickHouse to Tinybird
## Next steps (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#next-steps)
Ready to start your migration? Here are some immediate next steps:
1. **[Sign up for a free Tinybird account](https://cloud.tinybird.co/signup)** and explore the platform
2. **[Follow the quick start guide](/docs/forward/quickstarts/cli)** to understand core concepts
3. **[Install the CLI](/docs/forward/quickstarts/cli#install-the-cli)** and set up local development
4. **[Join the Tinybird community](https://www.tinybird.co/community)** for migration support and best practices
For complex migrations or enterprise requirements, [contact the Tinybird team](https://www.tinybird.co/contact-us) for personalized assistance and migration planning.
---
URL: https://www.tinybird.co/docs/forward/guides/migrate-from-classic
Last update: 2026-08-07T12:15:37.000Z
Content:
---
title: "Migrate from Tinybird Classic \| Tinybird"
theme-color: "#171612"
description: "Migrate your project from Tinybird Classic to Tinybird Forward."
inkeep:version: "forward"
---
# Migrate from Tinybird Classic (https://www.tinybird.co/docs/forward/guides/migrate-from-classic#migrate-from-tinybird-classic)
Tinybird Forward introduces a new way of working with your data projects, with changes to APIs and CLI that may be incompatible with Tinybird Classic. If you're starting a new project, see the [Get started](/docs/forward) guide.
Migration upgrades your existing workspace to Forward in place. Your tables and data remain in the workspace and are available in Forward after migration. `tb --cloud pull` downloads your project definitions as Forward-compatible datafiles. It does not download table contents. `tb migrate-to-forward` then updates the workspace’s project definition and deploys it using Forward.
## Why migrate to Forward (https://www.tinybird.co/docs/forward/guides/migrate-from-classic#why-migrate-to-forward)
Forward is a significant evolution of the Tinybird developer experience. Migrating unlocks the following benefits:
- **Deployments**: Project iteration is easier. You ship a new version of the project, and Tinybird handles all backfills and table exchanges automatically. No more `tb push datasource_v2` and copying data from `datasource_v1`.
- **Git-native deployments and CI/CD**: Every change ships through your repository. Use the built-in [CI/CD workflows](/docs/forward/development-workflow/cicd) for automated validation, [preview deployments](/docs/forward/core-concepts/deployments), and safe rollouts to production.
- **Local development**: Build and iterate against [Tinybird Local](/docs/forward/core-concepts/tinybird-local) with full parity to Tinybird Cloud. Test changes on your machine before they ever reach a shared environment. And easier to verify in your CI with the local image.
- **Faster, simpler testing**: [Fixture tests](/docs/forward/guides/test-your-project) make it easier to define realistic scenarios and catch regressions before deploy, replacing the heavier regression and data quality test workflows from Classic.
- **Cleaner project structure**: First-class [connection files](/docs/forward/dev-reference/datafiles/connection-files), [secrets](/docs/forward/dev-reference/commands/tb-secret), and [generic pipes](/docs/forward/core-concepts/pipes#create-generic-pipes) replace ad-hoc include files and per-environment glue, so projects stay portable and reviewable in code.
- **Modern, consistent CLI**: A redesigned `tb` command provides a unified workflow for local development, cloud deployments, and testing, with clearer commands and better feedback.
- **AI friendly**: Use the agent skills `npx skills add tinybirdco/tinybird-agent-skills` to teach Tinybird best practices to your coding agent.
- **Active development**: New features land in Forward first. Classic continues to receive critical fixes, but the roadmap (new connectors, deployment improvements, agent integrations) is Forward-only.
## Considerations before migrating (https://www.tinybird.co/docs/forward/guides/migrate-from-classic#considerations-before-migrating)
Before migrating your workspace from Tinybird Classic, understand these key differences in Forward:
- Development is file based. You edit `.pipe` and `.datasource` files in a Cloud Branch or locally using [Tinybird Local](/docs/forward/core-concepts/tinybird-local). Check [Development Workflow](/docs/forward/development-workflow) for more details.
- The following features have limitations or require changes:
| Feature | Status | Solution/Alternative |
| --- | --- | --- |
| BI Connector | Not supported | Use the [ClickHouse® HTTP Interface](/docs/forward/query-data/clickhouse-interface) instead. Most BI tools support ClickHouse® HTTP connections. |
| Include files | Not supported | Use [tb secret](/docs/forward/dev-reference/commands/tb-secret) for connector credentials and [generic pipes](/docs/forward/core-concepts/pipes#create-generic-pipes) to reuse query logic. See [Replace include files](https://www.tinybird.co/docs/forward/guides/migrate-from-classic#replace-include-files) for migration steps. |
| CI/CD workflows | Different commands | Forward uses different CLI commands. See [CI/CD](/docs/forward/development-workflow/cicd) for details. |
| Testing strategy | Different approach | Regression tests and data quality tests are not supported in Forward. Fixture tests have been enhanced for easier test creation and management. See [Test your project](/docs/forward/guides/test-your-project) for details. |
| Resource-scoped tokens with `:sql_filter` | Not supported | Remove all tokens using the `:sql_filter` suffix, for example, `DATASOURCES:READ:datasource_name:sql_filter`, before migrating. Use [JWTs](/docs/forward/core-concepts/jwt) instead. |
If these changes work for your use case, continue reading to learn how to migrate.
Migration is permanent and cannot be reversed. After deploying with Forward, you cannot switch your workspace back to Classic.
## Prepare to migrate (https://www.tinybird.co/docs/forward/guides/migrate-from-classic#prepare-to-migrate)
Both migration paths start from the same setup: install the Forward CLI, authenticate, and pull your project.
1
### Install the Tinybird Forward CLI (https://www.tinybird.co/docs/forward/guides/migrate-from-classic#install-the-tinybird-forward-cli)
Run the following command to install the Tinybird Forward CLI and the Tinybird Local container:
- For macOS and Linux
- For Windows
```
curl https://tinybird.co | sh
```
See [Tinybird Local](/docs/forward/core-concepts/tinybird-local#install-tinybird) for more information.
The migration requires Forward CLI version 4.6.4 or later. Check your version with `tb --version` and run `tb update` if it's lower.
**Managing CLI Versions**: Having both Tinybird Classic and Forward CLIs installed can cause version conflicts since both use the `tb` command. To avoid these conflicts, consider:
1. Using the uv Python package manager to keep both CLIs completely isolated (recommended):
```
# For Classic CLI
uvx --from tinybird-cli@latest tb
# For Forward CLI
uvx --from tinybird@latest tb
```
1. Creating aliases in your shell configuration:
```
# Add to .bashrc or .zshrc
alias tb-classic="path/to/classic/tb"
alias tb-forward="path/to/forward/tb"
```
1. Using separate virtual environments for each CLI version.
This ensures you use the correct CLI version for each operation during migration.
The following steps use the uv Python package.
2
### Authenticate to your workspace (https://www.tinybird.co/docs/forward/guides/migrate-from-classic#authenticate-to-your-workspace)
Authenticate to your workspace using the Classic CLI:
```
uvx --from tinybird-cli@latest tb auth --interactive
```
Follow the prompts to complete authentication.
3
### Pull your project (https://www.tinybird.co/docs/forward/guides/migrate-from-classic#pull-your-project)
Pull your datafiles from Tinybird Cloud with the Forward CLI:
```
tb --cloud pull
```
For connector-backed projects, this writes a `.connection` file per connector under `connections/` and the data sources under `datasources/`, and for S3, GCS, Kafka, and DynamoDB it stores the credentials as [secrets](/docs/forward/dev-reference/commands/tb-secret) referenced with `tb_secret(...)`. If a secret of the same name already exists, the pull reuses it and prints a warning so you can confirm the value matches before deploying.
## Migrate to Forward (https://www.tinybird.co/docs/forward/guides/migrate-from-classic#migrate-to-forward)
`tb migrate-to-forward` completes the migration end to end. It works for projects with and without connectors (S3, GCS, Kafka, DynamoDB). If it reports changes it can't reconcile, contact [Tinybird support](/docs/forward/support).
1
### Audit the pulled connector files (https://www.tinybird.co/docs/forward/guides/migrate-from-classic#audit-the-pulled-connector-files)
If your project doesn't use connectors, skip to the next step.
Open each pulled `.connection` file and confirm it has the connector type, credentials reference, and region. The exact fields depend on the connector:
- S3
- GCS
- Kafka
- DynamoDB
An access-key connection references the secret key with `tb_secret(...)` and keeps the access key ID and region inline:
connections/s3_events.connection
```
TYPE s3
S3_ACCESS_KEY AKIAEXAMPLE
S3_SECRET {{ tb_secret("s3_secret_s3_events") }}
S3_REGION eu-west-1
```
An IAM-role connection references the role ARN with `tb_secret(...)`:
connections/s3_events.connection
```
TYPE s3
S3_ARN {{ tb_secret("s3_role_arn_s3_events") }}
S3_REGION eu-west-3
```
Then open each connector-backed `.datasource` file and confirm it references the connection by name and keeps the import settings. For DynamoDB, confirm it keeps the `ReplacingMergeTree` settings that updates and deletes rely on:
datasources/orders.datasource
```
ENGINE "ReplacingMergeTree"
ENGINE_SORTING_KEY ", "
ENGINE_VER "_timestamp"
ENGINE_IS_DELETED "_is_deleted"
IMPORT_CONNECTION_NAME 'ddb_orders'
IMPORT_TABLE_ARN 'arn:aws:dynamodb:us-east-1:123456789012:table/orders'
IMPORT_EXPORT_BUCKET 'my-dynamodb-export-bucket'
```
For DynamoDB, use only the S3 bucket name in `IMPORT_EXPORT_BUCKET`, not an `s3://` URI. Treat `IMPORT_TABLE_ARN` and `IMPORT_EXPORT_BUCKET` as immutable: the first Forward deployment should preserve the existing DynamoDB binding, not change it.
2
### Run the migration (https://www.tinybird.co/docs/forward/guides/migrate-from-classic#run-the-migration)
Run the migration command:
```
tb migrate-to-forward
```
This command deletes branches and releases, runs a deploy check, generates a `tinybird.config.json` file, switches the workspace to Forward, and deploys it. After it finishes, query a few endpoints and confirm ingestion continues.
If you have CI/CD configured, push the changes to your repository to complete the migration. See [CI/CD](/docs/forward/development-workflow/cicd) for details on the CI workflow, CD workflow, and creating preview deployments.
If your workspace uses AWS PrivateLink for private networking, the migration doesn't change it. Private networking is configured at the workspace and region level, not in the datafiles you migrate, and the workspace keeps the same region. Connectors reach AWS through the same IAM role and endpoints before and after migration.
The migration is complete. Your project continues working as expected. You do not need to change your Tokens, Endpoint URLs, or anything else.
## Common migration errors (https://www.tinybird.co/docs/forward/guides/migrate-from-classic#common-migration-errors)
Common errors and changes include (but are not limited to):
### Replace include files (https://www.tinybird.co/docs/forward/guides/migrate-from-classic#replace-include-files)
Include files are not supported in Forward. The fix depends on your use of include files:
- If you use include files to store secrets, use [tb secret](/docs/forward/dev-reference/commands/tb-secret) to set secrets in your local and cloud environments.
- If you use include files to reuse query logic, you can create [generic pipes](/docs/forward/core-concepts/pipes#create-generic-pipes) and reference them in your endpoint pipes. For example:
reusable_filters.pipe
```
NODE apply_params
SQL >
%
SELECT * FROM my_datasource
WHERE
tenant_id = {{ String(tenant) }}
AND date BETWEEN {{ Date(start_date) }} AND {{ Date(end_date) }}
```
my_endpoint.pipe
```
NODE endpoint
SQL >
%
SELECT * FROM reusable_filters
TYPE endpoint
```
## Next steps (https://www.tinybird.co/docs/forward/guides/migrate-from-classic#next-steps)
- Learn about working with Forward in the [Forward documentation](/docs/forward).
---
URL: https://www.tinybird.co/docs/forward/guides/lambda-architecture
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Build a lambda architecture in Tinybird \| Tinybird"
theme-color: "#171612"
description: "In this guide, you'll learn a useful alternative processing pattern for when the typical Tinybird flow doesn't fit."
inkeep:version: "forward"
---
# Build a lambda architecture in Tinybird (https://www.tinybird.co/docs/forward/guides/lambda-architecture#build-a-lambda-architecture-in-tinybird)
Use this alternative processing pattern when the typical Tinybird flow doesn't fit.
The typical flow is Data Source --> incremental transformation through Materialized Views --> API Endpoint publication. When Materialized Views don't fit your processing requirements, use Copy Pipes to create intermediate Data Sources that keep your Endpoints performant.
## The ideal Tinybird flow (https://www.tinybird.co/docs/forward/guides/lambda-architecture#the-ideal-tinybird-flow)
You ingest data (usually streamed in, but can also be in batch), transform it using SQL, and serve the results of the queries via [parameterizable](/docs/forward/query-data/query-parameters) Endpoints. Tinybird provides freshness, low latency, and high concurrency: Your data is ready to be queried as soon as it arrives.
Data flow with Data Source and Endpoint
Sometimes, transforming the data at query time isn't ideal. Some operations - doing aggregations, or extracting fields from JSON - are better if done at ingest time, then you can query that prepared data. [Materialized Views](/docs/forward/core-concepts/materialized-views) are perfect for this kind of situation. They're triggered at ingest time and create intermediate tables (data sources in Tinybird lingo) to keep your Endpoints performance super efficient.
Data flow with Data Source, MV, and Endpoint
The best practice for this approach is usually having a Materialized View (MV) per use case:
Materialized Views for different use cases
If your use case fits in these first two paragraphs, stop reading. No need to over-engineer it.
## When the ideal flow isn't enough (https://www.tinybird.co/docs/forward/guides/lambda-architecture#when-the-ideal-flow-isnt-enough)
There are some cases where you may need intermediate Data Sources (tables) and Materialized Views don't fit.
- Most common: Things like Window Functions where you need to check the whole table to make calculations.
- Fairly common: Needing an Aggregation MV over a deduplication table (ReplacingMergeTree).
- Scenarios where Materialized Views fit but aren't super efficient (hey `uniqState`).
- And lastly, one of the hardest problems in syncing OLTP and OLAP databases: Change data capture (CDC).
Want to know more about *why* Materialized Views don't work in these cases? [Read the docs.](/docs/forward/core-concepts/materialized-views#limitations)
For example, consider the *Aggregation Materialized Views over deduplication DS* scenario.
Deduplication in Tinybird happens asynchronously, during merges, which you can't force in Tinybird. That's why you always have to add `FINAL` or the `-Merge` combinator when querying.
Materialized Views only see the block of data being processed. When materializing an aggregation, the Materialized View processes any new row, whether it has a new or duplicate ID. That's why this pattern fails.
Aggregating MV over deduplication DS doesn't work as expected
## Solution: Use an alternative architecture with Copy Pipes (https://www.tinybird.co/docs/forward/guides/lambda-architecture#solution-use-an-alternative-architecture-with-copy-pipes)
Tinybird has another kind of Pipe that helps here: [Copy Pipes](/docs/forward/core-concepts/copy-pipes).
At a high level, they're a helpful `INSERT INTO SELECT`, and they can be set to execute following a cron expression. You write your query, and (either on a recurring basis or on demand), the Copy Pipe appends the result in a different table.
So, in this example, you can have a clean, deduplicated snapshot of your data, with the correct Sorting Keys, and can use it to materialize:
Copy Pipes to the rescue
## Avoid loss of freshness (https://www.tinybird.co/docs/forward/guides/lambda-architecture#avoid-loss-of-freshness)
*"But if you recreate the snapshot every hour, day, or another interval, aren't you losing freshness?"* Yes. That's when the lambda architecture comes into play:
Lambda/Kappa Architecture
Combine the already-prepared data with the same operations over the fresh data being ingested at that moment. This provides higher performance despite complex logic over both fresh and old data.
## Next steps (https://www.tinybird.co/docs/forward/guides/lambda-architecture#next-steps)
- Read more about [Copy Pipes](/docs/forward/core-concepts/copy-pipes).
- Read more about [Materialized Views](/docs/forward/core-concepts/materialized-views).
---
URL: https://www.tinybird.co/docs/forward/guides/integrate-with-open-source
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Add Tinybird to your Open Source Project \| Tinybird"
theme-color: "#171612"
description: "Tips and best practices for adding Tinybird to your open source project and provide your users flexible deployment options."
inkeep:version: "forward"
---
# Add Tinybird to your open source project (https://www.tinybird.co/docs/forward/guides/integrate-with-open-source#add-tinybird-to-your-open-source-project)
Tinybird is a managed ClickHouse® service with flexible deployment options used by many open source developers to integrate ClickHouse®-based analytics features into their open source projects.
This guide walks you through best practices for adding Tinybird into your open source project.
[Apply to the Tinybird Open Source Program](https://www.tinybird.co/open-source-program) to get plan credits, swag, and support integrating Tinybird into your open source project.
## Including Tinybird in your open source repository (https://www.tinybird.co/docs/forward/guides/integrate-with-open-source#including-tinybird-in-your-open-source-repository)
Tinybird resources are represented as plaintext files that can be versioned alongside your app code. When you initialize a Tinybird project with `tb init`, it creates a default directory structure with folders for different resource types.
### Default project structure (https://www.tinybird.co/docs/forward/guides/integrate-with-open-source#default-project-structure)
Tinybird generates the following folder structure:
```
├── .tinyb # Project configuration and authentication
├── connections/ # Connection files
├── datasources/ # Data source definitions (.datasource)
│ └── user_actions.datasource
├── endpoints/ # API endpoint pipes (.pipe)
│ ├── daily_metrics.pipe
│ └── user_stats.pipe
├── pipes/ # Processing pipes (.pipe)
├── materializations/ # Materialized views
├── fixtures/ # Test data files
│ └── sample_data.ndjson
│ └── sample_data.sql
└── tests/ # Test suites
└── daily_metrics.yaml
```
You can include these Tinybird resources as a part of your open source distribution so that self-hosters and contributors can deploy the Tinybird project using their preferred deployment model.
Refer to the [reference implementations](https://www.tinybird.co/docs/forward/guides/integrate-with-open-source#reference-implementations) for example approaches to organizing Tinybird resources within your open source codebase.
### Documenting Tinybird (https://www.tinybird.co/docs/forward/guides/integrate-with-open-source#documenting-tinybird)
Include a `TINYBIRD.md` or a `README.md` within your `/tinybird` folder to explain Tinybird resources to contributors. The `tb init` command can generate starter project files and a structure that you can build on.
## Development workflow for contributors and self-hosters (https://www.tinybird.co/docs/forward/guides/integrate-with-open-source#development-workflow-for-contributors-and-self-hosters)
Tinybird projects follow standard Git-based development workflows. Contributors can pull repositories, run [Tinybird Local](/docs/forward/core-concepts/tinybird-local) for development, and use agent skills in their coding assistants to understand and evolve Tinybird resources.
```
npx skills add tinybirdco/tinybird-agent-skills
```
As with any software codebase, the typical workflow is: clone → install → build → develop → test → deploy.
### CI/CD and testing (https://www.tinybird.co/docs/forward/guides/integrate-with-open-source#cicd-and-testing)
Tinybird automatically generates CI/CD templates for GitHub and GitLab to automate testing and deployment.
You can also use Tinybird's testing framework to generate unit tests and data quality tests to detect breaking changes during CI/CD.
Learn more:
- [CI/CD workflows](/docs/forward/development-workflow/cicd)
- [Test your project](/docs/forward/guides/test-your-project)
## Select your query interface (https://www.tinybird.co/docs/forward/guides/integrate-with-open-source#select-your-query-interface)
Developers often choose Tinybird because of the simplified abstractions it provides over ClickHouse®, including its concept of pipes that can be instantly deployed as API endpoints.
Endpoint pipes are the most common way to query data in Tinybird, however Tinybird offers three different query interfaces, each with different trade-offs for open source projects.
### API Endpoints (https://www.tinybird.co/docs/forward/guides/integrate-with-open-source#api-endpoints)
Deploy pipes as hosted API endpoints with type-safe parameters and built-in monitoring. This provides the highest level of abstraction and separation of concerns, as query logic - including filter parameters - is maintained entirely within Tinybird pipe files.
An example endpoint pipe configuration:
```
TOKEN analytics_read READ
DESCRIPTION >
User activity metrics for the last 30 days
NODE user_activity
SQL >
%
SELECT
user_id,
event_type,
count() AS event_count,
max(timestamp) AS last_activity
FROM user_events
WHERE user_id = {{UInt32(user_id)}}
AND timestamp >= {{DateTime(start_date, '2024-01-01 00:00:00')}}
GROUP BY user_id, event_type
ORDER BY event_count DESC
TYPE ENDPOINT
```
```
// Calling the endpoint
const response = await fetch(
`https://api.tinybird.co/v0/pipes/user_activity.json?user_id=123&start_date=2024-01-01`,
{
headers: {
'Authorization': 'Bearer tb_your_read_token'
}
}
);
const data = await response.json();
```
Learn more:
- [Endpoints](/docs/forward/core-concepts/api-endpoints)
- [Pipes](/docs/forward/core-concepts/pipes)
### Query API (https://www.tinybird.co/docs/forward/guides/integrate-with-open-source#query-api)
Use the Query API to execute arbitrary SQL queries directly against your Tinybird Data Sources and Pipes. With this approach, queries are defined directly within your app code and passed to the Tinybird API. This provides some portability for open source projects that want to give self-hosters the option to port the analytics deployment to open source ClickHouse®, but it still relies on Tinybird APIs.
```
// Direct SQL query
const queryData = {
q: `
SELECT
event_type,
count() as total_events,
uniq(user_id) as unique_users
FROM user_events
WHERE timestamp >= today() - INTERVAL 7 DAY
GROUP BY event_type
ORDER BY total_events DESC
FORMAT JSON
`
};
const response = await fetch('https://api.tinybird.co/v0/sql', {
method: 'POST',
headers: {
'Authorization': 'Bearer tb_your_token',
'Content-Type': 'application/x-www-form-urlencoded'
},
body: new URLSearchParams(queryData)
});
```
Learn more: [Query API](/docs/api-reference/query-api)
### ClickHouse® HTTP interface (https://www.tinybird.co/docs/forward/guides/integrate-with-open-source#clickhouse-http-interface)
You can connect to the underlying ClickHouse® databases in your Tinybird Workspace using the native ClickHouse® HTTP interface. Query Tinybird with existing ORMs, language clients, and BI tools. This approach provides maximum portability and perhaps the easiest migration path to open source ClickHouse®, though it eliminates the convenient query abstractions that Tinybird provides.
```
Protocol: HTTPS
Host: clickhouse..tinybird.co # (adjust for your region)
Port: 443
Username:
Password:
```
```
// Using @clickhouse/client (HTTP interface)
import { createClient } from '@clickhouse/client'
const client = createClient({
host: 'https://clickhouse..tinybird.co',
username: 'optional_workspace_name', // optional
password: 'tb_your_read_token'
})
const resultSet = await client.query({
query: 'SELECT * FROM user_events LIMIT 10',
format: 'JSON'
})
const data = await resultSet.json()
console.log(data)
```
### Choosing the right query interface (https://www.tinybird.co/docs/forward/guides/integrate-with-open-source#choosing-the-right-query-interface)
Each query interface offers different tradeoffs: Endpoints provide standardization but may be less portable for those who want to self-host ClickHouse®. The Query API offers a middle ground with familiar SQL syntax defined in the app code while still using Tinybird APIs, and the ClickHouse® HTTP Interface maximizes portability through a raw connection to the underlying databases.
Many projects use a combination based on their specific needs - for example, using ClickHouse® Interface for BI tools, Query API for ad hoc queries, and Endpoints for common query patterns in user-facing analytics.
**Ingestion APIs**: All data ingestion into Tinybird uses Tinybird-specific APIs (Events API, Data Sources API, Connectors). If you plan to support ClickHouse® open source as an alternative, your project must implement separate ingestion patterns for pure ClickHouse® deployments.
## Choose your deployment model (https://www.tinybird.co/docs/forward/guides/integrate-with-open-source#choose-your-deployment-model)
Open source projects integrating Tinybird have several deployment options. The most common pattern for open source projects:
- Use **Tinybird Cloud** to power analytics for the hosted or commercial service.
- Offer **self-hosting users** the flexibility to choose between Tinybird Cloud, [Tinybird Self-Managed Regions](/docs/forward/core-concepts/self-managed), or, perhaps, open source ClickHouse®.
Learn more: [Self-managed](/docs/forward/core-concepts/self-managed)
### Considerations for open source ClickHouse® deployments (https://www.tinybird.co/docs/forward/guides/integrate-with-open-source#considerations-for-open-source-clickhouse-deployments)
If your [query interface](https://www.tinybird.co/docs/forward/guides/integrate-with-open-source#select-your-query-interface) choice supports it, self-hosting users can potentially deploy your project with open source ClickHouse® as their analytics database.
Note, however, there are some requirements you must consider:
- **Alternative ingestion patterns** - ClickHouse®-native ingestion instead of Tinybird APIs
- **Custom deployment logic** - Handling ClickHouse® cluster setup, scaling, and maintenance
- **Limited features** - No access to Tinybird's pipes, endpoints, or management tools
- **Additional complexity** - Users manage ClickHouse® operations, security, and monitoring
- **Zero-copy support** - Open source ClickHouse® currently doesn't support zero-copy replication, which can impact storage cost and performance for self-hosters.
Some open source projects might offer flexibility to self-hosters. Tinybird's ClickHouse® HTTP interface can help you design this flexibility into your open source project with these considerations in mind.
Read the [guide on migrating from ClickHouse® to Tinybird](/docs/forward/guides/migrate-from-clickhouse) to better understand the mapping of concepts between Tinybird and ClickHouse®.
## Reference implementations (https://www.tinybird.co/docs/forward/guides/integrate-with-open-source#reference-implementations)
The following open source projects demonstrate different approaches to integrating Tinybird and can be used as reference implementations:
- [Ghost](https://github.com/TryGhost/Ghost): a popular open source publishing platform that uses Tinybird for content and audience analytics.
- [Dub](https://github.com/dubinc/dub): an open source link attribution platform that uses Tinybird to capture link click events and build real-time analytics for link creators.
- [OpenStatus](https://github.com/openstatusHQ/openstatus): an open source synthetic monitoring platform that uses Tinybird to store and analyze uptime monitoring data.
- [Inbox Zero](https://github.com/elie222/inbox-zero): an open source email management tool that uses Tinybird to analyze email patterns and monitor LLM calls.
- [Papermark](https://github.com/mfts/papermark): an open source document sharing platform that uses Tinybird to track document views, engagement metrics, and sharing analytics.
---
URL: https://www.tinybird.co/docs/forward/guides/ingest-with-estuary
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Ingest with Estuary \| Tinybird"
theme-color: "#171612"
description: "In this guide, you'll learn how to use Estuary to push data streams to Tinybird."
inkeep:version: "forward"
---
# Ingest with Estuary (https://www.tinybird.co/docs/forward/guides/ingest-with-estuary#ingest-with-estuary)
Use Estuary to push data streams to Tinybird.
[Estuary](https://estuary.dev/) is a real-time ETL tool that allows you capture data from a range of source, and push it to a range of destinations. Using Estuary's Dekaf, you can connect Tinybird to Estuary as if it was a Kafka broker - meaning you can use Tinybird's native Kafka Connector to consume data from Estuary.
[Read more about Estuary Dekaf.](https://docs.estuary.dev/reference/Connectors/materialization-connectors/Dekaf/)
## Prerequisites (https://www.tinybird.co/docs/forward/guides/ingest-with-estuary#prerequisites)
- An Estuary account and collection
- A Tinybird account and workspace
## Connect to Estuary (https://www.tinybird.co/docs/forward/guides/ingest-with-estuary#connect-to-estuary)
In Estuary, create a new Dekaf materialization to use for the Tinybird connection.
You can create it from the [Estuary destinations tab](https://dashboard.estuary.dev/materializations/create). You have all the details on the [Tinybird Dekaf Estuary docs page.](https://docs.estuary.dev/reference/Connectors/materialization-connectors/Dekaf/tinybird/)
In your Tinybird workspace, create a new data source and use the [Kafka Connector](/docs/forward/ingest-data/connectors/kafka).
To configure the connection details, use the following settings (these can also be found in the [Estuary Dekaf docs](https://docs.estuary.dev/guides/dekaf_reading_collections_from_kafka/#connection-details)).
- Bootstrap servers: `dekaf.estuary-data.com`
- SASL Mechanism: `PLAIN`
- SASL Username: Your materialization task name, such as `YOUR-ORG/YOUR-PREFIX/YOUR-MATERIALIZATION`
- SASL Password: Auth token provided when you the Dekaf materialization was created on Estuary
Tick the `Decode Avro messages with Schema Register` box, and use the following settings:
- URL: `https://dekaf.estuary-data.com`
- Username: The same Materialization name from the preceding step, `YOUR-ORG/YOUR-PREFIX/YOUR-MATERIALIZATION`
- Password: The same Auth token created on the Dekaf materialization from the preceding step
Select **Next** and you see a list of topics. These topics are the collections you have in Estuary. Select the collection you want to ingest into Tinybird, and select **Next**.
Configure your consumer group as needed.
Finally, you see a preview of the data source schema. Feel free to make any modifications as required, then select **Create data source**.
This completes the connection with Estuary, and new data from the Estuary collection arrives in your Tinybird data source in real-time.
If you need *support for deletions*, check the [configuring support for deletions](https://docs.estuary.dev/reference/Connectors/materialization-connectors/Dekaf/tinybird) section on Estuary docs.
## Handling updates and deletes (https://www.tinybird.co/docs/forward/guides/ingest-with-estuary#handling-updates-and-deletes)
When capturing change data that includes updates and deletes, you need to deduplicate the data in Tinybird to maintain the latest state.
There are several strategies to deduplicate data in your data source, but with Estuary, the recommended approach is to use a [ReplacingMergeTree engine](/docs/sql-reference/engines/replacingmergetree) with appropriate settings and the `FINAL` modifier.
Do not build materialized views with an AggregatingMergeTree on top of a ReplacingMergeTree. The target data source always contains duplicates due to the incremental nature of materialized views.
## Learn more (https://www.tinybird.co/docs/forward/guides/ingest-with-estuary#learn-more)
For a complete step-by-step tutorial on setting up CDC with PostgreSQL, Estuary Flow, and Tinybird, see the [From CDC to real-time analytics with Tinybird and Estuary](https://www.tinybird.co/blog-posts/cdc-real-time-analytics-estuary-dekaf) blog post.
---
URL: https://www.tinybird.co/docs/forward/guides/ingest-vercel-logdrains
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Send Vercel log drains to Tinybird \| Tinybird"
theme-color: "#171612"
description: "Learn how to send Vercel events to Tinybird using webhooks and the Events API."
inkeep:version: "forward"
---
# Send Vercel log drains to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-vercel-logdrains#send-vercel-log-drains-to-tinybird)
[Vercel](https://vercel.com/) is a platform for building and deploying web applications. By integrating Vercel with Tinybird, you can analyze your Vercel events in real time and enrich it with other data sources.
Some common use cases for sending Vercel Log Drains to Tinybird include:
1. Analyze logs from your applications.
2. Monitor logs from your applications.
3. Create custom analytical dashboards.
4. Build an alert system based on logging patterns.
Read on to learn how to send logs from Vercel to Tinybird.
## Before you start (https://www.tinybird.co/docs/forward/guides/ingest-vercel-logdrains#before-you-start)
Before you connect Vercel Log Drains to Tinybird, ensure:
- You have a Vercel account.
- You have a Tinybird workspace.
## Connect Vercel to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-vercel-logdrains#connect-vercel-to-tinybird)
1. Choose your team scope on the dashboard, and go to **Team Settings** > **Log Drains**.
2. Select the **Projects** to send logs to Tinybird.
3. Select **Sources** you want to send logs to Tinybird.
4. Select **NDJSON** as Delivery Format.
5. Select **Environments** and **Sampling Rate**.
6. In your Tinybird project, create a data source called `vercel_logs`. You can follow this [schema](https://github.com/tinybirdco/tinynest/blob/main/tinybird/datasources/vercel_logs.datasource):
```
SCHEMA >
`event_time` DateTime `json:$.tinybirdIngestTime` DEFAULT now(),
`event_type` String `json:$.type` DEFAULT 'unknown',
`event` JSON `json:$` DEFAULT '{}'
ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(event_time)"
ENGINE_SORTING_KEY "event_time"
```
The proxy column is a JSON string. Use the [JSONExtract](/docs/sql-reference/functions/json-functions) functions to extract the data you need in your pipes.
1. From Tinybird Cloud, copy a token with privileges to append to the data source you created. You can use the admin token or create one with the required scope.
2. Back in Vercel, paste the Events API URL in your Log Drains Endpoint. Use the query parameter `name` to match the name of the data source you created in Tinybird.
Log Drains webhook needs to be verified by Vercel. You can do this by adding the `x-vercel-verify` parameter to the request.
```
https:///v0/events?name=vercel_logs&x-vercel-verify=
```
The API host in the following examples must match your Workspace's region. See the full list of [regions and hosts](/docs/api-reference#regions-and-endpoints)
1. Select **Custom Headers**, add `Authorization` with the value `Bearer ` and select **Add**.
2. Select **Verify** and optionally use **Test Log Drain** from Vercel to check data gets to the `vercel_logs` data source in Tinybird.
3. You're done. Any of the Vercel Log Drains you selected is automatically sent to Tinybird through the [Events API](/docs/forward/ingest-data/events-api).
Check the status of the from the **Log** tab in the Tinybird `vercel_logs` data source.
## Vercel Logs Explorer template (https://www.tinybird.co/docs/forward/guides/ingest-vercel-logdrains#vercel-logs-explorer-template)
Use the [Vercel Logs Explorer Template](https://www.tinybird.co/templates/vercel-log-drains) to bootstrap a multi-tenant, user-facing logs explorer for your Vercel account. You can fork it and make it your own.
## See also (https://www.tinybird.co/docs/forward/guides/ingest-vercel-logdrains#see-also)
- [Events API](/docs/forward/ingest-data/events-api)
- [Vercel Log Drains](https://vercel.com/docs/log-drains/log-drains-reference)
---
URL: https://www.tinybird.co/docs/forward/guides/ingest-vercel-ai-sdk
Last update: 2026-07-24T12:51:58.000Z
Content:
---
title: "Instrument LLM Calls With Vercel AI SDK \| Tinybird"
theme-color: "#171612"
description: "In this guide you'll learn how to instrument LLM calls with the Vercel AI SDK and Tinybird."
inkeep:version: "forward"
---
# Instrument LLM calls from Vercel AI SDK (https://www.tinybird.co/docs/forward/guides/ingest-vercel-ai-sdk#instrument-llm-calls-from-vercel-ai-sdk)
[Vercel AI SDK](https://sdk.vercel.ai/) is a powerful tool for building AI applications. It's a popular choice for many developers and organizations.
To start instrumenting LLM calls with the Vercel AI SDK and Tinybird, first create a data source with this schema:
```
SCHEMA >
`model` LowCardinality(String) `json:$.model` DEFAULT 'unknown',
`messages` Array(Map(String, String)) `json:$.messages[:]` DEFAULT [],
`user` String `json:$.user` DEFAULT 'unknown',
`start_time` DateTime `json:$.start_time` DEFAULT now(),
`end_time` DateTime `json:$.end_time` DEFAULT now(),
`id` String `json:$.id` DEFAULT '',
`stream` Bool `json:$.stream` DEFAULT false,
`call_type` LowCardinality(String) `json:$.call_type` DEFAULT 'unknown',
`provider` LowCardinality(String) `json:$.provider` DEFAULT 'unknown',
`api_key` String `json:$.api_key` DEFAULT '',
`log_event_type` LowCardinality(String) `json:$.log_event_type` DEFAULT 'unknown',
`llm_api_duration_ms` Float32 `json:$.llm_api_duration_ms` DEFAULT 0,
`cache_hit` Bool `json:$.cache_hit` DEFAULT false,
`response_status` LowCardinality(String) `json:$.standard_logging_object_status` DEFAULT 'unknown',
`response_time` Float32 `json:$.standard_logging_object_response_time` DEFAULT 0,
`proxy_metadata` String `json:$.proxy_metadata` DEFAULT '',
`organization` String `json:$.proxy_metadata.organization` DEFAULT '',
`environment` String `json:$.proxy_metadata.environment` DEFAULT '',
`project` String `json:$.proxy_metadata.project` DEFAULT '',
`chat_id` String `json:$.proxy_metadata.chat_id` DEFAULT '',
`response` String `json:$.response` DEFAULT '',
`response_id` String `json:$.response.id`,
`response_object` String `json:$.response.object` DEFAULT 'unknown',
`response_choices` Array(String) `json:$.response.choices[:]` DEFAULT [],
`completion_tokens` UInt16 `json:$.response.usage.completion_tokens` DEFAULT 0,
`prompt_tokens` UInt16 `json:$.response.usage.prompt_tokens` DEFAULT 0,
`total_tokens` UInt16 `json:$.response.usage.total_tokens` DEFAULT 0,
`cost` Float32 `json:$.cost` DEFAULT 0,
`exception` String `json:$.exception` DEFAULT '',
`traceback` String `json:$.traceback` DEFAULT '',
`duration` Float32 `json:$.duration` DEFAULT 0
ENGINE MergeTree
ENGINE_SORTING_KEY start_time, organization, project, model
ENGINE_PARTITION_KEY toYYYYMM(start_time)
```
Use a wrapper around the LLM provider you use, this is an example using OpenAI:
```
const openai = createOpenAI({ apiKey: apiKey });
const wrappedOpenAI = wrapModelWithTinybird(
openai('gpt-3.5-turbo'),
process.env.NEXT_PUBLIC_TINYBIRD_API_URL!,
process.env.TINYBIRD_TOKEN!,
{
event: 'search_filter',
environment: process.env.NODE_ENV,
project: 'ai-analytics',
organization: 'your-org',
}
);
```
Implement the wrapper in your app:
```
import type { LanguageModelV1 } from '@ai-sdk/provider';
type TinybirdConfig = {
event?: string;
organization?: string;
project?: string;
environment?: string;
user?: string;
chatId?: string;
};
export function wrapModelWithTinybird(
model: LanguageModelV1,
tinybirdHost: string,
tinybirdToken: string,
config: TinybirdConfig = {}
) {
const originalDoGenerate = model.doGenerate;
const originalDoStream = model.doStream;
const logToTinybird = async (
messageId: string,
startTime: Date,
status: 'success' | 'error',
// eslint-disable-next-line @typescript-eslint/no-explicit-any
args: any[],
result?: { text?: string; usage?: { promptTokens?: number; completionTokens?: number } },
error?: Error
) => {
const endTime = new Date();
const duration = endTime.getTime() - startTime.getTime();
const event = {
start_time: startTime.toISOString(),
end_time: endTime.toISOString(),
message_id: messageId,
model: model.modelId || 'unknown',
provider: 'openai',
duration,
llm_api_duration_ms: duration,
response: status === 'success' ? {
id: messageId,
object: 'chat.completion',
usage: {
prompt_tokens: result?.usage?.promptTokens || 0,
completion_tokens: result?.usage?.completionTokens || 0,
total_tokens: (result?.usage?.promptTokens || 0) + (result?.usage?.completionTokens || 0),
},
choices: [{ message: { content: result?.text ?? '' } }],
} : undefined,
messages: args[0]?.prompt ? [{ role: 'user', content: args[0].prompt }].map(m => ({
role: String(m.role),
content: String(m.content)
})) : [],
proxy_metadata: {
organization: config.organization || '',
project: config.project || '',
environment: config.environment || '',
chat_id: config.chatId || '',
},
user: config.user || 'unknown',
standard_logging_object_status: status,
standard_logging_object_response_time: duration,
log_event_type: config.event || 'chat_completion',
id: messageId,
call_type: 'completion',
cache_hit: false,
...(status === 'error' && {
exception: error?.message || 'Unknown error',
traceback: error?.stack || '',
}),
};
// Send to Tinybird
fetch(`${tinybirdHost}/v0/events?name=llm_events`, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Authorization': `Bearer ${tinybirdToken}`,
},
body: JSON.stringify(event),
}).catch(console.error);
};
model.doGenerate = async function (...args) {
const startTime = new Date();
const messageId = crypto.randomUUID();
try {
const result = await originalDoGenerate.apply(this, args);
await logToTinybird(messageId, startTime, 'success', args, result);
return result;
} catch (error) {
await logToTinybird(messageId, startTime, 'error', args, undefined, error as Error);
throw error;
}
};
model.doStream = async function (...args) {
const startTime = new Date();
const messageId = crypto.randomUUID();
try {
const result = await originalDoStream.apply(this, args);
await logToTinybird(messageId, startTime, 'success', args, { text: '', usage: { promptTokens: 0, completionTokens: 0 } });
return result;
} catch (error) {
await logToTinybird(messageId, startTime, 'error', args, undefined, error as Error);
throw error;
}
};
return model;
}
```
## AI analytics template (https://www.tinybird.co/docs/forward/guides/ingest-vercel-ai-sdk#ai-analytics-template)
Use the [LLM tracker template](https://github.com/tinybirdco/llm-performance-tracker) to bootstrap a multi-tenant, user-facing AI analytics dashboard and LLM cost calculator for your AI models. You can fork it and make it your own.
## See also (https://www.tinybird.co/docs/forward/guides/ingest-vercel-ai-sdk#see-also)
- [Vercel AI SDK docs](https://sdk.vercel.ai/)
---
URL: https://www.tinybird.co/docs/forward/guides/ingest-litellm
Last update: 2026-05-29T14:27:44.000Z
Content:
---
title: "Send LiteLLM Events to Tinybird \| Tinybird"
theme-color: "#171612"
description: "In this guide you'll learn how to send LiteLLM events to Tinybird using webhooks and the Events API."
inkeep:version: "forward"
---
# Send LiteLLM events to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-litellm#send-litellm-events-to-tinybird)
[LiteLLM](https://www.litellm.ai/) is an LLM gateway that provides AI models access, fallbacks, and spend tracking across 100+ LLMs. It's a popular choice for many developers and organizations.
LiteLLM is open source and can be self-hosted.
To start sending LiteLLM events to Tinybird, first create a data source with this schema:
```
SCHEMA >
`model` LowCardinality(String) `json:$.model` DEFAULT 'unknown',
`messages` Array(Map(String, String)) `json:$.messages[:]` DEFAULT [],
`user` String `json:$.user` DEFAULT 'unknown',
`start_time` DateTime `json:$.start_time` DEFAULT now(),
`end_time` DateTime `json:$.end_time` DEFAULT now(),
`id` String `json:$.id` DEFAULT '',
`stream` Bool `json:$.stream` DEFAULT false,
`call_type` LowCardinality(String) `json:$.call_type` DEFAULT 'unknown',
`provider` LowCardinality(String) `json:$.provider` DEFAULT 'unknown',
`api_key` String `json:$.api_key` DEFAULT '',
`log_event_type` LowCardinality(String) `json:$.log_event_type` DEFAULT 'unknown',
`llm_api_duration_ms` Float32 `json:$.llm_api_duration_ms` DEFAULT 0,
`cache_hit` Bool `json:$.cache_hit` DEFAULT false,
`response_status` LowCardinality(String) `json:$.standard_logging_object_status` DEFAULT 'unknown',
`response_time` Float32 `json:$.standard_logging_object_response_time` DEFAULT 0,
`proxy_metadata` String `json:$.proxy_metadata` DEFAULT '',
`organization` String `json:$.proxy_metadata.organization` DEFAULT '',
`environment` String `json:$.proxy_metadata.environment` DEFAULT '',
`project` String `json:$.proxy_metadata.project` DEFAULT '',
`chat_id` String `json:$.proxy_metadata.chat_id` DEFAULT '',
`response` String `json:$.response` DEFAULT '',
`response_id` String `json:$.response.id`,
`response_object` String `json:$.response.object` DEFAULT 'unknown',
`response_choices` Array(String) `json:$.response.choices[:]` DEFAULT [],
`completion_tokens` UInt16 `json:$.response.usage.completion_tokens` DEFAULT 0,
`prompt_tokens` UInt16 `json:$.response.usage.prompt_tokens` DEFAULT 0,
`total_tokens` UInt16 `json:$.response.usage.total_tokens` DEFAULT 0,
`cost` Float32 `json:$.cost` DEFAULT 0,
`exception` String `json:$.exception` DEFAULT '',
`traceback` String `json:$.traceback` DEFAULT '',
`duration` Float32 `json:$.duration` DEFAULT 0
ENGINE MergeTree
ENGINE_SORTING_KEY start_time, organization, project, model
ENGINE_PARTITION_KEY toYYYYMM(start_time)
```
Install the Tinybird AI Python SDK:
```
pip install tinybird-python-sdk[ai]
```
Finally, use the following handler in your app:
```
import litellm
from litellm import acompletion
from tb.litellm.handler import TinybirdLitellmAsyncHandler
customHandler = TinybirdLitellmAsyncHandler(
api_url="https://api.us-east.aws.tinybird.co",
tinybird_token=os.getenv("TINYBIRD_TOKEN"),
datasource_name="litellm"
)
litellm.callbacks = [customHandler]
response = await acompletion(
model="gpt-3.5-turbo",
messages=[{"role": "user", "content": "Hi 👋 - i'm openai"}],
stream=True
)
```
## AI analytics template (https://www.tinybird.co/docs/forward/guides/ingest-litellm#ai-analytics-template)
Use the [LLM tracker template](https://github.com/tinybirdco/llm-performance-tracker) to bootstrap a multi-tenant, user-facing AI analytics dashboard and LLM cost calculator for your AI models. You can fork it and make it your own.
## See also (https://www.tinybird.co/docs/forward/guides/ingest-litellm#see-also)
- [Events API](/docs/forward/ingest-data/events-api)
- [LiteLLM docs](https://docs.litellm.ai/)
---
URL: https://www.tinybird.co/docs/forward/guides/ingest-from-vercel
Last update: 2026-08-07T12:15:37.000Z
Content:
---
title: "Send Vercel Webhooks to Tinybird \| Tinybird"
theme-color: "#171612"
description: "In this guide you'll learn how to send Vercel events to Tinybird using webhooks and the Events API."
inkeep:version: "forward"
---
# Send Vercel events to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-vercel#send-vercel-events-to-tinybird)
[Vercel](https://vercel.com/) is a platform for building and deploying web applications. By integrating Vercel with Tinybird, you can analyze your Vercel events in real time and enrich it with other data sources.
Some common use cases for sending Vercel events to Tinybird include:
1. Tracking deployments, projects, integrations and domains status and errors.
2. Creating custom analytical dashboards.
3. Monitoring attacks.
Read on to learn how to send data from Vercel to Tinybird.
## Before you start (https://www.tinybird.co/docs/forward/guides/ingest-from-vercel#before-you-start)
Before you connect Vercel webhooks to Tinybird, ensure:
- You have a Vercel account.
- You have a Tinybird workspace.
## Connect Vercel to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-vercel#connect-vercel-to-tinybird)
1. Choose your team scope on the dashboard, and go to **Settings** > **Webhooks**.
2. Select the Webhooks and Projects you want to send to Tinybird.
3. In Tinybird, create a data source, called `vercel`. You can follow this [schema](https://github.com/tinybirdco/tinynest/blob/main/tinybird/datasources/vercel.datasource):
```
SCHEMA >
`event_time` DateTime `json:$.tinybirdIngestTime` DEFAULT now(),
`event_type` String `json:$.type` DEFAULT 'unknown',
`event` JSON `json:$` DEFAULT '{}'
ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(event_time)"
ENGINE_SORTING_KEY "event_time"
```
Using the [JSON Data Type](/docs/sql-reference/data-types/json) you can store the semi-structured data you receive from Vercel webhooks in a single column. You can later retrieve various events and their metadata as needed in your pipes.
The `JSON` data type is in private beta. If you are interested in using this type, contact Tinybird in the [Community Slack](/docs/community) or through [Support](/docs/forward/support).
1. In Tinybird, copy a token with privileges to append to the data source you created. You can use the admin token or create one with the required scope.
2. Back in Vercel, paste the Events API URL in your Webhook Endpoint URL. Use the query parameter `name` to match the name of the data source you created in Tinybird. For example:
```
https:///v0/events?name=vercel&token=
```
The API host in the following examples must match your Workspace's region. See the full list of [regions and hosts](/docs/api-reference#regions-and-endpoints)
1. You're done. Any of the Vercel events you selected is automatically sent to Tinybird through the [Events API](/docs/forward/ingest-data/events-api).
You can check the status of the integration from the **Log** tab in the Tinybird `vercel` data source.
## See also (https://www.tinybird.co/docs/forward/guides/ingest-from-vercel#see-also)
- [Events API](/docs/forward/ingest-data/events-api)
- [Vercel webhooks](https://vercel.com/docs/observability/webhooks-overview)
---
URL: https://www.tinybird.co/docs/forward/guides/ingest-from-vector
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Ingest data using Vector.dev \| Tinybird"
theme-color: "#171612"
description: "In this guide you'll learn how to use the Events API as a Vector sink."
inkeep:version: "forward"
---
# Ingest data using Vector.dev (https://www.tinybird.co/docs/forward/guides/ingest-from-vector#ingest-data-using-vector-dev)
[Vector.dev](https://vector.dev/) is an open source tool created by DataDog for collecting, transforming, and shipping logs, metrics, and traces.
Some common use cases for using Vector.dev as a Tinybird sink include:
1. Ingesting data from a number of [Vector.dev sources](https://vector.dev/components/) to Tinybird.
2. Enriching other data sources with real-time Vector metrics.
3. Aggregate logs and metrics from Vector.dev to Tinybird.
4. Transform and redact sensitive data before ingesting it into Tinybird.
Read on to learn how to use the Events API as a Vector.dev sink.
## Before you start (https://www.tinybird.co/docs/forward/guides/ingest-from-vector#before-you-start)
Before you connect Vector.dev to Tinybird, ensure:
- You have installed Vector.dev.
- You have a Tinybird workspace.
## Use the Events API as a Vector.dev sink (https://www.tinybird.co/docs/forward/guides/ingest-from-vector#use-the-events-api-as-a-vector-dev-sink)
To push events to Tinybird from Vector.dev, you need to configure Vector to use the Events API as a sink.
You can use the following example Vector configuration to push events, in this case Docker logs, to Tinybird:
```
sources:
docker_logs:
type: "docker_logs"
transforms:
remap_docker_logs:
inputs:
- "docker_logs"
type: "remap"
source: |
. = parse_json!(.log)
sinks:
push_docker_logs_to_tinybird:
inputs:
- "remap_docker_logs"
type: "http"
uri: "$TINYBIRD_HOST/v0/events?name=docker"
auth:
strategy: "bearer"
token: "$TINYBIRD_TOKEN"
encoding:
codec: "json"
framing:
method: "newline_delimited"
```
The previous snippet uses the `docker_logs` source to collect Docker logs, and the `remap_docker_logs` transform to parse the logs as JSON.
The `push_docker_logs_to_tinybird` sink uses the Events API to push the transformed logs to Tinybird in NDJSON (`newline_delimited`) format.
You can customize the `$TINYBIRD_HOST` and `$TINYBIRD_TOKEN` environment variables to use your Tinybird workspace.
Learn more about other sources you can use to ingest data into Tinybird in the [Vector.dev documentation](https://vector.dev/docs/reference/configuration/sources/).
---
URL: https://www.tinybird.co/docs/forward/guides/ingest-from-stripe
Last update: 2026-06-02T11:48:14.000Z
Content:
---
title: "Send Stripe Events to Tinybird \| Tinybird"
theme-color: "#171612"
description: "Learn how to send Stripe events to Tinybird using webhooks and the Events API."
inkeep:version: "forward"
---
# Send Stripe events to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-stripe#send-stripe-events-to-tinybird)
[Stripe](https://stripe.com/) is a platform for payments and financial services, and it provides a way to send events to Tinybird using webhooks.
Some common use cases for sending Stripe events to Tinybird include:
1. Monitor Stripe events.
2. Run analytical workflows based on Stripe events.
3. Create custom dashboards based on Stripe events.
4. Create alerts and notifications based on Stripe events.
5. Join Stripe events with other data sources to enrich your user data.
Read on to learn how to send events from Stripe to Tinybird.
## Before you start (https://www.tinybird.co/docs/forward/guides/ingest-from-stripe#before-you-start)
Before you connect Stripe to Tinybird, ensure:
- You have a Stripe account.
- You have a Tinybird workspace.
## Connect Stripe to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-stripe#connect-stripe-to-tinybird)
Stripe provides a variety of [webhook event types](https://docs.stripe.com/api/events/object) that you can use to send events to Tinybird.
This guide covers the base case for sending Stripe events to Tinybird.
1. In Stripe, go to [Webhooks](https://dashboard.stripe.com/webhooks)
2. Select **Add endpoint**.
3. In your Tinybird project, create a data source called `stripe`. You can follow this [schema](https://github.com/tinybirdco/tinynest/blob/main/tinybird/datasources/stripe.datasource):
```
SCHEMA >
`event_time` DateTime `json:$.tinybirdIngestTime` DEFAULT now(),
`event_type` String `json:$.type` DEFAULT 'unknown',
`event` String `json:$` DEFAULT '{}'
ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(event_time)"
ENGINE_SORTING_KEY "event_time"
```
Store the JSON-formatted data you receive from Stripe as a `String` in a single column. You can later parse the JSON string to retrieve various events and their metadata as needed in your Pipes.
1. From Tinybird Cloud, copy a token with privileges to append to the data source you created. You can use the admin token or create one with the required scope.
2. Back in Stripe, paste the Events API URL in your Webhook Endpoint URL. Use the query parameter `name` to match the name of the data source you created in Tinybird.
```
https:///v0/events?name=stripe&format=json&token=
```
Make sure to use the `format=json` query parameter.
The API host in the following examples must match your Workspace's region. See the full list of [regions and hosts](/docs/api-reference#regions-and-endpoints)
1. Select **Select events** and select the events you want to send to Tinybird.
2. Save and you're done.
Check the status of the integration by selecting the webhook in Stripe or from the **Log** tab in the Tinybird `stripe` data source.
## See also (https://www.tinybird.co/docs/forward/guides/ingest-from-stripe#see-also)
- [Stripe Webhooks](https://docs.stripe.com/webhooks)
- [Stripe Events](https://docs.stripe.com/api/events/object)
---
URL: https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-incremental-updates
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Ingest Snowflake Data Incrementally \| Tinybird"
theme-color: "#171612"
description: "Learn how to ingest data from Snowflake doing incremental appends, so you can keep last transactional data sources in sync with Tinybird."
inkeep:version: "forward"
---
# Ingest from Snowflake using incremental updates (https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-incremental-updates#ingest-from-snowflake-using-incremental-updates)
Read on to learn how to incrementally load data from a Snowflake table into Tinybird, using Amazon S3 as an intermediary staging area.
An incremental loading strategy ensures that only new or updated rows are transferred.
## Before you start (https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-incremental-updates#before-you-start)
Before you begin, ensure the following:
- Snowflake Account: A Snowflake instance with the data you want to load.
- Amazon S3 Bucket: Access to an S3 bucket for staging data, along with appropriate permissions (write from Snowflake and read from Tinybird).
- Tinybird Account: A Tinybird workspace with an appropriate data source set up.
- Snowflake Permissions: Ensure the Snowflake user has privileges to:
- Access the target table.
- Create and manage stages.
- Unload data into S3.
- AWS Credentials: Ensure Snowflake can use AWS credentials (IAM Role or Access Key/Secret Key pair) to write to the S3 bucket.
1
## Create the unload task in Snowflake (https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-incremental-updates#create-the-unload-task-in-snowflake)
Follow these steps to create the unload task in Snowflake:
1. Grant the required permissions in AWS IAM Console
Make sure the S3 bucket allows Snowflake to write files by setting up an appropriate IAM role or policy. You can use this template to create the policy and attach it to the AWS role:
```
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["s3:PutObject", "s3:AbortMultipartUpload"],
"Resource": "arn:aws:s3:::your-bucket-name/path/*"
}
]
}
```
Replace `your-bucket-name/path/*` with your bucket name, and optionally the path you want to grant access to.
1. Create the storage integration
Run the following SQL statement to create the storage integration:
```
/* Create the S3 integration.
*/
CREATE or replace STORAGE INTEGRATION tinybird_integration
TYPE = EXTERNAL_STAGE
STORAGE_PROVIDER = 'S3'
ENABLED = TRUE
STORAGE_AWS_ROLE_ARN = ''
STORAGE_ALLOWED_LOCATIONS = ('*');
-- describe integration tinybird_integration;
```
Replace `` with the ARN of the role created in the previous step.
1. Create the file format
Run the following SQL statement to create the file format:
```
/* Create the file format for the output files generated.
*/
CREATE OR REPLACE FILE FORMAT csv_unload_format
TYPE = 'CSV';
```
1. Create the stage
Run the following SQL statement to create the stage:
```
/* And finally the stage we'll use to unload the data to.
*/
CREATE or replace STAGE tinybird_stage
STORAGE_INTEGRATION = tinybird_integration
URL = 's3://your-bucket-name/path/'
FILE_FORMAT = csv_unload_format;
```
Replace `your-bucket-name` and `path` with your S3 bucket details.
2
## Create the unload task (https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-incremental-updates#create-the-unload-task)
Run the following SQL statement to create the scheduled task that unloads the new records since the last successful execution to the S3 bucket:
```
/* Create the scheduled task that unloads the new records since
* last successful execution to the S3 bucket.
*
* Note how it reads the timestamp of the last successful execution,
* and leaves a one hour margin.
*
* Orders need to be deduplicated later in Tinybird.
*/
CREATE or replace TASK export_order_deltas
WAREHOUSE = compute_wh
SCHEDULE = 'USING CRON 05 * * * * UTC'
AS
BEGIN
LET sql := 'COPY INTO @tinybird_stage/orders/orders_ from (
select
O_ORDERKEY, O_CUSTKEY, O_ORDERSTATUS, O_TOTALPRICE, O_ORDERDATE,
O_ORDERPRIORITY, O_CLERK
from tinybird.samples.orders_incremental
where o_orderdate >= (
SELECT coalesce(timestampadd(hour,-1,max(QUERY_START_TIME)),\'1970-01-01\')
FROM TABLE(INFORMATION_SCHEMA.TASK_HISTORY(TASK_NAME=>\'export_order_deltas\'))
where state = \'SUCCEEDED\'
ORDER BY SCHEDULED_TIME
)) max_file_size=1000000000';
sql := REPLACE(sql, '', TO_VARCHAR(CONVERT_TIMEZONE('UTC',current_timestamp()), 'YYYY_MM_DD__hh24_mi_ss'));
EXECUTE IMMEDIATE (sql);
RETURN sql;
END;
```
3
## Configure the ingestion in Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-incremental-updates#configure-the-ingestion-in-tinybird)
Create the S3 connection. See [S3 Connector](/docs/forward/ingest-data/connectors/s3).
You can use the following schema for the data source:
tinybird/datasources/s3_landing_ds.datasource - data source with S3 connection
```
SCHEMA >
`O_ORDERKEY` Int64,
`O_CUSTKEY` Int64,
`O_ORDERSTATUS` String,
`O_TOTALPRICE` Float32,
`O_ORDERDATE` DateTime64(3),
`O_ORDERPRIORITY` String,
`O_CLERK` String
ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYear(O_ORDERDATE)"
ENGINE_SORTING_KEY "O_ORDERDATE, O_ORDERPRIORITY, O_CLERK"
IMPORT_CONNECTION_NAME 'tinybird-tb-s3'
IMPORT_BUCKET_URI 's3://tinybird-tb/snowflake/csv/orders/*.csv.gz'
IMPORT_SCHEDULE '@auto'
```
Deploy the data source. The new files Snowflake writes to the bucket are automatically ingested by Tinybird in a few seconds.
4
## Handle duplicates in Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-incremental-updates#handle-duplicates-in-tinybird)
Use a materialized view to handle duplicates in Tinybird. For example:
tinybird/materializations/mat_s3_data.pipe - pipe to materialize data
```
NODE mat_s3_data_0
SQL >
SELECT *
FROM landing_ds
TYPE materialized
DATASOURCE deduplicate_rmt_mv
```
If you only need to work with the latest snapshot of the data, include a `file_ingested_at` field in the materializing pipe.
This is important when your incremental loads from Snowflake don’t indicate deleted records. Since a new ingest doesn’t overwrite a deleted record from a previous ingest, the record persists in the dataset. By filtering downstream on the `file_ingested_at` field, you can exclude these stale records and isolate only the most recent ingest. For example:
tinybird/materializations/mat_s3_data_latest.pipe - pipe to materialize data with timestamp
```
NODE mat_s3_data_latest_0
SQL >
SELECT *, toDateTime(now()) as file_ingested_at
FROM landing_ds
TYPE materialized
DATASOURCE deduplicate_rmt_mv
```
In the target Data Source, the `ReplacingMergeTree` and `ENGINE_VER` options [deduplicate records](/sql-reference/engines/replacingmergetree) with the same sorting key value.
tinybird/datasources/deduplicate_rmt_mv.datasource - Replacing Merge Tree to deduplicate data
```
SCHEMA >
`O_ORDERKEY` Int64,
`O_CUSTKEY` Int64,
`O_ORDERSTATUS` String,
`O_TOTALPRICE` Float32,
`O_ORDERDATE` DateTime64(3),
`O_ORDERPRIORITY` String,
`O_CLERK` String,
`file_ingested_at` DateTime
ENGINE "ReplacingMergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(O_ORDERDATE)"
ENGINE_SORTING_KEY "O_ORDERKEY"
ENGINE_VER "file_ingested_at"
```
Finally, create a pipe to query the deduplicated data source. Filter on the maximum timestamp to get the latest snapshot, [rounding the timestamp](/sql-reference/functions/date-time-functions#tostartofday) based on your import schedule. This ensures that you query the most recent snapshot with no duplicates.
tinybird/endpoints/get_snapshot.pipe - query for latest snapshot
```
NODE get_snapshot_0
SQL >
WITH (SELECT max(toStartOfDay(file_ingested_at)) FROM deduplicate_rmt_mv) AS latest_file_version
SELECT *
FROM deduplicate_rmt_mv
FINAL
WHERE toStartOfDay(file_ingested_at) = latest_file_version
TYPE endpoint
```
Remember to use `final` when querying a ReplacingMergeTree.
5
## Next steps (https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-incremental-updates#next-steps)
See the following resources:
- [Ingest from Snowflake using Azure Blob Storage](./ingest-from-snowflake-using-azure-blob-storage)
- [Ingest from Snowflake using AWS S3](./ingest-from-snowflake-using-aws-s3)
---
URL: https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-azure-blob-storage
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Ingest Snowflake via Azure Blob Storage \| Tinybird"
theme-color: "#171612"
description: "Learn how to send data from Snowflake to Tinybird using Azure Blob Storage."
inkeep:version: "forward"
---
# Ingest from Snowflake using Azure Blob Storage (https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-azure-blob-storage#ingest-from-snowflake-using-azure-blob-storage)
Read on to learn how to send data from Snowflake to Tinybird, for example when you need to periodically run full replaces of a table or do a one-off ingest.
This process relies on [unloading](https://docs.snowflake.com/en/user-guide/data-unload-overview), or bulk exporting, data as gzipped CSVs and then ingesting them using the Data Sources API.
## Prerequisites (https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-azure-blob-storage#prerequisites)
To follow these steps you need a Tinybird account and access to Snowflake and permissions to create SAS Tokens for Azure Blob Storage.
1
## Unload the Snowflake table (https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-azure-blob-storage#unload-the-snowflake-table)
Use Snowflake to [unload](https://docs.snowflake.com/en/user-guide/data-unload-overview) query results to flat files in an external storage service. For example:
```
COPY INTO 'azure://myaccount.blob.core.windows.net/unload/'
FROM mytable
CREDENTIALS = ( AZURE_SAS_TOKEN='****' )
FILE_FORMAT = ( TYPE = CSV COMPRESSION = GZIP )
HEADER = FALSE;
```
The most basic implementation is [unloading directly](https://docs.snowflake.com/en/sql-reference/sql/copy-into-location#unloading-data-from-a-table-directly-to-files-in-an-external-location), but for production use cases consider adding a [named stage](https://docs.snowflake.com/en/user-guide/data-unload-azure#unloading-data-into-an-external-stage) as suggested in the Snowflake docs. Stages give you fine-grained control to access rights.
2
## Create a SAS token for the file (https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-azure-blob-storage#create-a-sas-token-for-the-file)
Using [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli), generate a [shared access signature (SAS) token](https://learn.microsoft.com/en-us/azure/ai-services/translator/document-translation/how-to-guides/create-sas-tokens?tabs=blobs) so Tinybird can read the file:
```
az storage blob generate-sas \
--account-name myaccount \
--account-key '****' \
--container-name unload \
--name data.csv.gz \
--permissions r \
--expiry \
--https-only \
--output tsv \
--full-uri
> 'https://myaccount.blob.core.windows.net/unload/data.csv.gz?se=2024-05-31T10%3A57%3A41Z&sp=r&spr=https&sv=2022-11-02&sr=b&sig=PMC%2E9ZvOFtKATczsBQgFSsH1%2BNkuJvO9dDPkTpxXH0g%5D'
```
You can use the same behavior in S3 and GCS to generate presigned URLs.
3
## Ingest into Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-azure-blob-storage#ingest-into-tinybird)
Take the generated URL and make a call to Tinybird. You need a [token](/docs/administration/auth-tokens) with `DATASOURCES:CREATE` permissions:
```
curl \
-H "Authorization: Bearer " \
-X POST "https:///v0/datasources?name=my_datasource_name" \
-d url='https://myaccount.blob.core.windows.net/unload/data.csv.gz?se=2024-05-31T10%3A57%3A41Z&sp=r&spr=https&sv=2022-11-02&sr=b&sig=PMC%2E9ZvOFtKATczsBQgFSsH1%2BNkuJvO9dDPkTpxXH0g%5D'
```
You now have your Snowflake Table in Tinybird.
## Automation (https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-azure-blob-storage#automation)
To adapt to production scenarios, like having to append data on a timely basis or replacing data that has been updated in Snowflake, you might need to define scheduled actions to move the data.
## Limits (https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-azure-blob-storage#limits)
Because you're using the data sources API, its [limits](/docs/api-reference#limits) apply.
You might need to adjust your [COPY INTO ](https://docs.snowflake.com/en/sql-reference/sql/copy-into-location) expression adding `PARTITION` or `MAX_FILE_SIZE = 5000000000`. For example:
```
COPY INTO 'azure://myaccount.blob.core.windows.net/unload/'
FROM mytable
CREDENTIALS=( AZURE_SAS_TOKEN='****')
FILE_FORMAT = ( TYPE = CSV COMPRESSION = GZIP )
HEADER = FALSE
MAX_FILE_SIZE = 5000000000;
```
## Next steps (https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-azure-blob-storage#next-steps)
See the following resources:
- [Ingest from Snowflake using AWS S3](./ingest-from-snowflake-using-aws-s3)
- [Ingest from Snowflake using incremental updates](./ingest-from-snowflake-using-incremental-updates)
- [GCS Connector](/docs/forward/ingest-data/connectors/gcs)
---
URL: https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-aws-s3
Last update: 2026-06-02T11:48:14.000Z
Content:
---
title: "Ingest from Snowflake using AWS S3 \| Tinybird"
theme-color: "#171612"
description: "Learn how to send data from Snowflake to Tinybird using AWS S3."
inkeep:version: "forward"
---
# Ingest data from Snowflake using AWS S3 (https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-aws-s3#ingest-data-from-snowflake-using-aws-s3)
Read on to learn how to send data from Snowflake to Tinybird, for example when you need to periodically run full replaces of a table or do a one-off ingest.
This process relies on [unloading](https://docs.snowflake.com/en/user-guide/data-unload-overview), or bulk exporting, data as gzipped CSVs and then ingesting them using the Data Sources API. Data is then ingested using the [S3 Connector](/docs/forward/ingest-data/connectors/s3).
## Prerequisites (https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-aws-s3#prerequisites)
To follow these steps you need a Tinybird account and access to Snowflake and AWS S3.
1
## Unload the Snowflake table (https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-aws-s3#unload-the-snowflake-table)
The first step consists in unloading the Snowflake table to a gzipped CSV file.
1. Grant the required permissions in AWS IAM Console
Make sure the S3 bucket allows Snowflake to write files by setting up an appropriate IAM role or policy. You can use this template to create the policy and attach it to the AWS role:
```
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["s3:PutObject", "s3:AbortMultipartUpload"],
"Resource": "arn:aws:s3:::your-bucket-name/path/*"
}
]
}
```
Replace `your-bucket-name/path/*` with your bucket name, and optionally the path you want to grant access to. Attach the policy to the role you want use to unload the data from Snowflake.
1. Create the storage integration
Run the following SQL statement to create the storage integration:
```
/* Create the S3 integration.
*/
CREATE or replace STORAGE INTEGRATION tinybird_integration
TYPE = EXTERNAL_STAGE
STORAGE_PROVIDER = 'S3'
ENABLED = TRUE
STORAGE_AWS_ROLE_ARN = ''
STORAGE_ALLOWED_LOCATIONS = ('*');
-- describe integration tinybird_integration;
```
Replace `` with the ARN of the role created in the previous step.
1. Create the file format
Run the following SQL statement to create the file format:
```
/* Create the file format for the output files generated.
*/
CREATE OR REPLACE FILE FORMAT csv_unload_format
TYPE = 'CSV';
```
1. Create the stage
Run the following SQL statement to create the stage:
```
/* And finally the stage we'll use to unload the data to.
*/
CREATE or replace STAGE tinybird_stage
STORAGE_INTEGRATION = tinybird_integration
URL = 's3://your-bucket-name/path/'
FILE_FORMAT = csv_unload_format;
```
Replace `your-bucket-name` and `path` with your S3 bucket details.
1. Unload the data
Run the following SQL statement to unload the data:
```
COPY INTO @tinybird_stage/orders/ from (
select
O_ORDERKEY, O_CUSTKEY, O_ORDERSTATUS, O_TOTALPRICE, O_ORDERDATE,
O_ORDERPRIORITY, O_CLERK
from my_database.my_schema.orders
)
```
To automate the unloading, you can create a Snowflake task that runs the `COPY INTO` on a schedule. For example:
```
CREATE or replace TASK export_order_deltas
WAREHOUSE = compute_wh
SCHEDULE = 'USING CRON 05 * * * * UTC'
AS
COPY INTO @tinybird_stage/orders from (
select
O_ORDERKEY, O_CUSTKEY, O_ORDERSTATUS, O_TOTALPRICE, O_ORDERDATE,
O_ORDERPRIORITY, O_CLERK
from my_database.my_schema.orders
) max_file_size=1000000000
```
2
## Ingest data into Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-aws-s3#ingest-data-into-tinybird)
Before ingesting your Snowflake data from the S3 bucket, you need to create the S3 connection. See [S3 Connector](/docs/forward/ingest-data/connectors/s3).
You can use the following schema for the data source:
```
SCHEMA >
`O_ORDERKEY` Int64,
`O_CUSTKEY` Int64,
`O_ORDERSTATUS` String,
`O_TOTALPRICE` Float32,
`O_ORDERDATE` DateTime64(3),
`O_ORDERPRIORITY` String,
`O_CLERK` String
ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYear(O_ORDERDATE)"
ENGINE_SORTING_KEY "O_ORDERDATE, O_ORDERPRIORITY, O_CLERK"
IMPORT_CONNECTION_NAME 'tb-s3'
IMPORT_BUCKET_URI 's3://tb/snowflake/csv/orders/*.csv.gz'
IMPORT_SCHEDULE '@auto'
```
Deploy the data source. The new files Snowflake writes to the bucket are automatically ingested by Tinybird.
Each file is appended to the Tinybird data source. As records might be duplicated, consider using a materialized view to consolidate a stateful set of your Snowflake table.
## Limits (https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-aws-s3#limits)
Because you're using the data sources API, its [limits](/docs/api-reference#limits) apply.
You might need to adjust your [COPY INTO ](https://docs.snowflake.com/en/sql-reference/sql/copy-into-location) expression adding `PARTITION` or `MAX_FILE_SIZE = 5000000000`.
## Next steps (https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-aws-s3#next-steps)
See the following resources:
- [Ingest from Snowflake using Azure Blob Storage](./ingest-from-snowflake-using-azure-blob-storage)
- [Ingest from Snowflake using incremental updates](./ingest-from-snowflake-using-incremental-updates)
- [GCS Connector](/docs/forward/ingest-data/connectors/gcs)
---
URL: https://www.tinybird.co/docs/forward/guides/ingest-from-sentry
Last update: 2026-08-07T12:15:37.000Z
Content:
---
title: "Send Sentry Webhooks to Tinybird \| Tinybird"
theme-color: "#171612"
description: "Learn how to send Sentry events to Tinybird using webhooks and the Events API."
inkeep:version: "forward"
---
# Send Sentry events to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-sentry#send-sentry-events-to-tinybird)
[Sentry](https://sentry.io/) is a platform for monitoring and alerting on errors in your applications. By integrating Sentry with Tinybird, you can analyze your Sentry events in real time and enrich it with other data sources.
Some common use cases for sending Sentry events to Tinybird include:
1. Analyze errors from your applications.
2. Detect patterns in your error data.
3. Build an alert system based on error patterns.
4. Build custom analytical dashboards.
Read on to learn how to send logs from Sentry to Tinybird.
## Before you start (https://www.tinybird.co/docs/forward/guides/ingest-from-sentry#before-you-start)
Before you connect Sentry to Tinybird, ensure:
- You have a Sentry account.
- You have a Tinybird workspace.
## Connect Sentry to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-sentry#connect-sentry-to-tinybird)
1. In Sentry, go to **Settings** > **Developer Settings** > **Custom Integrations**.
2. Select **Create New Integration**.
3. In your Tinybird project, create a data source called `sentry`. You can follow this [schema](https://github.com/tinybirdco/tinynest/blob/main/tinybird/datasources/sentry.datasource):
```
SCHEMA >
`event_time` DateTime `json:$.tinybirdIngestTime` DEFAULT now(),
`event_type` String `json:$.action` DEFAULT 'unknown',
`event` JSON `json:$` DEFAULT '{}'
ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(event_time)"
ENGINE_SORTING_KEY "event_time"
```
Using the [JSON Data Type](/docs/sql-reference/data-types/json) you can store the semi-structured data you receive from Sentry in a single column. You can later retrieve various events and their metadata as needed in your pipes.
The `JSON` data type is in private beta. If you are interested in using this type, contact Tinybird in the [Community Slack](/docs/community) or through [Support](/docs/forward/support).
1. From Tinybird Cloud, copy a token with privileges to append to the data source you created. You can use the admin token or create one with the required scope.
2. Back in Sentry, paste the Events API URL in your Custom Integration. Use the query parameter `name` to match the name of the data source you created in Tinybird.
```
https:///v0/events?name=sentry&token=
```
The API host in the following examples must match your Workspace's region. See the full list of [regions and hosts](/docs/api-reference#regions-and-endpoints)
1. Select **Alert Rule Action**.
2. In the **Permissions** box **Issue and Event** > **Read**.
3. Check all webhooks and **Save Changes**.
4. If you also want to send alerts to Tinybird, select **Alerts** from the left menu, select an alert, and select **Edit Rule**. You can select **Send Notifications via** your previously created Custom Integration.
5. You can then select **Send Test Notification** to check the connection.
6. You're done. Any of the Sentry events you selected are automatically sent to Tinybird through the [Events API](/docs/forward/ingest-data/events-api).
Check the status of the integration from the **Log** tab in the Tinybird `sentry` data source.
## See also (https://www.tinybird.co/docs/forward/guides/ingest-from-sentry#see-also)
- [Sentry Webhooks](https://docs.sentry.io/organization/integrations/integration-platform/webhooks/)
---
URL: https://www.tinybird.co/docs/forward/guides/ingest-from-rudderstack
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Stream from RudderStack \| Tinybird"
theme-color: "#171612"
description: "In this guide, you'll learn two different methods to send events from RudderStack to Tinybird."
inkeep:version: "forward"
---
# Stream from RudderStack (https://www.tinybird.co/docs/forward/guides/ingest-from-rudderstack#stream-from-rudderstack)
This guide shows you two methods to send events from RudderStack to Tinybird.
To better understand the behavior of their customers, companies need to unify timestamped data coming from a wide variety of products and platforms. Typical events to track would be 'sign up', 'login', 'page view' or 'item purchased'. A customer data platform can be used to capture complete customer data like this from wherever your customers interact with your brand. It defines events, collects them from different platforms and products, and routes them to where they need to be consumed.
[RudderStack](https://www.rudderstack.com/) is an open source customer data pipeline tool. It collects, processes and routes data from your websites, apps, cloud tools, and data warehouse. By using Tinybird's event ingestion Endpoint for [high-frequency ingestion](/docs/forward/ingest-data/events-api) as a Webhook in RudderStack, you can stream customer data in real time to Data Sources.
## Option 1: A separate data source for each event type (https://www.tinybird.co/docs/forward/guides/ingest-from-rudderstack#option-1-a-separate-data-source-for-each-event-type)
This is the preferred approach. It sends each type of event to a corresponding data source.
The advantages of this method are:
- Your data is well organized from the start.
- Different event types can have different attributes (columns in their data source).
- Whenever new attributes are added to an event type, a prompt asks you to add new columns.
- New event types get a new Data Source.
Start by getting a Token to allow RudderStack to write to Tinybird.
### Get a Tinybird Token (https://www.tinybird.co/docs/forward/guides/ingest-from-rudderstack#get-a-tinybird-token)
Use your **Workspace admin token**, which has full access to your workspace. You can find it in **Tokens** in the sidebar, or list your tokens with the CLI:
```
tb token ls
```
### Create a RudderStack destination (https://www.tinybird.co/docs/forward/guides/ingest-from-rudderstack#create-a-rudderstack-destination)
In RudderStack, select **Destinations** in the side panel and then **New destination**.
Select **Webhook**:
1. Give the destination a descriptive name.
2. Connect your sources, you can test with the RudderStack Sample HTTP Source.
3. Input the following Connection Settings:
- Webhook URL: *< https:// /v0/events>*
- URL Method: *POST*
- Headers Key: *Authorization*
- Headers Value: *Bearer TINYBIRD_AUTH_TOKEN*
On the next page, select **Create new transformation**.
You can code a function in the box to apply to events when this transformation is active using the following example snippet. In this function, you can dynamically append the target data source to the target URL of the Webhook. Give your transformation a descriptive name and a helpful description.
Transformation code
```
export function transformEvent(event, metadata){
event.appendPath="?name=rudderstack_"+event.event.toLowerCase().replace(/[\s\.]/g, '_')
return event;
}
```
This example snippet uses the prefix `*rudderstack\_*` followed by the name of the event in lower case, with its words separated by an underscore. For instance, a "Product purchased" event would go to a data source named `rudderstack_product_purchased`.
Save the transformation. Your destination has been created successfully.
### Test ingestion (https://www.tinybird.co/docs/forward/guides/ingest-from-rudderstack#test-ingestion)
In RudderStack, select **Sources** > **RudderStack Sample HTTP** > **Live events** > **Send test event** and paste the provided curl command into your terminal. The event appears on the screen and is sent to Tinybird.
## Option 2: All events in the same data source (https://www.tinybird.co/docs/forward/guides/ingest-from-rudderstack#option-2-all-events-in-the-same-data-source)
This alternative approach sends all events into a single Data Source and then splits them using Tinybird. By pre-configuring the Data Source, Tinybird ingests any events that RudderStack sends with the full JSON object as a String in a single column. This is useful for complex JSON objects, but using JSONExtract to parse data from the JSON object after ingestion affects performance.
Tinybird detects new columns from parsing the data and asks whether you want to save them. You can adjust the inferred data types before saving any new columns. Use Pipes to filter the Data Source by different events.
### Preconfigure a data source (https://www.tinybird.co/docs/forward/guides/ingest-from-rudderstack#preconfigure-a-data-source)
Create a new file in your local project, named `rudderstack_events.datasource`, for example. Add an `APPEND` token and configure the schema:
rudderstack_events.datasource
```
TOKEN rudderstack_append APPEND
SCHEMA >
`value` String `json:$`
ENGINE "MergeTree"
ENGINE_SORTING_KEY "value"
```
Deploy the changes using `tb deploy`. Tinybird creates the `rudderstack_append` token automatically during deployment. Retrieve its value with:
```
tb token ls
```
This pre-configured Data Source is required only if you need a column containing the full JSON object as a String. Otherwise, skip this step and let Tinybird infer the columns and data types when you send the first event. Then select which columns to save and adjust their data types.
### Create a RudderStack destination (https://www.tinybird.co/docs/forward/guides/ingest-from-rudderstack#create-a-rudderstack-destination)
In RudderStack, select **Destinations** in the side panel and then **New destination**.
Select **Webhook**:
1. Give the destination a descriptive name.
2. Connect your sources, you can test with the RudderStack Sample HTTP Source.
3. Input the following Connection Settings:
- Webhook URL: *< https:// /v0/events?name=rudderstack_events>*
- URL Method: *POST*
- Headers Key: *Authorization*
- Headers Value: *Bearer TINYBIRD_AUTH_TOKEN*
Select **No transformation needed** and save. Your destination has been created successfully.
### Test ingestion (https://www.tinybird.co/docs/forward/guides/ingest-from-rudderstack#test-ingestion)
Select **Sources** > **RudderStack Sample HTTP** > **Live events** > **Send test event** and paste the provided curl command into your terminal. The event appears on the screen and is sent to Tinybird.
The `value` column contains the full JSON object. You can also parse the data into columns. When viewing the new columns, select which ones to save and adjust their data types.
Whenever Tinybird detects new columns in the stream of events, it asks whether you want to save them.
---
URL: https://www.tinybird.co/docs/forward/guides/ingest-from-resend
Last update: 2026-08-07T12:15:37.000Z
Content:
---
title: "Send Resend webhooks to Tinybird \| Tinybird"
theme-color: "#171612"
description: "In this guide you'll learn how to send data from Resend to Tinybird."
inkeep:version: "forward"
---
# Send Resend webhooks to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-resend#send-resend-webhooks-to-tinybird)
With [Resend](https://resend.com/) you can send and receive emails programmatically. By integrating Resend with Tinybird, you can analyze your email data in real time.
Some common use cases for sending Resend webhooks to Tinybird include:
1. Tracking email opens and clicks.
2. Monitoring delivery rates and bounces.
3. Analyzing user engagement patterns.
4. Creating custom dashboards for email performance.
5. Enriching other data sources with real-time email metrics.
Read on to learn how to send data from Resend to Tinybird.
## Before you start (https://www.tinybird.co/docs/forward/guides/ingest-from-resend#before-you-start)
Before you connect Resend to Tinybird, ensure:
- You have a Resend account.
- You have a Tinybird workspace.
## Connect Resend to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-resend#connect-resend-to-tinybird)
1. Open the Resend UI and go to the Webhooks page.
2. Select **Add Webhook**.
3. In your Tinybird project, create a data source called `resend`. You can follow this [schema](https://github.com/tinybirdco/tinynest/blob/main/tinybird/datasources/resend.datasource):
```
SCHEMA >
`event_time` DateTime `json:$.tinybirdIngestTime` DEFAULT now(),
`event_type` String `json:$.type` DEFAULT 'unknown',
`event` JSON `json:$` DEFAULT '{}'
ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(event_time)"
ENGINE_SORTING_KEY "event_time"
```
Using the [JSON Data Type](/docs/sql-reference/data-types/json) you can store the semi-structured data you receive from Resend in a single column. You can later retrieve various events and their metadata as needed in your pipes.
The `JSON` data type is in private beta. If you are interested in using this type, contact Tinybird in the [Community Slack](/docs/community) or through [Support](/docs/forward/support).
1. From Tinybird Cloud, copy a token with privileges to append to the data source you created. You can use the admin token or create one with the required scope.
2. Back in Resend, paste the Events API URL in your Webhook URL. Use the query parameter `name` to match the name of the data source you created in Tinybird. For example:
```
https:///v0/events?name=resend&token=
```
The API host in the following examples must match your Workspace's region. See the full list of [regions and hosts](/docs/api-reference#regions-and-endpoints)
1. Select the checkboxes for the Resend events you want to send to Tinybird, and select **Add**.
2. You're done. Sending emails to Resend now pushes events to Tinybird through the [Events API](/docs/forward/ingest-data/events-api).
## See also (https://www.tinybird.co/docs/forward/guides/ingest-from-resend#see-also)
- [Resend event types](https://resend.com/docs/dashboard/webhooks/event-types)
---
URL: https://www.tinybird.co/docs/forward/guides/ingest-from-pagerduty
Last update: 2026-08-07T12:15:37.000Z
Content:
---
title: "Send PagerDuty events to Tinybird \| Tinybird"
theme-color: "#171612"
description: "In this guide you'll learn how to send PagerDuty events to Tinybird using webhooks and the Events API."
inkeep:version: "forward"
---
# Send PagerDuty events to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-pagerduty#send-pagerduty-events-to-tinybird)
[PagerDuty](https://www.pagerduty.com/) is a platform for incident management and alerting. By integrating PagerDuty with Tinybird, you can analyze your incident data in real time and enrich it with other data sources.
Some common use cases for sending PagerDuty events to Tinybird include:
1. Monitoring and alerting on incidents.
2. Creating custom dashboards for incident analysis.
3. Incident logs.
Read on to learn how to send events from PagerDuty to Tinybird.
## Before you start (https://www.tinybird.co/docs/forward/guides/ingest-from-pagerduty#before-you-start)
Before you connect PagerDuty to Tinybird, ensure:
- You have an PagerDuty account.
- You have a Tinybird workspace.
## Connect PagerDuty to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-pagerduty#connect-pagerduty-to-tinybird)
1. From the PagerDuty dashboard, select **Integrations** > **Developer Tools** > **Webhooks**.
2. Select **New Webhook**.
3. In your Tinybird project, create a data source called `pagerduty`. You can follow this [schema](https://github.com/tinybirdco/tinynest/blob/main/tinybird/datasources/pagerduty.datasource):
```
SCHEMA >
`event_time` DateTime `json:$.tinybirdIngestTime` DEFAULT now(),
`event_type` String `json:$.event.event_type` DEFAULT 'unknown',
`event` JSON `json:$` DEFAULT '{}'
ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(event_time)"
ENGINE_SORTING_KEY "event_time"
```
Using the [JSON Data Type](/docs/sql-reference/data-types/json) you can store the semi-structured data you receive from PagerDuty in a single column. You can later retrieve various events and their metadata as needed in your pipes.
The `JSON` data type is in private beta. If you are interested in using this type, contact Tinybird in the [Community Slack](/docs/community) or through [Support](/docs/forward/support).
1. From Tinybird Cloud, copy a token with privileges to append to the data source you created. You can use the admin token or create one with the required scope.
2. Back in PagerDuty, paste the Events API URL in your Webhook URL. Use the query parameter `name` to match the name of the data source you created in Tinybird. For example:
```
https:///v0/events?name=pagerduty
```
The API host in the following examples must match your Workspace's region. See the full list of [regions and hosts](/docs/api-reference#regions-and-endpoints)
1. Select **Add custom header** and add 'Authorization' as **Name** and paste the token you created in Tinybird as **Value**.
```
Bearer
```
1. Select all event subcriptions and **Add webhook**
2. You're done. Any of the PagerDuty events is automatically sent to Tinybird through the [Events API](/docs/forward/ingest-data/events-api).
You can check the status of the integration by testing the Webhook integration in PagerDuty or from the **Log** tab in the Tinybird `pagerduty` data source.
## See also (https://www.tinybird.co/docs/forward/guides/ingest-from-pagerduty#see-also)
- [PagerDuty webhooks](https://support.pagerduty.com/docs/webhooks)
---
URL: https://www.tinybird.co/docs/forward/guides/ingest-from-orb
Last update: 2026-08-07T12:15:37.000Z
Content:
---
title: "Send Orb events to Tinybird \| Tinybird"
theme-color: "#171612"
description: "In this guide you'll learn how to send Orb events to Tinybird using webhooks and the Events API."
inkeep:version: "forward"
---
# Send Orb events to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-orb#send-orb-events-to-tinybird)
[Orb](https://withorb.com/) is a developer-focused platform to manage your subscription billing and revenue operations. By integrating Orb with Tinybird, you can analyze your subscription billing data in real time and enrich it with other data sources.
Some common use cases for sending Orb events to Tinybird include:
1. Tracking and monitoring subscriptions.
2. Monitoring user churn.
3. Creating custom dashboards for subscription analysis.
4. Subscriptions logs.
Read on to learn how to send events from Orb to Tinybird.
## Before you start (https://www.tinybird.co/docs/forward/guides/ingest-from-orb#before-you-start)
Before you connect Orb to Tinybird, ensure:
- You have an Orb account.
- You have a Tinybird workspace.
## Connect Orb to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-orb#connect-orb-to-tinybird)
1. From the Orb dashboard, select **Developers** > **Webhooks**.
2. Select **Add Endpoint**.
3. In your Tinybird project, create a data source called `orb`. You can follow this [schema](https://github.com/tinybirdco/tinynest/blob/main/tinybird/datasources/orb.datasource):
```
SCHEMA >
`event_time` DateTime `json:$.tinybirdIngestTime` DEFAULT now(),
`event_type` String `json:$.type` DEFAULT 'unknown',
`event` JSON `json:$` DEFAULT '{}'
ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(event_time)"
ENGINE_SORTING_KEY "event_time"
```
Using the [JSON Data Type](/docs/sql-reference/data-types/json) you can store the semi-structured data you receive from Orb in a single column. You can later retrieve various events and their metadata as needed in your pipes.
The `JSON` data type is in private beta. If you are interested in using this type, contact Tinybird in the [Community Slack](/docs/community) or through [Support](/docs/forward/support).
1. From Tinybird Cloud, copy a token with privileges to append to the data source you created. You can use the admin token or create one with the required scope.
2. Back in Orb, paste the Events API URL in your Webhook Endpoint URL. Use the query parameter `name` to match the name of the data source you created in Tinybird. For example:
```
https:///v0/events?name=orb&token=
```
The API host in the following examples must match your Workspace's region. See the full list of [regions and hosts](/docs/api-reference#regions-and-endpoints)
1. Select **Send test request** to test the connection and check the data gets to the `orb` data source in Tinybird.
2. You're done. Any of the Orb events is automatically sent to Tinybird through the [Events API](/docs/forward/ingest-data/events-api).
You can check the status of the integration by clicking on the Webhook endpoint in Orb or from the **Log** tab in the Tinybird `orb` data source.
## See also (https://www.tinybird.co/docs/forward/guides/ingest-from-orb#see-also)
- [Orb webhooks](https://docs.withorb.com/guides/integrations-and-exports/webhooks)
---
URL: https://www.tinybird.co/docs/forward/guides/ingest-from-opentelemetry
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Ingest from OpenTelemetry \| Tinybird"
theme-color: "#171612"
description: "Learn how to ingest metrics, traces, and logs from OpenTelemetry into Tinybird using the Tinybird OpenTelemetry Collector distribution."
inkeep:version: "forward"
---
# Ingest from OpenTelemetry (https://www.tinybird.co/docs/forward/guides/ingest-from-opentelemetry#ingest-from-opentelemetry)
[OpenTelemetry](https://opentelemetry.io/) is an open source observability framework for collecting, processing, and exporting telemetry data (metrics, traces, and logs) from your apps and infrastructure.
By integrating OpenTelemetry with Tinybird, you can analyze observability data in real time, build dashboards, and enrich it with other data sources.
Some common use cases for sending OpenTelemetry data to Tinybird include:
1. Centralizing metrics, traces, and logs for unified analytics.
2. Building custom dashboards and alerts on top of observability data.
3. Enriching telemetry with business or app data.
Read on to learn how to send data from OpenTelemetry to Tinybird.
## Before you start (https://www.tinybird.co/docs/forward/guides/ingest-from-opentelemetry#before-you-start)
Before you connect OpenTelemetry to Tinybird, ensure:
- You have a Tinybird workspace.
- You have a Tinybird Token with **append** permissions to the target Data Sources.
- You are running a release version of the OpenTelemetry Collector higher than release v0.131.0.
- [GitHub Releases](https://github.com/open-telemetry/opentelemetry-collector-contrib/releases)
- [Docker Hub](https://hub.docker.com/r/otel/opentelemetry-collector-contrib)
The Tinybird OpenTelemetry Exporter is officially available in the OpenTelemetry Collector Contrib repository as of release v0.131.0. You can use the official OpenTelemetry Collector distributions that include the Tinybird exporter by default.
## Use the Tinybird OpenTelemetry project template (https://www.tinybird.co/docs/forward/guides/ingest-from-opentelemetry#use-the-tinybird-opentelemetry-project-template)
To get started quickly, you can use the [Tinybird OpenTelemetry project template](https://www.tinybird.co/templates/opentelemetry). This template provides ready-to-use Data Sources and Pipes for storing and analyzing your telemetry data in Tinybird.
```
# authenticate and select your Workspace
tb login
# deploy the template
tb --cloud deploy --template https://github.com/tinybirdco/tinybird-otel-template/tree/main/
```
## Recommended OpenTelemetry Collector configuration (https://www.tinybird.co/docs/forward/guides/ingest-from-opentelemetry#recommended-opentelemetry-collector-configuration)
The following example configures the Tinybird OpenTelemetry Collector to export metrics, traces, and logs to Tinybird:
config.yaml
```
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
batch:
timeout: 10s
send_batch_size: 8192
exporters:
tinybird:
endpoint: ${OTEL_TINYBIRD_API_HOST} # Your Events API endpoint, e.g. https://api.us-east.aws.tinybird.co
token: ${OTEL_TINYBIRD_TOKEN} # Token with append permissions
sending_queue:
enabled: true
queue_size: 104857600 # Total memory buffer in bytes (100 MB)
sizer: bytes
batch:
flush_timeout: 5s # Max wait time before flushing
min_size: 1024000 # Min batch size: 1 MB
max_size: 8388608 # Max batch size: 8 MB (Events API limit is 10 MB)
retry_on_failure:
enabled: true
metrics:
sum:
datasource: otel_metrics_sum
histogram:
datasource: otel_metrics_histogram
exponential_histogram:
datasource: otel_metrics_exponential_histogram
gauge:
datasource: otel_metrics_gauge
logs:
datasource: otel_logs
traces:
datasource: otel_traces
service:
pipelines:
metrics:
receivers: [otlp]
processors: [batch]
exporters: [tinybird]
traces:
receivers: [otlp]
processors: [batch]
exporters: [tinybird]
logs:
receivers: [otlp]
processors: [batch]
exporters: [tinybird]
```
### Environment variables (https://www.tinybird.co/docs/forward/guides/ingest-from-opentelemetry#environment-variables)
- `OTEL_TINYBIRD_API_HOST`: The API host for your Tinybird Workspace, for example `{% user("apiHost", "https://api.tinybird.co") %}`.
- `OTEL_TINYBIRD_TOKEN`: A Tinybird token with **append** permissions to the target Data Sources (`otel_metrics`, `otel_traces`, `otel_logs`).
You can create a token in the Tinybird UI under **Tokens**. Make sure it has the required append permissions for the Data Sources you want to ingest into.
### Run the collector (https://www.tinybird.co/docs/forward/guides/ingest-from-opentelemetry#run-the-collector)
#### Using the binary (https://www.tinybird.co/docs/forward/guides/ingest-from-opentelemetry#using-the-binary)
Precompiled binaries for Linux and macOS are available for both `amd64` and `arm64` architectures.
Download them from the [GitHub Releases page](https://github.com/tinybirdco/opentelemetry-collector-contrib/releases).
```
./otelcontribcol_linux_amd64 --config config.yaml
```
#### Using Docker (https://www.tinybird.co/docs/forward/guides/ingest-from-opentelemetry#using-docker)
```
docker run \
--platform linux/amd64 \
-v $(pwd)/config.yaml:/etc/otelcol-contrib/config.yaml \
-p 4317:4317 \
-p 4318:4318 \
-e OTEL_TINYBIRD_API_HOST="${OTEL_TINYBIRD_API_HOST}" \
-e OTEL_TINYBIRD_TOKEN="${OTEL_TINYBIRD_TOKEN}" \
tinybirdco/opentelemetry-collector-contrib:v0.128.0
```
Use the `--platform linux/amd64` flag to ensure compatibility when running on ARM-based systems like Apple Silicon Macs.
### Troubleshooting (https://www.tinybird.co/docs/forward/guides/ingest-from-opentelemetry#troubleshooting)
To troubleshoot your setup, edit your `config.yaml` to enable detailed logging from the Collector. You can also check the [Tinybird Service Data Sources](/docs/forward/monitoring/service-datasources) to confirm data is arriving in your Workspace.
To enable detailed logging, define the `debug` exporter and add it to your service pipelines:
config.yaml
```
#(...)
exporters:
debug:
verbosity: detailed
exporters:
tinybird:
endpoint: ${OTEL_TINYBIRD_API_HOST}
#(...)
service:
pipelines:
logs:
receivers: [otlp]
processors: [batch]
exporters: [debug,tinybird]
#(...)
```
You can send mock data with [otelgen](https://github.com/krzko/otelgen).
## Next steps (https://www.tinybird.co/docs/forward/guides/ingest-from-opentelemetry#next-steps)
- Explore and customize the [Tinybird OpenTelemetry project template](https://github.com/tinybirdco/tinybird-otel-template) to fit your needs.
- Run the official [opentelemetry-demo](https://github.com/tinybirdco/opentelemetry-demo) locally.
- Connect to [Grafana](/forward/guides/connect-grafana), HyperDX and other [visualization tools](/forward/guides).
For more details on the available configuration options, see the [Tinybird OpenTelemetry Collector documentation](https://github.com/tinybirdco/opentelemetry-collector-contrib).
---
URL: https://www.tinybird.co/docs/forward/guides/ingest-from-mongodb
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Ingest data from MongoDB \| Tinybird"
theme-color: "#171612"
description: "In this guide, you'll learn how to ingest data into Tinybird from MongoDB."
inkeep:version: "forward"
---
# Connect MongoDB to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-mongodb#connect-mongodb-to-tinybird)
This guide shows you how to ingest data into Tinybird from MongoDB.
Use these components:
- MongoDB Atlas as the source MongoDB database.
- Confluent Cloud's MongoDB Atlas Source connector to capture change events from MongoDB Atlas and push to Kafka
- Tinybird Confluent Cloud connector to ingest the data from Kafka
This guide uses Confluent Cloud as a managed Kafka service, and MongoDB Atlas as a managed MongoDB service. You can use any Kafka service and MongoDB instance, but the setup steps may vary.
## Prerequisites (https://www.tinybird.co/docs/forward/guides/ingest-from-mongodb#prerequisites)
This guide assumes you have:
- An existing Tinybird account & workspace
- An existing Confluent Cloud account
- An existing MongoDB Atlas account & collection
## 1. Create a Confluent Cloud source (https://www.tinybird.co/docs/forward/guides/ingest-from-mongodb#1-create-a-confluent-cloud-source)
[Create a new MongoDB Atlas Source in Confluent Cloud](https://docs.confluent.io/cloud/current/connectors/cc-mongo-db-source.html#get-started-with-the-mongodb-atlas-source-connector-for-ccloud). Use the following template to configure the Source:
```
{
"name": "",
"config": {
"name": "",
"connection.host": "",
"connection.user": "",
"connection.password": "",
"database": "",
"collection": "",
"cloud.provider": "",
"cloud.environment": "",
"kafka.region": "",
"kafka.auth.mode": "KAFKA_API_KEY",
"kafka.api.key": "",
"kafka.api.secret": "",
"kafka.endpoint": "",
"topic.prefix": "",
"errors.deadletterqueue.topic.name": "",
"startup.mode": "copy_existing",
"copy.existing": "true",
"copy.existing.max.threads": "1",
"copy.existing.queue.size": "16000",
"poll.await.time.ms": "5000",
"poll.max.batch.size": "1000",
"heartbeat.interval.ms": "10000",
"errors.tolerance": "all",
"max.batch.size": "100",
"connector.class": "MongoDbAtlasSource",
"output.data.format": "JSON",
"output.json.format": "SimplifiedJson",
"json.output.decimal.format": "NUMERIC",
"change.stream.full.document": "updateLookup",
"change.stream.full.document.before.change": "whenAvailable",
"tasks.max": "1"
}
}
```
When the source is created, a new Kafka topic appears in your Confluent Cloud account. This topic contains the change events from your MongoDB collection.
## 2. Create Tinybird data source (https://www.tinybird.co/docs/forward/guides/ingest-from-mongodb#2-create-tinybird-data-source)
Create a new Kafka data source in your Tinybird project. See [Kafka](/docs/forward/ingest-data/connectors/kafka) for more information.
The data source should have the following schema:
```
SCHEMA >
`_id` String `json:$.documentKey._id` DEFAULT JSONExtractString(__value, '_id._id'),
`operation_type` LowCardinality(String) `json:$.operationType`,
`database` LowCardinality(String) `json:$.ns.db`,
`collection` LowCardinality(String) `json:$.ns.coll`
ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(__timestamp)"
ENGINE_SORTING_KEY "__timestamp, _id"
KAFKA_CONNECTION_NAME ''
KAFKA_TOPIC ''
KAFKA_GROUP_ID ''
KAFKA_AUTO_OFFSET_RESET 'earliest'
KAFKA_STORE_RAW_VALUE 'True'
KAFKA_STORE_HEADERS 'False'
KAFKA_STORE_BINARY_HEADERS 'True'
KAFKA_TARGET_PARTITIONS 'auto'
KAFKA_KEY_AVRO_DESERIALIZATION ''
```
Deploy the Data Source before proceeding to the next step.
## 3. Validate the data source (https://www.tinybird.co/docs/forward/guides/ingest-from-mongodb#3-validate-the-data-source)
Go to Tinybird Cloud and validate that a data source has been created.
As changes occur in MongoDB, you should see the data being ingested into Tinybird. This is an append log of all changes, so multiple records appear for the same document as it's updated.
## 4. Deduplicate with ReplacingMergeTree (https://www.tinybird.co/docs/forward/guides/ingest-from-mongodb#4-deduplicate-with-replacingmergetree)
Tinybird creates a new data source using the ReplacingMergeTree engine to store the deduplicated data, and a pipe to process the data from the original data source and write to the new data source.
First, create a new data source to store the deduplicated data.
Create a new file called `deduped_ds.datasource` and add the following content:
```
SCHEMA >
`fullDocument` String,
`_id` String,
`database` LowCardinality(String),
`collection` LowCardinality(String),
`k_timestamp` DateTime,
`is_deleted` UInt8
ENGINE "ReplacingMergeTree"
ENGINE_SORTING_KEY "_id"
ENGINE_VER "k_timestamp"
ENGINE_IS_DELETED "is_deleted"
```
Then, create a new file called `dedupe_mongo.pipe` and add the following content:
```
NODE mv
SQL >
SELECT
JSONExtractRaw(__value, 'fullDocument') as fullDocument,
_id,
database,
collection,
__timestamp as k_timestamp,
if(operation_type = 'delete', 1, 0) as is_deleted
FROM