Ingestion protection

Tinybird provides ingestion flow control mechanisms to prevent resource saturation and ensure fair usage across shared infrastructure.
These mechanisms protect your ingestion pipelines from causing general service degradation or even data loss, maintaining overall system health and reliability.

When needed, these mechanisms may temporarily delay ingestion (reducing data freshness) or temporarily reject ingestion requests.
In all cases, Tinybird monitors these events closely and sends you notifications so that you can take timely corrective action.

Understanding the challenges of data ingestion

ClickHouse® is not optimized for high-frequency writes.
Frequent inserts can saturate system resources, but many real-time use cases benefit from near-continuous data ingestion.
Asynchronous Inserts can help in some cases, but they come with important limitations, including potential issues with Materialized Views and the risk of data duplication.

Tinybird goes beyond the base capabilities of ClickHouse® to offer a complete data ingestion solution.
You can send high-frequency data through:

To make the most of these services, it’s important to:

  • Understand your ingestion patterns.
  • Plan your throughput and resource requirements.
  • Monitor and act when ingestion begins to saturate your current plan’s limits.

This section explains how Tinybird’s High-Frequency Ingestion (HFI) platform and backpressure mechanisms preserve system stability for both dedicated and shared environments.

How frequent ingestion works

To avoid overwhelming the ClickHouse® clusters, Tinybird gathers incoming insertions over a short period known as the flush interval before writing (flushing) them into the ClickHouse® tables that back your landing Data Sources.

  • The flush interval duration depends on your workspace’s billing plan.
  • A longer flush interval reduces cluster load but increases data freshness latency.

When ingestion flushes occur, data is written separately for each partition of the destination table.
Therefore:

When writes fail

Ingestion into a landing Data Source may fail for multiple reasons, including:

  • Cluster saturation
  • Network connectivity problems
  • Timeout errors
  • SQL execution errors

Tinybird classifies errors as either:

  • Retriable — temporary issues that can be retried later.
  • Non-retriable — permanent issues such as invalid data or SQL errors.

Retriable failures

Data affected by retriable errors is stored in a staging area and automatically retried later.
During this process:

  • You may experience ingestion delays.
  • Tinybird’s ingestion logic avoids duplicates caused for partially inserted data for up to 5 hours.
    After this 5-hour window, de-duplication is not guaranteed.

Non-retriable failures

Data affected by non-retriable errors is kept in the quarantine table associated with your Data Source.
You can inspect these records and take corrective action manually.

Be aware of failures

Backpressure and flow control

Tinybird’s backpressure mechanisms preserve the health of your ingestion infrastructure by isolating problematic Data Sources before they affect others.

When a particular Data Source causes excessive resource consumption — for example, due to:

  • Inefficient materialized views
  • Poor partitioning configuration
  • Long-running queries

— the system automatically applies backpressure in two escalating phases.

1. Delayed ingestion

The first line of defense is delayed ingestion.
When backpressure is activated, insertions are temporarily delayed and resumed when resource consumption returns to safe levels.

Tinybird continuously monitors:

  • The number of delayed insertions
  • The age of pending insertions

If these metrics increase beyond safe limits, the system escalates protection to the next phase.

Typical triggers for delayed ingestion:

  • High memory usage
  • Excessive throughput, exceeding cluster capacity

Throughput overload can result not only from ingestion frequency but also from:

  • Large payload sizes
  • Long insert durations (e.g., due to complex materialized views)
  • High partition counts
  • Your cluster’s capacity, defined by your billing plan, which determines when these limits are reached.

Whenever possible, Tinybird applies the delay only to the affected Data Source, minimizing the impact on your workspace or cluster.

During delayed ingestion:

  • Data is accepted, but its insertion into landing tables is postponed.
  • You’ll see reduced data freshness (similar to a longer flush interval).
  • If the issue is transient, ingestion will recover automatically.

You will receive an email notification whenever delayed ingestion is activated.
Always review these notifications and take corrective action if necessary.

2. Temporary ingestion restriction

If ingestion delays persist and accumulated data grows beyond safe thresholds, Tinybird temporarily rate limits ingestion for the affected Data Source (by applying a temporary rate limit of 1 request/message per second for one minute).

These restrinctions can be applied also when the CPU usage exceeds the plan limits.

When this occurs:

  • Events API calls return HTTP 429 responses with details about the cause.
  • Data Connectors are paused temporarily and will automatically attempt recovery.

You’ll receive an email alert immediately when ingestion is denied.

This temporary service denial prevents cascading issues that could otherwise impact system stability.
In most cases, ingestion resumes automatically once the issue is resolved.

If the problem persists:

  • Review the metrics and logs associated with your ingestion workload.
  • Check for inefficient queries, heavy partitioning, or high system load.
  • Contact Tinybird Support if you cannot identify the root cause.

In rare cases, ingestion protection may be triggered by issues unrelated to your ingestion (e.g., cloud provider disruptions).
Rest assured that our monitoring systems will detect and remediate such situations as quickly as possible.

Best practices

To reduce the likelihood of backpressure activation:

  • Design efficient partitioning strategies.
  • Optimize materialized views and avoid long-running transformations during ingestion.
  • Batch inserts where possible rather than sending individual events. Use the appropriate ingestion mechanism to your scenario (e.g. for large, infrequent writes you can ingest data from files or from S3.
  • Monitor ingestion metrics regularly.
  • Pay attention to flush intervals, notifications, and system alerts.
  • Choose the right billing plan for your ingestion needs, and consider upgrading when ingestion volume or concurrency approaches the limits of your current plan.

Summary

Tinybird’s ingestion protection mechanisms ensure:

  • Fair usage across users and workspaces
  • Reliability of ingestion pipelines
  • Automatic recovery from transient issues
  • Transparent communication through alerts and monitoring

These mechanisms — combining delayed ingestion, automatic retries, and temporary rate limiting — form a robust framework to keep your ingestion stable, efficient, and predictable.

See also:

Updated