---
last_updated: "2026-09-29T12:51:56.000Z"
title: Copy Pipes
meta:
    description: Copy pipes capture the result of a pipe at a moment in time and write the result into a target data source. They can be run on a schedule, or executed on demand.
---

# Copy Pipes {% id="copy-pipes" %}

Copy pipes capture the result of a pipe at a moment in time and write the result into a target data source. They can be run on a schedule, or executed on demand.

Copy Pipes should not be confused with [Materialized Views](/forward/core-concepts/materialized-views). While Materialized Views continuously update as new events are inserted, Copy Pipes generate a single snapshot at a specific point in time.

## Create a copy pipe

Copy pipes are defined with SQL nodes and a target Data Source. Use `TYPE COPY` in `.pipe` files, `defineCopyPipe` in the TypeScript SDK, or `define_copy_pipe` in the Python SDK.

Define the queries that filter and transform the data as needed. The final result is written into the target Data Source. Before pushing a Copy Pipe to your Workspace, make sure that the target Data Source already exists and has a schema that matches the query output.

The following example creates a Copy Pipe scheduled to run every hour and write the query result into the `sales_hour_copy` Data Source.

Copy Pipes can use Tinybird's [templating language](/forward/core-concepts/templating-language) in SQL nodes. Scheduled Copy Pipes must provide defaults for parameters because there is no request-time value.

{% tabs initial="Tinybird CLI" %}
{% tab label="Tinybird CLI" %}

```tinybird {% title="datasources/teams.datasource" %}
SCHEMA >
    `starting_date` DateTime,
    `country` String,
    `sales` Float64

ENGINE "MergeTree"
ENGINE_SORTING_KEY "starting_date, country"
```

```tinybird {% title="datasources/sales_hour_copy.datasource" %}
SCHEMA >
    `day` DateTime,
    `country` String,
    `total_sales` Float64

ENGINE "MergeTree"
ENGINE_SORTING_KEY "day, country"
```

```tinybird {% title="pipes/sales_hour_copy.pipe" %}
NODE daily_sales
SQL >
    %
    SELECT toStartOfDay(starting_date) day, country, sum(sales) as total_sales
    FROM teams
    WHERE
    day BETWEEN toStartOfDay({{DateTime(job_timestamp)}}) - interval 1 day AND toStartOfDay({{DateTime(job_timestamp)}})
    and country = {{ String(country, 'US')}}
    GROUP BY day, country

TYPE COPY
TARGET_DATASOURCE sales_hour_copy
COPY_SCHEDULE 0 * * * *
COPY_MODE append
```

{% /tab %}

{% tab label="TypeScript SDK" %}

```typescript {% title="tinybird.ts" %}
import { defineCopyPipe, defineDatasource, engine, node, t } from "@tinybirdco/sdk";

export const teams = defineDatasource("teams", {
  schema: {
    starting_date: t.dateTime(),
    country: t.string(),
    sales: t.float64(),
  },
  engine: engine.mergeTree({
    sortingKey: ["starting_date", "country"],
  }),
});

export const salesHourCopy = defineDatasource("sales_hour_copy", {
  schema: {
    day: t.dateTime(),
    country: t.string(),
    total_sales: t.float64(),
  },
  engine: engine.mergeTree({
    sortingKey: ["day", "country"],
  }),
});

export const salesHourCopyPipe = defineCopyPipe("sales_hour_copy_pipe", {
  datasource: salesHourCopy,
  schedule: "0 * * * *",
  mode: "append",
  nodes: [
    node({
      name: "daily_sales",
      sql: `
        SELECT toStartOfDay(starting_date) day, country, sum(sales) as total_sales
        FROM teams
        WHERE day BETWEEN toStartOfDay({{DateTime(job_timestamp)}}) - interval 1 day
          AND toStartOfDay({{DateTime(job_timestamp)}})
          AND country = {{String(country, 'US')}}
        GROUP BY day, country
      `,
    }),
  ],
});
```

{% /tab %}

{% tab label="Python SDK" %}

```python {% title="tinybird.py" %}
from tinybird_sdk import define_copy_pipe, define_datasource, engine, node, t

teams = define_datasource("teams", {
    "schema": {
        "starting_date": t.date_time(),
        "country": t.string(),
        "sales": t.float64(),
    },
    "engine": engine.merge_tree({
        "sorting_key": ["starting_date", "country"],
    }),
})

sales_hour_copy = define_datasource("sales_hour_copy", {
    "schema": {
        "day": t.date_time(),
        "country": t.string(),
        "total_sales": t.float64(),
    },
    "engine": engine.merge_tree({
        "sorting_key": ["day", "country"],
    }),
})

sales_hour_copy_pipe = define_copy_pipe("sales_hour_copy_pipe", {
    "datasource": sales_hour_copy,
    "copy_schedule": "0 * * * *",
    "copy_mode": "append",
    "nodes": [
        node({
            "name": "daily_sales",
            "sql": """
                SELECT toStartOfDay(starting_date) day, country, sum(sales) as total_sales
                FROM teams
                WHERE day BETWEEN toStartOfDay({{DateTime(job_timestamp)}}) - interval 1 day
                  AND toStartOfDay({{DateTime(job_timestamp)}})
                  AND country = {{String(country, 'US')}}
                GROUP BY day, country
            """,
        }),
    ],
})
```

{% /tab %}
{% /tabs %}

See all syntax options in the [Copy Pipes reference](/forward/dev-reference/datafiles/pipe-files#copy-pipes), [TypeScript SDK reference](/forward/dev-reference/typescript-sdk-resources#copy-pipes), and [Python SDK reference](/forward/dev-reference/python-sdk-resources#copy-pipes).

## Schedule a copy pipe

You can schedule Copy Pipes to run at a specific time using a cron expression. To schedule a Copy Pipe, configure `COPY_SCHEDULE` with a cron expression. On-demand Copy Pipes are defined by configuring `COPY_SCHEDULE` with the value `@on-demand`.

{% callout type="caution" %}
Copy Pipes scheduled with a cron expression have a maximum timeout defined by the lower of the limit enforced by your plan and half the interval of your cron expression. For example, a Copy Pipe scheduled to run every 5 minutes (300 seconds) has a maximum timeout of 2.5 minutes (150 seconds), even if your plan permits an hour. This prevents your Copy Pipes from overlapping.
{% /callout %}

{% callout type="info" %}
All schedules use the UTC time zone. When you configure a schedule that runs at a specific time, convert the desired time from your local time zone to UTC.
{% /callout %}

## Run, pause, or resume a Copy Pipe {% id="run-pause-or-resume-a-copy-pipe" %}

Use the `tb copy` command to run, pause, or resume a Copy Pipe. See [tb copy](/forward/dev-reference/commands/tb-copy).

You can query `tinybird.jobs_log` to see any running jobs, as well as any jobs that have finished during the last 48 hours.

### Run Copy Pipes in Tinybird Local

When running Copy Pipes locally, you might encounter this error:

```text
Error: Failed creating copy job: Error while running job: There was a problem while copying data: [Error] Cannot schedule a task: no free thread (timeout=0) (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK)
```

This occurs when the local ClickHouse® instance runs out of available threads. To fix this, limit the number of threads the copy job can use by adding the `max_threads` parameter at the top of your SQL query:

```tinybird
NODE daily_sales
SQL >
    %
    {{max_threads(1)}}
    SELECT toStartOfDay(starting_date) day, country, sum(sales) as total_sales
    FROM teams
    WHERE
    day BETWEEN toStartOfDay({{DateTime(job_timestamp)}}) - interval 1 day AND toStartOfDay({{DateTime(job_timestamp)}})
    and country = {{ String(country, 'US')}}
    GROUP BY day, country

TYPE COPY
TARGET_DATASOURCE sales_hour_copy
COPY_SCHEDULE 0 * * * *
```

The `max_threads(1)` parameter ensures the copy job uses only one thread, preventing the scheduling error in local environments with limited resources.

## On-demand compute for copy jobs

Copy jobs run on your Workspace compute by default, so a heavy copy competes with your Endpoints and ingestion for resources. You can run copy jobs on on-demand compute instead: Tinybird provisions a dedicated ClickHouse® instance for the duration of the job and releases it when the job finishes. This keeps long or heavy copies from slowing down production queries.

On-demand compute for copy jobs is available on paid plans, on both shared and dedicated infrastructure. Unlike deployment populates, it never activates automatically based on data size. Opt in per Copy Pipe for scheduled runs, or per run for manual runs.

### Scheduled runs

To run every scheduled execution of a Copy Pipe on on-demand compute, add `ON_DEMAND_COMPUTE true` to the copy node of your `.pipe` file and deploy:

```tinybird {% title="pipes/sales_hour_copy.pipe" %}
NODE daily_sales
SQL >
    %
    SELECT toStartOfDay(starting_date) day, country, sum(sales) as total_sales
    FROM teams
    WHERE
    day BETWEEN toStartOfDay(now()) - interval 1 day AND toStartOfDay(now())
    and country = {{ String(country, 'US')}}
    GROUP BY day, country

TYPE COPY
TARGET_DATASOURCE sales_hour_copy
COPY_MODE append
COPY_SCHEDULE 0 * * * *
ON_DEMAND_COMPUTE true
```

After the deployment, every execution triggered by `COPY_SCHEDULE` runs on on-demand compute. To move scheduled runs back to your Workspace compute, set the value to `false` or remove the line, then deploy again. You can change this setting on its own, without touching the schedule or the query.

`ON_DEMAND_COMPUTE` only affects scheduled executions. It has no effect on Copy Pipes with `COPY_SCHEDULE @on-demand`, and it doesn't change how manual runs behave. See [Manual runs](#manual-runs).

`ON_DEMAND_COMPUTE` requires Tinybird Forward CLI 4.6.20 or later.

### Manual runs {% id="manual-runs" %}

Manual runs don't read `ON_DEMAND_COMPUTE` from the datafile. To run a copy job on on-demand compute from the CLI, pass `--on-demand-compute` on every run:

```shell {% title="Run a copy job on on-demand compute" %}
tb copy run <copy_pipe_name> --on-demand-compute
```

If the local datafile of the Copy Pipe doesn't set `ON_DEMAND_COMPUTE true`, the CLI asks for confirmation before running, because on-demand compute is billed separately from your regular usage. Pass `--yes` to skip the prompt, for example in CI. See [tb copy](/forward/dev-reference/commands/tb-copy).

You can also trigger a manual run on on-demand compute through the [Copy Pipes API](/api-reference/pipe-api/copy-pipes-api) by passing `on_demand_compute=true`.

### Pricing

On-demand compute is billed by compute time, in credits per core per minute, on top of your plan. Query `tinybird.jobs_log` to review which copy jobs ran and how long they took. For regional rates and example calculations, see [On-demand CPUs](/forward/pricing#on-demand-cpus).

## Next steps

- Learn about [Materialized Views](/forward/core-concepts/materialized-views).
- Publish your data with [API Endpoints](/forward/core-concepts/api-endpoints).