--- title: "On-demand compute for scheduled Copy Pipes" excerpt: "Run scheduled Copy Pipes on temporary dedicated compute by adding ON_DEMAND_COMPUTE true to the datafile." authors: "Rubén del Campo" categories: "Product updates" createdOn: "2026-09-25 13:00:00" publishedOn: "2026-09-25 13:00:00" updatedOn: "2026-09-25 13:00:00" status: "published" --- Scheduled Copy Pipes can now run on on-demand compute. Add `ON_DEMAND_COMPUTE true` to a Copy Pipe datafile and each scheduled run uses temporary, dedicated ClickHouse{% sup %}®{% /sup %} compute instead of the primary compute in your Tinybird workspace. Here is a Copy Pipe that builds an hourly rollup every night: ```tinybird NODE hourly_api_usage SQL > SELECT toStartOfHour(timestamp) AS hour, workspace_id, endpoint, count() AS requests, quantile(0.95)(response_time_ms) AS response_time_p95 FROM api_events WHERE timestamp >= now() - INTERVAL 1 DAY GROUP BY hour, workspace_id, endpoint TYPE COPY TARGET_DATASOURCE api_usage_hourly COPY_SCHEDULE 0 2 * * * COPY_MODE append ON_DEMAND_COMPUTE true ``` The configuration lives with the query, target, mode, and schedule. Deploy the datafile as usual, and Tinybird uses on-demand compute when the schedule triggers the Copy Pipe. ## Isolate recurring copy work from primary compute Copy Pipes run a query and write its result to a target Data Source. They are useful when you need to periodically reshape a large dataset rather than calculate the same expensive result for every request. That work can be substantial. Common examples include: - Creating snapshots from change data capture (CDC) streams, keeping the latest version of each record. - Deduplicating event data before downstream queries consume it. - Calculating hourly or daily aggregates for dashboards and APIs. Without separate compute, a large scheduled copy query uses the same CPU and memory as other work in the workspace. Choosing a quiet hour can help, but it does not isolate the job. Data volumes, job duration, and application traffic all change over time. With `ON_DEMAND_COMPUTE true`, Tinybird provisions temporary dedicated compute for the scheduled job and removes it afterward. The copy query is separated from primary workspace compute while it runs. This is useful for recurring work that is resource-intensive or whose timing overlaps with production traffic. ## Scheduled and manual runs use different controls The datafile property controls scheduled executions: ```tinybird ON_DEMAND_COMPUTE true ``` It is an explicit opt-in. Copy Pipes without the property continue to use their existing execution behavior. For a manual run in Forward, pass the CLI flag: ```bash tb copy run my_copy_pipe --on-demand-compute ``` For a manual run in Classic, use: ```bash tb pipe copy run my_copy_pipe --on-demand-compute ``` The datafile setting does not replace the flag for a manual run. Update your Tinybird CLI before deploying the datafile or triggering a manual job so it recognizes the current Copy Pipe syntax and options. Both `append` and `replace` Copy Pipe modes can use on-demand compute. Choose the mode based on the result you need. `append` adds each result to the target, which suits incremental rollups. `replace` replaces the target contents, which suits a current-state snapshot built from a CDC stream. ## A batch alternative to Materialized Views Materialized Views are usually the first choice for preparing data as it arrives. They process each inserted block incrementally. Copy Pipes are the batch alternative when a transformation needs to see a broader slice of the data, such as a deduplicated snapshot, a window calculation, or an aggregation over a `ReplacingMergeTree` Data Source. That distinction also makes Copy Pipes useful in a [lambda architecture](https://www.tinybird.co/docs/forward/guides/lambda-architecture). A scheduled Copy Pipe periodically builds the batch view from historical data. An Endpoint can combine that snapshot with rows ingested since the last copy, keeping results fresh between runs. On-demand compute isolates the heavier batch step without changing the real-time path. ## Where on-demand compute fits On-demand compute is most useful when the cost of a copy query is large enough to affect other workloads or make capacity planning awkward. Consider a CDC stream containing multiple versions of every customer record. A nightly Copy Pipe can group or order those versions to produce a deduplicated current-state Data Source. As the history grows, the snapshot query must scan and process more rows. Running it on temporary dedicated compute keeps that recurring batch workload separate from primary workspace resources. Periodic aggregations have a similar profile. An hourly or daily job might scan raw events, group by customer and endpoint, and calculate counts or latency quantiles. The resulting Data Source makes downstream API queries cheaper, but building it can require much more CPU and memory than reading it. On-demand compute separates those two workloads. It is also useful when several scheduled copies overlap. Moving the heaviest job to dedicated compute can reduce pressure on the workspace without changing its SQL or target schema. ## Cost and provisioning tradeoffs On-demand compute is available on paid plans. The default on-demand instance has 64 cores. Billing covers the actual compute time, charged in credits per core per minute. The credit price varies by region, so use the [current pricing documentation](https://www.tinybird.co/docs/forward/pricing#on-demand-cpus) when estimating a job rather than relying on a fixed example. There is provisioning overhead before the query starts. For a large deduplication pass or aggregation, that overhead may be a reasonable trade for dedicated resources. A small Copy Pipe that finishes quickly on primary compute may cost less and complete sooner without provisioning another instance. Measure representative runs before enabling the setting across every schedule. Look at job duration, frequency, overlap with other workloads, and the effect the current job has on primary compute. The goal is to isolate the copy jobs that warrant it, not to move every scheduled query by default. ## Get started Add `ON_DEMAND_COMPUTE true` to a scheduled Copy Pipe that benefits from dedicated resources, then deploy it with an up-to-date CLI. Keep using the corresponding `--on-demand-compute` flag when you trigger a manual run. Read the [Copy Pipe documentation](https://www.tinybird.co/docs/forward/core-concepts/copy-pipes) for the full workflow. See [on-demand CPU pricing](https://www.tinybird.co/docs/forward/pricing#on-demand-cpus) for regional credit rates before enabling it on a recurring schedule.