--- title: "ClickHouse cluster operations in production" excerpt: "What production ClickHouse cluster operations actually are: add replicas, rebalance traffic, and decide who executes the change." authors: "Tinybird" categories: "AI Resources" createdOn: "2026-08-20 00:00:00" publishedOn: "2026-08-20 00:00:00" updatedOn: "2026-08-20 00:00:00" status: "published" --- ClickHouse{% sup %}®{% /sup %} cluster operations are the jobs that start after `CREATE TABLE`. Add a replica. Take one out. Move ingest off a hot node. Keep queries on a replica you are willing to size for p99. Upgrade without splitting the cluster across incompatible versions. Drain a node without leaving ghost metadata in Keeper. Search results for "ClickHouse cluster operations" often jump to `system` tables and config snippets. Those matter. They are not the operating-model question. The operating-model question is: when production needs more capacity, who is allowed to change the cluster, and who is responsible if the change fails. This post is the ops-shaped entry. For the buyer's comparison, see [self-hosted vs managed ClickHouse](https://www.tinybird.co/blog/self-hosted-vs-managed-clickhouse). For the four production models, see [ClickHouse operating models](https://www.tinybird.co/blog/clickhouse-operating-models). ## What cluster operations means in production A healthy ClickHouse cluster is a set of replicas (and sometimes shards), a coordination layer (ClickHouse Keeper or ZooKeeper), and a way to route reads and writes. Operations are the changes you make to that topology under load. The recurring ones: - **Add a replica.** Provision compute and storage, register it in Keeper, wait until replication is current, then send traffic. - **Remove a replica.** Drain traffic, confirm the remaining set can take the load, drop Keeper metadata, destroy the node. - **Rebalance workloads.** Ingest, queries, and maintenance jobs do not want the same replica. A write-heavy replica will wreck your p99 if it also serves user-facing queries. - **Vertical resize.** Bigger SKU on existing replicas when the bottleneck is per-query CPU or memory, not concurrency. - **Upgrade.** Add a replica on the new version, send read traffic, then writes, then roll the rest. [Tinybird's own upgrade path](https://www.tinybird.co/blog/what-i-learned-operating-clickhouse) took years to make non-events. - **Survive merges.** Inserts create parts. Merges compact them. If merges lose, you hit `Too many parts` and ingest dies while the process still looks "up." If you self-host, you own every line. If you use a conventional managed database, you own the plan and often wait on a control plane or a ticket for the change. If you use Tinybird dedicated infrastructure with Cluster Management, you own the decision (replica count, SKU, weights). Tinybird owns execution and platform operations. ## Self-hosted: the actual add-replica runbook Self-hosted production tables use `ReplicatedMergeTree`. Keeper stores the replication log and table metadata. The guiding principle: bring the node fully into sync before you advertise it to clients. 1. Provision a machine (or pod) that matches CPU, memory, and disk of the existing replicas. Mismatched SKUs make one replica the straggler forever. 2. Point `macros` (`{shard}`, `{replica}`) and Keeper config at the same ensemble. Production Keeper is three nodes, odd count, not colocated with ClickHouse. Two nodes cannot form quorum if one dies. 3. Create the replicated tables with the same Keeper path and a new replica name. ClickHouse registers the replica and starts `GET_PART` from peers. You do not rsync data files. 4. Watch `system.replicas`: `queue_size` and `absolute_delay` need to be zero (or as close as you tolerate). `is_readonly` and `is_session_expired` must be 0. 5. Only then add the host to `remote_servers` on every node, and only then add it to the load balancer. Advertise early and Distributed / HTTP clients will hit a node still fetching parts and return partial results. 6. Shift traffic. Confirm query latency and insert success before you go back to bed. Removal is the reverse, with one extra landmine: stopping the server is not enough. `SYSTEM DROP REPLICA` clears Keeper. Leave ghost entries and you bloat queues with references to a node that no longer exists. ### What the system tables are for ```sql SELECT database, table, replica_name, is_readonly, is_session_expired, queue_size, absolute_delay, insert_quorum_exception FROM system.replicas WHERE queue_size > 0 OR is_readonly FORMAT Vertical; SELECT type, count() AS n, max(create_time) AS oldest FROM system.replication_queue GROUP BY type ORDER BY n DESC; ``` `GET_PART` backlog: fetches are slow, raise `background_fetches_pool_size` or fix the network. `MERGE_PARTS` stuck for hours: merges lost to CPU/IO, or a Keeper session dropped. `SYSTEM RESTART REPLICA` reconciles an in-memory queue that drifted after a Keeper reconnect. It is not a substitute for fixing insert size. `Too many parts` is inserts creating parts faster than merges can compact them. Defaults delay inserts around 1000 active parts per partition and throw around 3000. `OPTIMIZE TABLE ... FINAL` is a panic button, not a strategy. Batch inserts, write one partition at a time, use Compact parts on object storage. ### Shards are not replicas Replicas are copies. Shards are splits. ClickHouse has no built-in online resharding. Add a shard and new inserts spread. Historical partitions stay on the old shard until you `INSERT SELECT` them through a new Distributed table or a migration tool. Size shard count with headroom. Do not plan to "just add a shard later" as the 2 a.m. move. `internal_replication = true` on Distributed tables that sit on `Replicated*` locals. If it is false, Distributed writes every replica and ReplicatedMergeTree replicates again. Duplicates. Materialized views lie. You find out in the quarterly board deck. ### Workload isolation without a vendor Self-hosted teams put a load balancer in front and route by request type: a replica that takes inserts, replicas that take user queries, maybe one kept under 40% load for a p99 SLO. Javi's architecture at Tinybird started as replicas only, no shards, writes isolated, HTTP not native TCP. You can build that. You operate the balancer, the health checks, and the night the write replica OOMs because a materialized view used more memory than the insert. ## Conventional managed: topology is a console, the app layer is still yours ClickHouse Cloud's `SharedMergeTree` stores durable data in object storage. Adding compute does not copy parts between local disks. Metadata still goes through Keeper, which they run. That removes the worst of the self-hosted add-replica wait, on their control plane, on their SKU. You still do not get a production API out of the box. Connection pools, retries, and ingest batching remain yours. Basic cannot autoscale. Scale and Enterprise can, with policies you configure in advance, which is not the same as a human adding a replica during an incident. Altinity.Cloud is closer to OSS operations with a support contract: you may add replicas and shards yourself, on their Kubernetes, with their on-call for the database. You still design tables and build the serving path. Managed does not mean `Too many parts` goes away. It means you are not SSH-ing into Keeper at 2 a.m. You are still on the hook for the query that created the parts. ## Tinybird Cluster Management Tinybird dedicated infrastructure gives Enterprise customers an isolated ClickHouse cluster. Cluster Management is how you change that cluster without taking the platform pager. The path here was incremental, which is the honest version: 1. Usage charts for dedicated infra (observability without control). 2. Resize requests in the UI (vertical, still a request). 3. [Compute-compute separation for populates](https://www.tinybird.co/blog/compute-compute-separation-for-populates): ephemeral replicas for backfills, so a populate does not fight APIs for CPU. 4. Self-serve replicas in the UI: add/remove replicas, set weights, no ticket. 5. [Organizations API](https://www.tinybird.co/blog/cluster-management-api) (March 2026): the same operations in a script. Eligible dedicated clusters only. Shared Tinybird workspaces do not expose replica topology. Tinybird operates those clusters without a customer-facing replica API. Self-Managed Tinybird means you operate replicas yourself. ### Why weights exist ClickHouse does not magically isolate ingest from queries. You isolate them with routing. Tinybird dedicated clusters expose three weights per replica, each in `0-65535`, distributed with weighted round-robin: | Weight | What it controls | Rule of thumb | | --- | --- | --- | | **Read** | Query traffic | At least one replica must be non-zero. `0` means that replica serves no queries. | | **Write** | Events API and Kafka ingest | At least one replica must be non-zero. `0` parks ingest off that replica. | | **Copy** | Copy jobs | Default and recommended: `1` on every replica. | A common production layout is one replica biased to writes, the others biased to reads. Every replica still holds the data. You are placing load, not splitting the dataset. Guardrails refuse a config that would leave the cluster with no reads or no writes. When you add a replica, Tinybird provisions it in your cloud and region (AWS or GCP), replicates the dataset, and prepares it for production workloads. Any replica can serve any query or handle any ingest once weights say so. You do not edit `remote_servers`. You do not `SYSTEM DROP REPLICA` by hand. This is not autoscaling. You (or your script) request the change. Tinybird does not infer capacity from CPU and add nodes on its own. It is not sharding on demand. It is not a fix for a bad partition key. Vertical resize (changing SKU on existing replicas) is a separate dedicated-infra control. Horizontal replica changes and weight changes are the Cluster Management surface. ### Scripting the change Until 2026, dedicated customers did this in the UI (**Settings → Plan & Billing → Manage Cluster**). Mutating API calls require `old_weights`. GET `/v0/organizations//clusters-configuration` first and send the returned weights back. Optimistic concurrency: if someone else changed the cluster between GET and PUT, the request fails instead of applying a stale plan. Rebalance reads toward a second replica, keep writes on the first: ```bash curl -X PUT \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ "https://api.tinybird.co/v0/organizations//clusters//weights" \ -d '{ "old_weights": { "copyjob": {"": 1, "": 1}, "writer": {"": 100, "": 0}, "reader": {"": 100, "": 0} }, "new_weights": { "copyjob": {"": 1, "": 1}, "writer": {"": 100, "": 0}, "reader": {"": 50, "": 50} } }' ``` Add a replica and give it 30% of writes and 20% of reads in the same request: ```bash curl -X POST \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ "https://api.tinybird.co/v0/organizations//clusters//replicas" \ -d '{ "replica_size": "4-16", "existing_replicas": { "old_weights": { "copyjob": {"": 1}, "writer": {"": 100}, "reader": {"": 100} }, "new_weights": { "copyjob": {"": 1}, "writer": {"": 70}, "reader": {"": 80} } }, "new_replica": { "copyjob": 1, "writer": 30, "reader": 20 } }' ``` Drain a replica and remove it: ```bash curl -X DELETE \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ "https://api.tinybird.co/v0/organizations//clusters//replicas/" \ -d '{ "old_weights": { "copyjob": {"": 1, "": 1, "": 1}, "writer": {"": 50, "": 30, "": 20}, "reader": {"": 50, "": 30, "": 20} }, "new_weights": { "copyjob": {"": 1, "": 1}, "writer": {"": 60, "": 40}, "reader": {"": 60, "": 40} } }' ``` Endpoints: - `PUT /v0/organizations/{organization_id}/clusters/{cluster_id}/weights` - `POST /v0/organizations/{organization_id}/clusters/{cluster_id}/replicas` - `DELETE /v0/organizations/{organization_id}/clusters/{cluster_id}/replicas/{replica_name}` Permissions match the UI. HTTP 200 on POST is not "queries are already landing there." Tinybird still has to provision and replicate. Watch cluster configuration and endpoint latency after the operation completes. If an operation fails partway, GET `clusters-configuration` again and retry with fresh `old_weights`. Do not fire a second POST from a stale payload. Adding a replica starts billing for that Dedicated ClickHouse cluster capacity. Removing one stops it. Do not assume overlap cost or duration from this post. Confirm against the product and your contract before you automate. ## The 2:07 a.m. runbook, three ways Production needs more read capacity. Ingest is fine. Queries are not. **Self-managed.** Page the person with AWS or GCP access and Keeper credentials. Run the add-replica sequence above. If the cluster is sharded and the hot shard is the problem, adding a replica of that shard helps. Adding a new shard does not move yesterday's partitions. Budget longer than the incident if this is the first time in six months. **Conventional managed.** Open the vendor console or file a ticket. ClickHouse Cloud may add compute against object storage without a part copy. Your application still needs its own pool and retry story while you wait. Basic cannot do this. **Tinybird dedicated, Cluster Management enabled.** Add a replica or raise read weights on the replicas that should take queries. Tinybird provisions, replicates, applies weights. Built-in guardrails refuse a config that would leave the cluster with no reads or no writes. That third path is the product proof behind Tinybird dedicated. You decide. Tinybird operates. {% cta title="Cluster operations without the ClickHouse shift" text="Walk the same capacity incident on self-managed, conventional managed, and Tinybird dedicated infrastructure." button={href: "https://www.tinybird.co/control?utm_source=blog&utm_medium=organic&utm_campaign=control-without-self-hosting&entry=cluster-ops", target: "_blank", text: "Run the cluster-ops path"} /%} ## Frequently Asked Questions (FAQs) ### Can I automate replica changes from my own control plane? Yes, if Cluster Management is enabled on your dedicated cluster. Use the Organizations API, always GET current weights first, and send them as `old_weights`. Wire it into the same place you keep other production runbooks. Shared plans have no replica API. ### What happens if I set every write weight to 0? The API and UI reject it. At least one replica must accept writes, and at least one must accept reads. ### Do new replicas serve traffic immediately? Tinybird provisions the replica and replicates the dataset before it is ready for production workloads. Do not assume HTTP 200 on POST equals queries already landing there. Watch cluster configuration and query latency after the operation completes. Self-hosted: never add the host to `remote_servers` until `queue_size` and `absolute_delay` are acceptable. ### Is this available on Tinybird Self-Managed? Cluster Management as described here is a Tinybird Cloud dedicated-infrastructure feature. Self-Managed means you operate the environment, including replica topology and Keeper, yourself. ### Where do I start if I am not on dedicated infrastructure? Tinybird Enterprise includes dedicated clusters. If you are evaluating isolation and replica control, talk to us about the production workload rather than trying to approximate this on shared infrastructure. ### Does Cluster Management replace query optimization? No. Weights move traffic. They do not fix a partition key that writes to thousands of parts, a materialized view that OOMs, or an unbounded `SELECT *`. Treat Cluster Management as topology. Treat `system.query_log` as homework. ### Why not just use ClickHouse Cloud autoscaling? If you want a hosted database that adds compute against object storage on a policy you configured last quarter, ClickHouse Cloud Scale/Enterprise is a coherent product. If you want to decide replica count and ingest-vs-query placement yourself, keep SQL-to-API and git deploys, and not staff Keeper, Tinybird dedicated plus Cluster Management is the other product. Autoscaling is a policy. Cluster Management is a decision you make, including at 2 a.m.