ClickHouse cluster operations in production

ClickHouse^® cluster operations are the jobs that start after CREATE TABLE. Add a replica. Take one out. Move ingest off a hot node. Keep queries on a replica you are willing to size for p99. Upgrade without splitting the cluster across incompatible versions. Drain a node without leaving ghost metadata in Keeper.

Search results for "ClickHouse cluster operations" often jump to system tables and config snippets. Those matter. They are not the operating-model question. The operating-model question is: when production needs more capacity, who is allowed to change the cluster, and who is responsible if the change fails.

This post is the ops-shaped entry. For the buyer's comparison, see self-hosted vs managed ClickHouse. For the four production models, see ClickHouse operating models.

What cluster operations means in production

A healthy ClickHouse cluster is a set of replicas (and sometimes shards), a coordination layer (ClickHouse Keeper or ZooKeeper), and a way to route reads and writes. Operations are the changes you make to that topology under load.

The recurring ones:

• Add a replica. Provision compute and storage, register it in Keeper, wait until replication is current, then send traffic.
• Remove a replica. Drain traffic, confirm the remaining set can take the load, drop Keeper metadata, destroy the node.
• Rebalance workloads. Ingest, queries, and maintenance jobs do not want the same replica. A write-heavy replica will wreck your p99 if it also serves user-facing queries.
• Vertical resize. Bigger SKU on existing replicas when the bottleneck is per-query CPU or memory, not concurrency.
• Upgrade. Add a replica on the new version, send read traffic, then writes, then roll the rest. Tinybird's own upgrade path took years to make non-events.
• Survive merges. Inserts create parts. Merges compact them. If merges lose, you hit Too many parts and ingest dies while the process still looks "up."

If you self-host, you own every line. If you use a conventional managed database, you own the plan and often wait on a control plane or a ticket for the change. If you use Tinybird dedicated infrastructure with Cluster Management, you own the decision (replica count, SKU, weights). Tinybird owns execution and platform operations.

Self-hosted: the actual add-replica runbook

Self-hosted production tables use ReplicatedMergeTree. Keeper stores the replication log and table metadata. The guiding principle: bring the node fully into sync before you advertise it to clients.

1. Provision a machine (or pod) that matches CPU, memory, and disk of the existing replicas. Mismatched SKUs make one replica the straggler forever.
2. Point macros ({shard}, {replica}) and Keeper config at the same ensemble. Production Keeper is three nodes, odd count, not colocated with ClickHouse. Two nodes cannot form quorum if one dies.
3. Create the replicated tables with the same Keeper path and a new replica name. ClickHouse registers the replica and starts GET_PART from peers. You do not rsync data files.
4. Watch system.replicas: queue_size and absolute_delay need to be zero (or as close as you tolerate). is_readonly and is_session_expired must be 0.
5. Only then add the host to remote_servers on every node, and only then add it to the load balancer. Advertise early and Distributed / HTTP clients will hit a node still fetching parts and return partial results.
6. Shift traffic. Confirm query latency and insert success before you go back to bed.

Removal is the reverse, with one extra landmine: stopping the server is not enough. SYSTEM DROP REPLICA clears Keeper. Leave ghost entries and you bloat queues with references to a node that no longer exists.

What the system tables are for

SELECT
    database,
    table,
    replica_name,
    is_readonly,
    is_session_expired,
    queue_size,
    absolute_delay,
    insert_quorum_exception
FROM system.replicas
WHERE queue_size > 0 OR is_readonly
FORMAT Vertical;

SELECT type, count() AS n, max(create_time) AS oldest
FROM system.replication_queue
GROUP BY type
ORDER BY n DESC;

GET_PART backlog: fetches are slow, raise background_fetches_pool_size or fix the network. MERGE_PARTS stuck for hours: merges lost to CPU/IO, or a Keeper session dropped. SYSTEM RESTART REPLICA reconciles an in-memory queue that drifted after a Keeper reconnect. It is not a substitute for fixing insert size.

Too many parts is inserts creating parts faster than merges can compact them. Defaults delay inserts around 1000 active parts per partition and throw around 3000. OPTIMIZE TABLE ... FINAL is a panic button, not a strategy. Batch inserts, write one partition at a time, use Compact parts on object storage.

Shards are not replicas

Replicas are copies. Shards are splits. ClickHouse has no built-in online resharding. Add a shard and new inserts spread. Historical partitions stay on the old shard until you INSERT SELECT them through a new Distributed table or a migration tool. Size shard count with headroom. Do not plan to "just add a shard later" as the 2 a.m. move.

internal_replication = true on Distributed tables that sit on Replicated* locals. If it is false, Distributed writes every replica and ReplicatedMergeTree replicates again. Duplicates. Materialized views lie. You find out in the quarterly board deck.

Workload isolation without a vendor

Self-hosted teams put a load balancer in front and route by request type: a replica that takes inserts, replicas that take user queries, maybe one kept under 40% load for a p99 SLO. Javi's architecture at Tinybird started as replicas only, no shards, writes isolated, HTTP not native TCP. You can build that. You operate the balancer, the health checks, and the night the write replica OOMs because a materialized view used more memory than the insert.

Conventional managed: topology is a console, the app layer is still yours

ClickHouse Cloud's SharedMergeTree stores durable data in object storage. Adding compute does not copy parts between local disks. Metadata still goes through Keeper, which they run. That removes the worst of the self-hosted add-replica wait, on their control plane, on their SKU.

You still do not get a production API out of the box. Connection pools, retries, and ingest batching remain yours. Basic cannot autoscale. Scale and Enterprise can, with policies you configure in advance, which is not the same as a human adding a replica during an incident.

Altinity.Cloud is closer to OSS operations with a support contract: you may add replicas and shards yourself, on their Kubernetes, with their on-call for the database. You still design tables and build the serving path.

Managed does not mean Too many parts goes away. It means you are not SSH-ing into Keeper at 2 a.m. You are still on the hook for the query that created the parts.

Tinybird Cluster Management

Tinybird dedicated infrastructure gives Enterprise customers an isolated ClickHouse cluster. Cluster Management is how you change that cluster without taking the platform pager.

The path here was incremental, which is the honest version:

1. Usage charts for dedicated infra (observability without control).
2. Resize requests in the UI (vertical, still a request).
3. Compute-compute separation for populates: ephemeral replicas for backfills, so a populate does not fight APIs for CPU.
4. Self-serve replicas in the UI: add/remove replicas, set weights, no ticket.
5. Organizations API (March 2026): the same operations in a script.

Eligible dedicated clusters only. Shared Tinybird workspaces do not expose replica topology. Tinybird operates those clusters without a customer-facing replica API. Self-Managed Tinybird means you operate replicas yourself.

Why weights exist

ClickHouse does not magically isolate ingest from queries. You isolate them with routing. Tinybird dedicated clusters expose three weights per replica, each in 0-65535, distributed with weighted round-robin:

A common production layout is one replica biased to writes, the others biased to reads. Every replica still holds the data. You are placing load, not splitting the dataset. Guardrails refuse a config that would leave the cluster with no reads or no writes.

When you add a replica, Tinybird provisions it in your cloud and region (AWS or GCP), replicates the dataset, and prepares it for production workloads. Any replica can serve any query or handle any ingest once weights say so. You do not edit remote_servers. You do not SYSTEM DROP REPLICA by hand.

This is not autoscaling. You (or your script) request the change. Tinybird does not infer capacity from CPU and add nodes on its own. It is not sharding on demand. It is not a fix for a bad partition key.

Vertical resize (changing SKU on existing replicas) is a separate dedicated-infra control. Horizontal replica changes and weight changes are the Cluster Management surface.

Scripting the change

Until 2026, dedicated customers did this in the UI (Settings → Plan & Billing → Manage Cluster). Mutating API calls require old_weights. GET /v0/organizations/<organization_id>/clusters-configuration first and send the returned weights back. Optimistic concurrency: if someone else changed the cluster between GET and PUT, the request fails instead of applying a stale plan.

Rebalance reads toward a second replica, keep writes on the first:

curl -X PUT \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  "https://api.tinybird.co/v0/organizations/<organization_id>/clusters/<cluster_id>/weights" \
  -d '{
    "old_weights": {
      "copyjob": {"<replica-1>": 1, "<replica-2>": 1},
      "writer": {"<replica-1>": 100, "<replica-2>": 0},
      "reader": {"<replica-1>": 100, "<replica-2>": 0}
    },
    "new_weights": {
      "copyjob": {"<replica-1>": 1, "<replica-2>": 1},
      "writer": {"<replica-1>": 100, "<replica-2>": 0},
      "reader": {"<replica-1>": 50, "<replica-2>": 50}
    }
  }'

Add a replica and give it 30% of writes and 20% of reads in the same request:

curl -X POST \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  "https://api.tinybird.co/v0/organizations/<organization_id>/clusters/<cluster_id>/replicas" \
  -d '{
    "replica_size": "4-16",
    "existing_replicas": {
      "old_weights": {
        "copyjob": {"<replica-1>": 1},
        "writer": {"<replica-1>": 100},
        "reader": {"<replica-1>": 100}
      },
      "new_weights": {
        "copyjob": {"<replica-1>": 1},
        "writer": {"<replica-1>": 70},
        "reader": {"<replica-1>": 80}
      }
    },
    "new_replica": {
      "copyjob": 1,
      "writer": 30,
      "reader": 20
    }
  }'

Drain a replica and remove it:

curl -X DELETE \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  "https://api.tinybird.co/v0/organizations/<organization_id>/clusters/<cluster_id>/replicas/<replica_name>" \
  -d '{
    "old_weights": {
      "copyjob": {"<replica-1>": 1, "<replica-2>": 1, "<replica-3>": 1},
      "writer": {"<replica-1>": 50, "<replica-2>": 30, "<replica-3>": 20},
      "reader": {"<replica-1>": 50, "<replica-2>": 30, "<replica-3>": 20}
    },
    "new_weights": {
      "copyjob": {"<replica-1>": 1, "<replica-2>": 1},
      "writer": {"<replica-1>": 60, "<replica-2>": 40},
      "reader": {"<replica-1>": 60, "<replica-2>": 40}
    }
  }'

Endpoints:

• PUT /v0/organizations/{organization_id}/clusters/{cluster_id}/weights
• POST /v0/organizations/{organization_id}/clusters/{cluster_id}/replicas
• DELETE /v0/organizations/{organization_id}/clusters/{cluster_id}/replicas/{replica_name}

Permissions match the UI. HTTP 200 on POST is not "queries are already landing there." Tinybird still has to provision and replicate. Watch cluster configuration and endpoint latency after the operation completes.

If an operation fails partway, GET clusters-configuration again and retry with fresh old_weights. Do not fire a second POST from a stale payload.

Adding a replica starts billing for that Dedicated ClickHouse cluster capacity. Removing one stops it. Do not assume overlap cost or duration from this post. Confirm against the product and your contract before you automate.

The 2:07 a.m. runbook, three ways

Production needs more read capacity. Ingest is fine. Queries are not.

Self-managed. Page the person with AWS or GCP access and Keeper credentials. Run the add-replica sequence above. If the cluster is sharded and the hot shard is the problem, adding a replica of that shard helps. Adding a new shard does not move yesterday's partitions. Budget longer than the incident if this is the first time in six months.

Conventional managed. Open the vendor console or file a ticket. ClickHouse Cloud may add compute against object storage without a part copy. Your application still needs its own pool and retry story while you wait. Basic cannot do this.

Tinybird dedicated, Cluster Management enabled. Add a replica or raise read weights on the replicas that should take queries. Tinybird provisions, replicates, applies weights. Built-in guardrails refuse a config that would leave the cluster with no reads or no writes.

That third path is the product proof behind Tinybird dedicated. You decide. Tinybird operates.

Frequently Asked Questions (FAQs)

Can I automate replica changes from my own control plane?

Yes, if Cluster Management is enabled on your dedicated cluster. Use the Organizations API, always GET current weights first, and send them as old_weights. Wire it into the same place you keep other production runbooks. Shared plans have no replica API.

What happens if I set every write weight to 0?

The API and UI reject it. At least one replica must accept writes, and at least one must accept reads.

Do new replicas serve traffic immediately?

Tinybird provisions the replica and replicates the dataset before it is ready for production workloads. Do not assume HTTP 200 on POST equals queries already landing there. Watch cluster configuration and query latency after the operation completes. Self-hosted: never add the host to remote_servers until queue_size and absolute_delay are acceptable.

Is this available on Tinybird Self-Managed?

Cluster Management as described here is a Tinybird Cloud dedicated-infrastructure feature. Self-Managed means you operate the environment, including replica topology and Keeper, yourself.

Where do I start if I am not on dedicated infrastructure?

Tinybird Enterprise includes dedicated clusters. If you are evaluating isolation and replica control, talk to us about the production workload rather than trying to approximate this on shared infrastructure.

Does Cluster Management replace query optimization?

No. Weights move traffic. They do not fix a partition key that writes to thousands of parts, a materialized view that OOMs, or an unbounded SELECT *. Treat Cluster Management as topology. Treat system.query_log as homework.

Why not just use ClickHouse Cloud autoscaling?

If you want a hosted database that adds compute against object storage on a policy you configured last quarter, ClickHouse Cloud Scale/Enterprise is a coherent product. If you want to decide replica count and ingest-vs-query placement yourself, keep SQL-to-API and git deploys, and not staff Keeper, Tinybird dedicated plus Cluster Management is the other product. Autoscaling is a policy. Cluster Management is a decision you make, including at 2 a.m.