--- title: "Cluster Management: keep the controls, skip the cluster ops" excerpt: "Observe workload signals, resize Dedicated clusters, and rebalance traffic from the Organization UI or API while Tinybird operates ClickHouse." authors: "Aitana Azcona" categories: "Product updates" createdOn: "2026-09-03 15:00:00" publishedOn: "2026-09-03 15:00:00" updatedOn: "2026-09-03 15:00:00" status: "published" --- At 2 a.m., a saturated ClickHouse{% sup %}®{% /sup %} cluster usually leaves you operating infrastructure or waiting for someone who does. [Cluster Management](https://www.tinybird.co/product/cluster-management) gives Organizations on Dedicated infrastructure direct control over supported capacity and workload distribution while Tinybird operates the underlying platform. Earlier this year, we made replica management scriptable through the Organizations API. Cluster Management now covers the full operational loop: observe workload signals, decide what to change, resize through the Organization UI or API, and verify the result. ## You decide. Tinybird operates Cluster Management separates capacity decisions from ClickHouse operations. | Your team controls | Tinybird operates | | --- | --- | | When capacity changes | Replica provisioning and replication | | Replica count and size | Object storage, local caches, and recovery | | Read and write distribution | Infrastructure monitoring and operational response | | The signals and thresholds in your automation | ClickHouse and the surrounding platform | This is manual control, not automatic scaling by Tinybird. You can build automation around your own thresholds with the Organizations API, but your team still decides when to initiate a capacity change. ## Observe before changing Dedicated Organizations can query hardware metrics from `organization.metrics_logs`. New records arrive every minute, and queries against Service Data Sources are free of charge. ```tinybird-cli tb --cloud sql " SELECT cluster, host, round(avg(toFloat64(value)) * 100, 1) AS cpu_usage_pct FROM organization.metrics_logs WHERE timestamp > now() - INTERVAL 5 MINUTE AND metric = 'CPUUsage' GROUP BY cluster, host ORDER BY cluster, host " ``` For a two-replica cluster under sustained load, the result looks like this: ```text ------------------------------------------------- | cluster | host | cpu_usage_pct | ------------------------------------------------- | production | replica-1 | 96.0 | | production | replica-2 | 88.0 | ------------------------------------------------- ``` The same Service Data Source includes memory, executing queries, system load, and replication queue metrics. Use `organization.endpoint_errors` for API errors and the other [Organization Service Data Sources](https://www.tinybird.co/docs/forward/monitoring/service-datasources#organization-service-data-sources) for recent service activity. ## Manage the cluster from the UI Organization admins can go to **Organization settings → Plan & Billing → Manage cluster**. The interface shows every replica, its CPU and memory, and its current read and write weights. From one screen, you can: - Add a replica, choose its CPU and memory, set its initial weights, and review the estimated daily credit impact. - Change read and write weights across existing replicas. - Remove a replica after assigning its traffic to the remaining replicas. - Follow operation status while Tinybird provisions or removes capacity. The UI prevents configurations with no replica handling reads or no replica handling writes. It also blocks another cluster change while an operation is in progress. {% html %} {% /html %} ## How weights distribute work Weights use proportional, weighted round-robin distribution. A read weight of `2` receives twice the query traffic of a read weight of `1`. | Weight | Workload | Notes | | --- | --- | --- | | `reader` | Query traffic | A weight of `0` removes the replica from query traffic. | | `writer` | Events API and Kafka ingestion | Proportional distribution is not guaranteed for file imports or Copy Pipes. | | `copyjob` | Copy Pipe jobs | Keep the recommended value of `1` on every replica unless your setup requires otherwise. | Each weight can be from `0` to `65535`. At least one replica must have a non-zero read weight, and at least one must have a non-zero write weight. The UI presents read and write controls. API payloads also carry the `copyjob` map. ## Resize without downtime Changing replica size uses an add, reroute, verify, remove workflow: 1. Add replicas with the target CPU and memory configuration. 2. Wait for the new replicas to become available. 3. Rebalance read and write traffic onto them. 4. Verify query latency, ingestion throughput, and replica queues. 5. Remove the old replicas. The old replicas keep serving until you remove them, so the documented workflow keeps the cluster available. Object storage remains the durable source of truth while each new replica warms its local cache. You pay for both old and new capacity during the overlap. Adding or removing replicas usually takes a few minutes. The cluster keeps operating normally, but you cannot start another cluster operation until the current one finishes. ## Automate the same workflow with the API The Organizations API exposes current cluster state, available replica sizes, and three mutations: - `GET /v0/organizations/{organization_id}/clusters` returns replicas, weights, capabilities, and operation state. - `GET /v0/organizations/{organization_id}/clusters-configuration` returns available regions, replica sizes, and roles. - `PUT /v0/organizations/{organization_id}/clusters/{cluster_id}/weights` rebalances workloads. - `POST /v0/organizations/{organization_id}/clusters/{cluster_id}/replicas` adds a replica. - `DELETE /v0/organizations/{organization_id}/clusters/{cluster_id}/replicas/{replica_name}` removes a replica. Use a user token belonging to an Organization admin, then set the values used by the examples: ```bash export TB_HOST="https://api.tinybird.co" export TB_TOKEN="" export ORG_ID="" export CLUSTER_ID="" ``` ### Read current state first Every mutation requires `old_weights`. Read them from `GET /clusters` immediately before changing the cluster: ```bash curl -sS \ -H "Authorization: Bearer $TB_TOKEN" \ "$TB_HOST/v0/organizations/$ORG_ID/clusters" | \ jq --arg id "$CLUSTER_ID" ' .clusters[] | select(.cluster_id == $id) | {operation_in_progress, can_scale, can_rebalance, cluster_weights, replicas} ' ``` Example output: ```json { "operation_in_progress": false, "can_scale": true, "can_rebalance": true, "cluster_weights": { "reader": {"replica-1": 1, "replica-2": 1}, "writer": {"replica-1": 1, "replica-2": 1}, "copyjob": {"replica-1": 1, "replica-2": 1} }, "replicas": { "replica-1": { "cpu": 8, "memory_gb": 32, "is_being_created": false, "is_being_destroyed": false, "launch_time": "2026-08-20T09:00:00Z", "error": null }, "replica-2": { "cpu": 8, "memory_gb": 32, "is_being_created": false, "is_being_destroyed": false, "launch_time": "2026-08-20T09:05:00Z", "error": null } } } ``` Use `GET /clusters-configuration` to choose a `replica_size` supported by the cluster's cloud provider: ```bash curl -sS \ -H "Authorization: Bearer $TB_TOKEN" \ "$TB_HOST/v0/organizations/$ORG_ID/clusters-configuration" | \ jq '.available_sizes' ``` ### Add a replica This request adds an 8 vCPU, 32 GiB replica. Equal weights distribute each supported workload evenly across the three replicas after provisioning completes. ```bash curl -sS -X POST \ -H "Authorization: Bearer $TB_TOKEN" \ -H "Content-Type: application/json" \ "$TB_HOST/v0/organizations/$ORG_ID/clusters/$CLUSTER_ID/replicas" \ -d '{ "replica_size": "8-32", "existing_replicas": { "old_weights": { "reader": {"replica-1": 1, "replica-2": 1}, "writer": {"replica-1": 1, "replica-2": 1}, "copyjob": {"replica-1": 1, "replica-2": 1} }, "new_weights": { "reader": {"replica-1": 1, "replica-2": 1}, "writer": {"replica-1": 1, "replica-2": 1}, "copyjob": {"replica-1": 1, "replica-2": 1} } }, "new_replica": { "reader": 1, "writer": 1, "copyjob": 1 } }' | jq ``` ```json { "cluster_id": "", "new_replica": { "name": "replica-3", "cpu": 8, "memory_gb": 32, "state": "adding" }, "operation": "replica_add_requested" } ``` The response confirms that the operation was requested. Poll `GET /clusters` and wait for `operation_in_progress` to become `false` before making another change: ```bash curl -sS \ -H "Authorization: Bearer $TB_TOKEN" \ "$TB_HOST/v0/organizations/$ORG_ID/clusters" | \ jq --arg id "$CLUSTER_ID" ' .clusters[] | select(.cluster_id == $id) | { operation_in_progress, replicas: (.replicas | with_entries( .value = if .value.is_being_created then "adding" elif .value.is_being_destroyed then "removing" elif .value.error then "error" else "running" end )) } ' ``` ```json { "operation_in_progress": false, "replicas": { "replica-1": "running", "replica-2": "running", "replica-3": "running" } } ``` ### Rebalance replica weights This example keeps ingestion on the first two replicas while sending half of query traffic to the new replica. ```bash curl -sS -X PUT \ -H "Authorization: Bearer $TB_TOKEN" \ -H "Content-Type: application/json" \ "$TB_HOST/v0/organizations/$ORG_ID/clusters/$CLUSTER_ID/weights" \ -d '{ "old_weights": { "reader": {"replica-1": 1, "replica-2": 1, "replica-3": 1}, "writer": {"replica-1": 1, "replica-2": 1, "replica-3": 1}, "copyjob": {"replica-1": 1, "replica-2": 1, "replica-3": 1} }, "new_weights": { "reader": {"replica-1": 1, "replica-2": 1, "replica-3": 2}, "writer": {"replica-1": 1, "replica-2": 1, "replica-3": 0}, "copyjob": {"replica-1": 1, "replica-2": 1, "replica-3": 1} } }' | jq ``` ```json { "cluster_id": "", "updated_weights": { "reader": {"replica-1": 1, "replica-2": 1, "replica-3": 2}, "writer": {"replica-1": 1, "replica-2": 1, "replica-3": 0}, "copyjob": {"replica-1": 1, "replica-2": 1, "replica-3": 1} }, "operation": "weight_update_requested" } ``` ### Remove a replica After verifying the remaining capacity, this request removes `replica-2` and excludes it from every new weight map. ```bash curl -sS -X DELETE \ -H "Authorization: Bearer $TB_TOKEN" \ -H "Content-Type: application/json" \ "$TB_HOST/v0/organizations/$ORG_ID/clusters/$CLUSTER_ID/replicas/replica-2" \ -d '{ "old_weights": { "reader": {"replica-1": 1, "replica-2": 1, "replica-3": 2}, "writer": {"replica-1": 1, "replica-2": 1, "replica-3": 0}, "copyjob": {"replica-1": 1, "replica-2": 1, "replica-3": 1} }, "new_weights": { "reader": {"replica-1": 1, "replica-3": 2}, "writer": {"replica-1": 1, "replica-3": 0}, "copyjob": {"replica-1": 1, "replica-3": 1} } }' | jq ``` ```json { "cluster_id": "", "removed_replica": "replica-2", "updated_weights": { "reader": {"replica-1": 1, "replica-3": 2}, "writer": {"replica-1": 1, "replica-3": 0}, "copyjob": {"replica-1": 1, "replica-3": 1} }, "operation": "replica_delete_requested" } ``` Always fetch fresh `cluster_weights` before a mutation and send them as `old_weights`. If another change landed after your GET, the stale request fails instead of overwriting the newer configuration. ## What happens at 2 a.m.? We turned that question into [2amclickhouse](https://www.tinybird.co/2amclickhouse), a pixel-art choose-your-own-adventure. Pick a door and see where your decisions take you. Cluster Management is available for Dedicated infrastructure where self-service replica management is enabled. Shared plans do not expose replica topology. On self-managed infrastructure, your team operates the environment and its replicas. Read the [Cluster Management guide](https://www.tinybird.co/docs/forward/guides/cluster-management) for the complete workflow and the [Organizations API reference](https://www.tinybird.co/docs/api-reference/organizations-api) for endpoint details. {% cta title="Keep the controls. Skip the cluster ops." text="See how Cluster Management gives you direct control over Dedicated capacity while Tinybird operates the platform." button={href: "https://www.tinybird.co/product/cluster-management", target: "_blank", text: "Explore Cluster Management"} /%}