Cluster Management: keep the controls, skip the cluster ops

At 2 a.m., a saturated ClickHouse^® cluster usually leaves you operating infrastructure or waiting for someone who does. Cluster Management gives Organizations on Dedicated infrastructure direct control over supported capacity and workload distribution while Tinybird operates the underlying platform.

Earlier this year, we made replica management scriptable through the Organizations API. Cluster Management now covers the full operational loop: observe workload signals, decide what to change, resize through the Organization UI or API, and verify the result.

You decide. Tinybird operates

Cluster Management separates capacity decisions from ClickHouse operations.

This is manual control, not automatic scaling by Tinybird. You can build automation around your own thresholds with the Organizations API, but your team still decides when to initiate a capacity change.

Observe before changing

Dedicated Organizations can query hardware metrics from organization.metrics_logs. New records arrive every minute, and queries against Service Data Sources are free of charge.

tb --cloud sql "
SELECT
  cluster,
  host,
  round(avg(toFloat64(value)) * 100, 1) AS cpu_usage_pct
FROM organization.metrics_logs
WHERE timestamp > now() - INTERVAL 5 MINUTE
  AND metric = 'CPUUsage'
GROUP BY cluster, host
ORDER BY cluster, host
"

For a two-replica cluster under sustained load, the result looks like this:

-------------------------------------------------
| cluster    | host      | cpu_usage_pct        |
-------------------------------------------------
| production | replica-1 |                 96.0 |
| production | replica-2 |                 88.0 |
-------------------------------------------------

The same Service Data Source includes memory, executing queries, system load, and replication queue metrics. Use organization.endpoint_errors for API errors and the other Organization Service Data Sources for recent service activity.

Manage the cluster from the UI

Organization admins can go to Organization settings → Plan & Billing → Manage cluster. The interface shows every replica, its CPU and memory, and its current read and write weights.

From one screen, you can:

• Add a replica, choose its CPU and memory, set its initial weights, and review the estimated daily credit impact.
• Change read and write weights across existing replicas.
• Remove a replica after assigning its traffic to the remaining replicas.
• Follow operation status while Tinybird provisions or removes capacity.

The UI prevents configurations with no replica handling reads or no replica handling writes. It also blocks another cluster change while an operation is in progress.

How weights distribute work

Weights use proportional, weighted round-robin distribution. A read weight of 2 receives twice the query traffic of a read weight of 1.

Each weight can be from 0 to 65535. At least one replica must have a non-zero read weight, and at least one must have a non-zero write weight. The UI presents read and write controls. API payloads also carry the copyjob map.

Resize without downtime

Changing replica size uses an add, reroute, verify, remove workflow:

1. Add replicas with the target CPU and memory configuration.
2. Wait for the new replicas to become available.
3. Rebalance read and write traffic onto them.
4. Verify query latency, ingestion throughput, and replica queues.
5. Remove the old replicas.

The old replicas keep serving until you remove them, so the documented workflow keeps the cluster available. Object storage remains the durable source of truth while each new replica warms its local cache. You pay for both old and new capacity during the overlap.

Adding or removing replicas usually takes a few minutes. The cluster keeps operating normally, but you cannot start another cluster operation until the current one finishes.

Automate the same workflow with the API

The Organizations API exposes current cluster state, available replica sizes, and three mutations:

• GET /v0/organizations/{organization_id}/clusters returns replicas, weights, capabilities, and operation state.
• GET /v0/organizations/{organization_id}/clusters-configuration returns available regions, replica sizes, and roles.
• PUT /v0/organizations/{organization_id}/clusters/{cluster_id}/weights rebalances workloads.
• POST /v0/organizations/{organization_id}/clusters/{cluster_id}/replicas adds a replica.
• DELETE /v0/organizations/{organization_id}/clusters/{cluster_id}/replicas/{replica_name} removes a replica.

Use a user token belonging to an Organization admin, then set the values used by the examples:

export TB_HOST="https://api.tinybird.co"
export TB_TOKEN="<user_token>"
export ORG_ID="<organization_id>"
export CLUSTER_ID="<cluster_id>"

Read current state first

Every mutation requires old_weights. Read them from GET /clusters immediately before changing the cluster:

curl -sS \
  -H "Authorization: Bearer $TB_TOKEN" \
  "$TB_HOST/v0/organizations/$ORG_ID/clusters" | \
  jq --arg id "$CLUSTER_ID" '
    .clusters[] | select(.cluster_id == $id) |
    {operation_in_progress, can_scale, can_rebalance, cluster_weights, replicas}
  '

Example output:

{
  "operation_in_progress": false,
  "can_scale": true,
  "can_rebalance": true,
  "cluster_weights": {
    "reader": {"replica-1": 1, "replica-2": 1},
    "writer": {"replica-1": 1, "replica-2": 1},
    "copyjob": {"replica-1": 1, "replica-2": 1}
  },
  "replicas": {
    "replica-1": {
      "cpu": 8,
      "memory_gb": 32,
      "is_being_created": false,
      "is_being_destroyed": false,
      "launch_time": "2026-08-20T09:00:00Z",
      "error": null
    },
    "replica-2": {
      "cpu": 8,
      "memory_gb": 32,
      "is_being_created": false,
      "is_being_destroyed": false,
      "launch_time": "2026-08-20T09:05:00Z",
      "error": null
    }
  }
}

Use GET /clusters-configuration to choose a replica_size supported by the cluster's cloud provider:

curl -sS \
  -H "Authorization: Bearer $TB_TOKEN" \
  "$TB_HOST/v0/organizations/$ORG_ID/clusters-configuration" | \
  jq '.available_sizes'

Add a replica

This request adds an 8 vCPU, 32 GiB replica. Equal weights distribute each supported workload evenly across the three replicas after provisioning completes.

curl -sS -X POST \
  -H "Authorization: Bearer $TB_TOKEN" \
  -H "Content-Type: application/json" \
  "$TB_HOST/v0/organizations/$ORG_ID/clusters/$CLUSTER_ID/replicas" \
  -d '{
    "replica_size": "8-32",
    "existing_replicas": {
      "old_weights": {
        "reader": {"replica-1": 1, "replica-2": 1},
        "writer": {"replica-1": 1, "replica-2": 1},
        "copyjob": {"replica-1": 1, "replica-2": 1}
      },
      "new_weights": {
        "reader": {"replica-1": 1, "replica-2": 1},
        "writer": {"replica-1": 1, "replica-2": 1},
        "copyjob": {"replica-1": 1, "replica-2": 1}
      }
    },
    "new_replica": {
      "reader": 1,
      "writer": 1,
      "copyjob": 1
    }
  }' | jq

{
  "cluster_id": "<cluster_id>",
  "new_replica": {
    "name": "replica-3",
    "cpu": 8,
    "memory_gb": 32,
    "state": "adding"
  },
  "operation": "replica_add_requested"
}

The response confirms that the operation was requested. Poll GET /clusters and wait for operation_in_progress to become false before making another change:

curl -sS \
  -H "Authorization: Bearer $TB_TOKEN" \
  "$TB_HOST/v0/organizations/$ORG_ID/clusters" | \
  jq --arg id "$CLUSTER_ID" '
    .clusters[] | select(.cluster_id == $id) |
    {
      operation_in_progress,
      replicas: (.replicas | with_entries(
        .value =
          if .value.is_being_created then "adding"
          elif .value.is_being_destroyed then "removing"
          elif .value.error then "error"
          else "running"
          end
      ))
    }
  '

{
  "operation_in_progress": false,
  "replicas": {
    "replica-1": "running",
    "replica-2": "running",
    "replica-3": "running"
  }
}

Rebalance replica weights

This example keeps ingestion on the first two replicas while sending half of query traffic to the new replica.

curl -sS -X PUT \
  -H "Authorization: Bearer $TB_TOKEN" \
  -H "Content-Type: application/json" \
  "$TB_HOST/v0/organizations/$ORG_ID/clusters/$CLUSTER_ID/weights" \
  -d '{
    "old_weights": {
      "reader": {"replica-1": 1, "replica-2": 1, "replica-3": 1},
      "writer": {"replica-1": 1, "replica-2": 1, "replica-3": 1},
      "copyjob": {"replica-1": 1, "replica-2": 1, "replica-3": 1}
    },
    "new_weights": {
      "reader": {"replica-1": 1, "replica-2": 1, "replica-3": 2},
      "writer": {"replica-1": 1, "replica-2": 1, "replica-3": 0},
      "copyjob": {"replica-1": 1, "replica-2": 1, "replica-3": 1}
    }
  }' | jq

{
  "cluster_id": "<cluster_id>",
  "updated_weights": {
    "reader": {"replica-1": 1, "replica-2": 1, "replica-3": 2},
    "writer": {"replica-1": 1, "replica-2": 1, "replica-3": 0},
    "copyjob": {"replica-1": 1, "replica-2": 1, "replica-3": 1}
  },
  "operation": "weight_update_requested"
}

Remove a replica

After verifying the remaining capacity, this request removes replica-2 and excludes it from every new weight map.

curl -sS -X DELETE \
  -H "Authorization: Bearer $TB_TOKEN" \
  -H "Content-Type: application/json" \
  "$TB_HOST/v0/organizations/$ORG_ID/clusters/$CLUSTER_ID/replicas/replica-2" \
  -d '{
    "old_weights": {
      "reader": {"replica-1": 1, "replica-2": 1, "replica-3": 2},
      "writer": {"replica-1": 1, "replica-2": 1, "replica-3": 0},
      "copyjob": {"replica-1": 1, "replica-2": 1, "replica-3": 1}
    },
    "new_weights": {
      "reader": {"replica-1": 1, "replica-3": 2},
      "writer": {"replica-1": 1, "replica-3": 0},
      "copyjob": {"replica-1": 1, "replica-3": 1}
    }
  }' | jq

{
  "cluster_id": "<cluster_id>",
  "removed_replica": "replica-2",
  "updated_weights": {
    "reader": {"replica-1": 1, "replica-3": 2},
    "writer": {"replica-1": 1, "replica-3": 0},
    "copyjob": {"replica-1": 1, "replica-3": 1}
  },
  "operation": "replica_delete_requested"
}

Always fetch fresh cluster_weights before a mutation and send them as old_weights. If another change landed after your GET, the stale request fails instead of overwriting the newer configuration.

What happens at 2 a.m.?

We turned that question into 2amclickhouse, a pixel-art choose-your-own-adventure. Pick a door and see where your decisions take you.

Cluster Management is available for Dedicated infrastructure where self-service replica management is enabled. Shared plans do not expose replica topology. On self-managed infrastructure, your team operates the environment and its replicas.

Read the Cluster Management guide for the complete workflow and the Organizations API reference for endpoint details.