Keep clickhouse integration tempo queryable

Grafana Tempo stores traces in object storage and serves TraceQL for investigation. ClickHouse^® stores span facts in columnar tables and serves analytical SQL over months of traffic. A working clickhouse integration tempo setup keeps Tempo as the trace backend for Grafana dashboards while ClickHouse answers product analytics, SLO rollups, and cross-signal joins that TraceQL was not built to optimize.

Tempo does not export spans to ClickHouse natively. The bridge is always an OpenTelemetry Collector (or Grafana Alloy) fan-out: OTLP in, multiple exporters out.

Tempo blocks vs ClickHouse span tables

Tempo ingests OTLP traces, batches them into Parquet blocks, and writes to S3, GCS, or Azure Blob. In microservices deployments, distributors acknowledge after Kafka; block-builders flush blocks to object storage and backend workers compact and enforce retention. In monolithic mode, the live-store cuts blocks directly. TraceQL queries scan block metadata and object storage. That architecture is cost-efficient for trace ID lookup and service graphs inside Grafana.

ClickHouse stores denormalized span rows with sort keys aligned to aggregation queries: p95 latency by route, error rate by dependency, tail latency by tenant. The two stores answer different questions.

For the broader OTLP collector design, see clickhouse integration opentelemetry. For time-series rollups on metrics derived from traces, see ClickHouse for time series metrics.

Collector fan-out: one pipeline, two backends

Apps (OTLP) ──► Collector / Alloy
                  ├── processors (memory_limiter, attributes, tail_sampling, batch)
                  ├── exporter: otlp/tempo  ──► Tempo ──► object storage
                  └── exporter: clickhouse ──► ClickHouse span tables + rollups

Processor order that ages well:

1. memory_limiter (fail closed under burst)
2. resource / attributes (normalize service.name, redact secrets)
3. tail_sampling (keep errors and slow traces; drop health-check noise)
4. batch (multi-thousand row inserts)

Pin Collector and exporter versions. Config knobs move between releases. Treat collector config like application code: PR review, staging soak, then prod.

Tail sampling before dual-write

Dual-writing unsampled production traffic doubles storage cost in both Tempo and ClickHouse. Define sampling policy in the collector, not independently in each backend.

Document the effective sample rate. Product dashboards that show "requests per minute" from ClickHouse must apply the inverse sample rate or use unsampled counter spans.

Span schema for Tempo-shaped OTLP

CREATE TABLE tempo.spans
(
    timestamp DateTime64(9),
    trace_id String,
    span_id String,
    parent_span_id String,
    service LowCardinality(String),
    operation LowCardinality(String),
    duration_ns UInt64,
    status_code LowCardinality(String),
    http_route LowCardinality(String),
    deployment_environment LowCardinality(String),
    attributes Map(String, String)
)
ENGINE = MergeTree
PARTITION BY toDate(timestamp)
ORDER BY (service, operation, timestamp);

Keep http_route and operation as LowCardinality. Put high-cardinality IDs (user_id, session_id) in attributes. Add a 1-minute rollup materialized view for dashboards:

CREATE TABLE tempo.spans_service_1m
(
    minute DateTime,
    service LowCardinality(String),
    operation LowCardinality(String),
    count_state AggregateFunction(count),
    p95_state AggregateFunction(quantile(0.95), UInt64)
)
ENGINE = AggregatingMergeTree
ORDER BY (service, operation, minute);

Query rollups with -Merge combinators instead of scanning raw spans for every chart.

Tempo metrics generator vs ClickHouse rollups

Tempo can generate metrics from traces (service graph, span metrics) and remote-write to Prometheus or Mimir. That path covers Grafana dashboards. ClickHouse rollups cover SQL joins, tenant-scoped APIs, and ad-hoc exploration without standing up another metrics tier.

Pick one primary rollup owner per KPI. If Grafana reads span metrics from Mimir and product reads from ClickHouse, define the same aggregation window and label set or numbers will diverge.

Block replay and backfill

Tempo block exports in object storage can backfill ClickHouse when you add the analytical layer after Tempo is already in production. Treat block replay as batch ingest with dedupe on (trace_id, span_id).

Use a worker that reads Tempo block format or re-export via OTLP, maps spans to the DDL above, and inserts in batches. Expect hours of lag for PB-scale backfill.

Tinybird on the Tempo pipeline

Tinybird fits the ClickHouse side of a Tempo dual-write:

1. Events API or Kafka connector as the collector export target
2. Pipes for p95-by-route endpoints consumed by product UIs per build real-time APIs on ClickHouse
3. JWT fixed_params to scope queries by deployment_environment or tenant
4. Branches to test new attribute mappings before collector rollout
5. Service Data Sources for ingest lag and endpoint latency monitoring

Example endpoint for route latency:

NODE route_p95
SQL >
    SELECT
        http_route,
        quantile(0.95)(duration_ns) / 1000000 AS p95_ms,
        count() AS spans
    FROM tempo_spans
    WHERE timestamp >= now() - INTERVAL 1 HOUR
      AND service = {{String(service)}}
    GROUP BY http_route
    ORDER BY p95_ms DESC
    LIMIT 50

TYPE endpoint

Canva reports 3.6 PB processed per month and 54 ms p99 query latency on Tinybird's product page. Resend reports 100 TB per month and 62 ms p90 query latency without relying on cache. Tinybird is SOC 2 Type II certified.

5 operational mistakes on Tempo ClickHouse integrations

1. Dual-write without tail sampling

Full-fidelity spans in Tempo blocks and ClickHouse explode storage cost.

Fix: Centralize tail sampling in the collector. Document sample rates in rollup definitions.

2. Sort key led by trace_id

Trace ID lookup is valid but rare as a primary aggregation pattern.

Fix: ORDER BY (service, operation, timestamp). Index trace_id separately if needed.

3. Different attribute normalization in each exporter

Tempo and ClickHouse receive different service.name values when processors differ.

Fix: One attributes processor chain before the exporter fork.

4. Raw span scans for dashboard queries

Every chart hits the full span table and melts query nodes.

Fix: 1-minute AggregatingMergeTree rollups. Query -Merge states for dashboards.

5. TraceQL numbers compared to ClickHouse without parity

Grafana panels and product APIs show different p95 during incidents.

Fix: Nightly parity job on one route with identical time window and filters.

What the integration comes down to

Tempo integration with ClickHouse is a collector design problem, not a Tempo plugin. Tempo stays the Grafana-native trace store with block compression and TraceQL. ClickHouse holds span facts and rollups for SQL, joins, and product APIs at longer retention.

Fan-out OTLP from one collector pipeline, sample before export, and align rollup definitions with any Tempo-generated metrics. Write down which backend owns each SLO before you wire the second exporter.

Frequently Asked Questions (FAQs)

Does Grafana Tempo write to ClickHouse directly?

No. Use an OpenTelemetry Collector or Grafana Alloy exporter to ClickHouse alongside the Tempo OTLP exporter.

Should I store every span in ClickHouse?

No. Apply tail sampling in the collector. Keep errors and slow traces at full fidelity. Sample success paths.

Can TraceQL replace ClickHouse for analytics?

TraceQL fits trace investigation in Grafana. ClickHouse fits multi-month SQL aggregations, joins, and HTTP APIs.

What exporter should I use for ClickHouse?

Community ClickHouse exporters, HTTP INSERT with JSONEachRow, or Tinybird Events API depending on ops appetite.

How do I backfill ClickHouse from existing Tempo blocks?

Batch replay from object storage with dedupe on trace_id and span_id. Expect hours of loader lag at scale.

When should teams add Tinybird on top of ClickHouse?

When you want managed ingestion, SQL endpoints, and branch workflows instead of operating ClickHouse for product-facing trace analytics.