Keep clickhouse integration graylog queryable

Graylog owns stream routing, extractors, dashboards, and alert conditions on indexed logs. ClickHouse^® owns months of typed events, cross-service SQL, and product APIs at storage costs that do not track OpenSearch shard count. A working clickhouse integration graylog setup keeps Graylog as the operator console and ClickHouse as the analytical store instead of forcing one backend to do both jobs.

Graylog's GELF HTTP input listens on http://<host>:<port>/gelf (POST). Configure the port when you launch the input in System → Inputs.

Graylog does not ship a ClickHouse output. The bridge is always explicit: GELF or HTTP dual-write at the producer, a forwarder sidecar on the Graylog pipeline, or batch replay from archived index segments in object storage.

OpenSearch backend vs ClickHouse archive

Graylog stores searchable logs in OpenSearch (or legacy Elasticsearch). Index sets define rotation, retention, and shard counts. That model is excellent for keyword search and stream-based alerts inside Graylog's UI. It is expensive when every dashboard query scans hot shards or when retention beyond 30-90 days requires oversized clusters.

Write the field contract before you wire the bridge. Graylog dashboards and ClickHouse rollups must count the same events or on-call will not trust either number during an outage. For architecture patterns at scale, see real-time logs analytics architectures.

Three ingest paths: live, pipeline, and archive

Most production teams combine live dual-write for facts and archive replay for backfill, with pipeline sidecars only when you cannot change producers.

Apps ──► GELF/HTTP ──┬──► Graylog ──► OpenSearch (search, alerts)
                     └──► ClickHouse HTTP / Tinybird Events API

Graylog pipeline ──► forwarder sidecar ──► Kafka ──► ClickHouse Kafka engine

Index archive (S3) ──► batch loader ──► ClickHouse (cold / compliance)

GELF and HTTP: dual-write at the producer

GELF (Graylog Extended Log Format) wraps your payload with version, host, short_message, timestamp, and optional _-prefixed fields. Graylog extractors and pipelines enrich after intake. ClickHouse should receive the inner event as typed columns from day one.

async function emitLog(event) {
  const gelf = {
    version: "1.1",
    host: event.host,
    short_message: event.message,
    timestamp: Date.now() / 1000,
    level: event.level,
    _service: event.service,
    _env: event.env,
  };

  await Promise.allSettled([
    fetch("https://graylog.example.com:12201/gelf", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify(gelf),
    }),
    fetch("https://api.tinybird.co/v0/events?name=app_logs", {
      method: "POST",
      headers: { Authorization: "Bearer TB_INGEST_TOKEN", "Content-Type": "application/json" },
      body: JSON.stringify({
        event_time: new Date().toISOString(),
        host: event.host,
        message: event.message,
        level: event.level,
        service: event.service,
        env: event.env,
      }),
    }),
  ]);
}

Use Promise.allSettled or a local buffer so Graylog backpressure does not block request handlers. Batch when volume exceeds a few thousand events per second per source.

Streams, index sets, and the ClickHouse sort key

Graylog streams route messages to index sets based on rules (facility, source, regex on message). ClickHouse sort keys must lead with bounded dimensions that match how you aggregate, not how Graylog routes for search.

Typical MergeTree contract for Graylog-shaped logs:

CREATE TABLE graylog.logs
(
    event_time DateTime64(3),
    host LowCardinality(String),
    facility LowCardinality(String),
    level LowCardinality(String),
    stream LowCardinality(String),
    message String,
    attributes Map(String, String)
)
ENGINE = MergeTree
PARTITION BY toYYYYMM(event_time)
ORDER BY (stream, level, event_time, host);

Map Graylog stream names to stream at ingest. Keep high-cardinality IDs (request_id, user_id) in attributes, not the sort key. For Kafka-based ingest, use the Kafka table engine pattern from kafka to ClickHouse example.

Pipeline forwarder sidecar

When producers cannot change, insert a forwarder after Graylog stream processing:

Input ──► Graylog streams/extractors ──► OpenSearch index
                              │
                              └──► HTTP/Kafka forwarder ──► ClickHouse

The sidecar should emit the post-extractor shape so Graylog dashboards and ClickHouse rollups share field names. Log the stream ID and index set name on every row for parity checks.

Archive replay from index backups

Teams that export Graylog index snapshots to S3 for compliance can replay into ClickHouse with a scheduled loader. Treat archive replay as batch backfill, not live ingest. Loader lag of hours is normal.

Use S3Queue or a worker that reads NDJSON exports, maps extractors to columns, and inserts in 10k-row batches. Deduplicate on (stream, event_time, host, cityHash64(message)) if replays overlap.

Tinybird on the Graylog pipeline

Tinybird is managed ClickHouse with ingestion paths that fit Graylog dual-write:

1. Events API for app-level GELF mirrors (1K+ req/sec per token, NDJSON rows)
2. Kafka connector when the sidecar publishes to a topic instead of HTTP
3. Pipes to publish rollups as HTTP endpoints per build real-time APIs on ClickHouse
4. JWT fixed_params for tenant-scoped log explorers in product UIs
5. Branches to test new extractor mappings before production forwarder rollout

Example Pipe for error rate by stream:

NODE stream_error_rate
SQL >
    SELECT
        stream,
        countIf(level IN ('error', 'critical', 'alert')) AS errors,
        count() AS total,
        errors / total AS error_rate
    FROM graylog_logs
    WHERE event_time >= now() - INTERVAL 15 MINUTE
    GROUP BY stream
    ORDER BY error_rate DESC

TYPE endpoint

Canva reports 3.6 PB processed per month and 54 ms p99 query latency on Tinybird's product page. Resend reports 100 TB per month and 62 ms p90 query latency without relying on cache. Tinybird is SOC 2 Type II certified.

5 operational mistakes on Graylog ClickHouse integrations

1. Mirroring OpenSearch mapping into ClickHouse

OpenSearch dynamic mappings favor search flexibility. ClickHouse needs a stable DDL contract.

Fix: Define core columns up front. Park extras in Map or JSON. Fail on unknown fields in staging.

2. Sort key led by request_id or _id

Graylog encourages high-cardinality custom fields. ClickHouse sort keys must lead with bounded dimensions.

Fix: ORDER BY (stream, level, event_time, host). Keep trace IDs in attributes.

3. Sidecar emits pre-extractor raw GELF

Graylog dashboards use post-pipeline fields. ClickHouse rollups on raw GELF diverge.

Fix: Forward after streams and extractors. Include stream and facility on every row.

4. Archive replay without dedupe keys

Index re-exports overlap windows. Inserts duplicate rows and inflates error rates.

Fix: Dedupe on hash of (stream, event_time, host, message) or use ReplacingMergeTree with version column.

5. Two KPI definitions and no parity job

Graylog alerts fire on stream conditions. Product teams query ClickHouse rollups. Filters differ.

Fix: One definition document per KPI with Graylog stream rule and SQL form. Nightly parity query with alert on greater than 1% divergence.

What the integration comes down to

Graylog integration with ClickHouse is not rip-and-replace. Graylog stays the system of record for stream rules, extractors, and operator search inside OpenSearch retention. ClickHouse holds long retention, high-cardinality SQL, joins to product data, and customer-facing analytics APIs.

Use live dual-write for facts, pipeline sidecars when producers cannot change, and archive replay for compliance backfill. Write down which system owns each KPI before you wire the second destination.

Frequently Asked Questions (FAQs)

Does Graylog natively write to ClickHouse?

No. Graylog indexes into OpenSearch. ClickHouse ingest uses dual-write at producers, a pipeline forwarder, or batch loaders from archived indices.

Should ClickHouse store the full GELF envelope?

Store typed columns for fields you aggregate. Keep the envelope in raw_line or attributes only if compliance requires it.

Can I replace OpenSearch with ClickHouse for Graylog search?

Not as a drop-in backend. Graylog requires OpenSearch for its UI and stream engine. ClickHouse complements OpenSearch for SQL analytics and long retention.

What is the best path when apps already send GELF?

Dual-write GELF to Graylog and NDJSON to ClickHouse or Tinybird at the producer. Simplest contract, lowest pipeline lag.

How do index set rotations affect ClickHouse?

Rotations in Graylog do not automatically sync to ClickHouse. Plan TTL and partition drops in ClickHouse independently.

When should teams add Tinybird on top of ClickHouse?

When you want managed ingestion, SQL APIs, and schema workflows rather than operating ClickHouse clusters and auth layers yourself.