---
title: "Keep clickhouse integration graylog queryable"
excerpt: "Graylog streams, GELF, and OpenSearch retention vs ClickHouse® SQL. Sidecar dual-write, index-set replay, and rollups that stay aligned."
authors: "Tinybird"
categories: "AI Resources"
createdOn: "2026-08-27 00:00:00"
publishedOn: "2026-08-27 00:00:00"
updatedOn: "2026-08-27 00:00:00"
status: "published"
---

Graylog owns stream routing, extractors, dashboards, and alert conditions on indexed logs. ClickHouse{% sup %}®{% /sup %} owns months of typed events, cross-service SQL, and product APIs at storage costs that do not track OpenSearch shard count. A working **clickhouse integration graylog** setup keeps Graylog as the operator console and ClickHouse as the analytical store instead of forcing one backend to do both jobs.

Graylog's GELF HTTP input listens on `http://<host>:<port>/gelf` (POST). Configure the port when you launch the input in **System → Inputs**.

Graylog does not ship a ClickHouse output. The bridge is always explicit: GELF or HTTP dual-write at the producer, a forwarder sidecar on the Graylog pipeline, or batch replay from archived index segments in object storage.

## OpenSearch backend vs ClickHouse archive

Graylog stores searchable logs in OpenSearch (or legacy Elasticsearch). Index sets define rotation, retention, and shard counts. That model is excellent for keyword search and stream-based alerts inside Graylog's UI. It is expensive when every dashboard query scans hot shards or when retention beyond 30-90 days requires oversized clusters.

| Workload | Graylog + OpenSearch | ClickHouse |
| --- | --- | --- |
| Stream rules and extractors | Primary | Mirror field contract only |
| Keyword search on `_message` | Primary | Optional; not the default path |
| 90-day error rate by `facility` | Possible; shard-heavy | Cheap with rollups |
| Join logs to billing or tenant tables | Awkward | Native SQL |
| Customer-facing log API | Not the default | Pipes / HTTP endpoints |
| Compliance archive at PB scale | Index rotation + cold tiers | TTL + object storage |

Write the field contract before you wire the bridge. Graylog dashboards and ClickHouse rollups must count the same events or on-call will not trust either number during an outage. For architecture patterns at scale, see [real-time logs analytics architectures](https://www.tinybird.co/blog/real-time-logs-analytics-architectures).

## Three ingest paths: live, pipeline, and archive

| Path | Graylog feature | ClickHouse role | Latency |
| --- | --- | --- | --- |
| **Live dual-write** | GELF/HTTP input at producer | Real-time facts + rollups | Seconds |
| **Pipeline sidecar** | Output forwarder after streams | Normalized rows from Graylog pipeline | Seconds to minutes |
| **Archive replay** | S3/GCS index archives | Cold backfill and compliance | Hours |

Most production teams combine **live dual-write for facts** and **archive replay for backfill**, with pipeline sidecars only when you cannot change producers.

```text
Apps ──► GELF/HTTP ──┬──► Graylog ──► OpenSearch (search, alerts)
                     └──► ClickHouse HTTP / Tinybird Events API

Graylog pipeline ──► forwarder sidecar ──► Kafka ──► ClickHouse Kafka engine

Index archive (S3) ──► batch loader ──► ClickHouse (cold / compliance)
```

## GELF and HTTP: dual-write at the producer

GELF (Graylog Extended Log Format) wraps your payload with `version`, `host`, `short_message`, `timestamp`, and optional `_`-prefixed fields. Graylog extractors and pipelines enrich after intake. ClickHouse should receive the inner event as typed columns from day one.

| GELF field | Graylog use | ClickHouse handling |
| --- | --- | --- |
| `timestamp` | Sets message time | `event_time DateTime64(3)` |
| `host` | Source host | `host LowCardinality(String)` |
| `short_message` | Primary message | `message String` |
| `_level` / `level` | Severity | `level LowCardinality(String)` |
| `_facility` | Facility metadata | `facility LowCardinality(String)` |
| Custom `_` fields | Extractors | Typed columns or `Map(String, String)` |

```javascript
async function emitLog(event) {
  const gelf = {
    version: "1.1",
    host: event.host,
    short_message: event.message,
    timestamp: Date.now() / 1000,
    level: event.level,
    _service: event.service,
    _env: event.env,
  };

  await Promise.allSettled([
    fetch("https://graylog.example.com:12201/gelf", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify(gelf),
    }),
    fetch("https://api.tinybird.co/v0/events?name=app_logs", {
      method: "POST",
      headers: { Authorization: "Bearer TB_INGEST_TOKEN", "Content-Type": "application/json" },
      body: JSON.stringify({
        event_time: new Date().toISOString(),
        host: event.host,
        message: event.message,
        level: event.level,
        service: event.service,
        env: event.env,
      }),
    }),
  ]);
}
```

Use `Promise.allSettled` or a local buffer so Graylog backpressure does not block request handlers. Batch when volume exceeds a few thousand events per second per source.

## Streams, index sets, and the ClickHouse sort key

Graylog **streams** route messages to index sets based on rules (`facility`, `source`, regex on `message`). ClickHouse **sort keys** must lead with bounded dimensions that match how you aggregate, not how Graylog routes for search.

Typical MergeTree contract for Graylog-shaped logs:

```sql
CREATE TABLE graylog.logs
(
    event_time DateTime64(3),
    host LowCardinality(String),
    facility LowCardinality(String),
    level LowCardinality(String),
    stream LowCardinality(String),
    message String,
    attributes Map(String, String)
)
ENGINE = MergeTree
PARTITION BY toYYYYMM(event_time)
ORDER BY (stream, level, event_time, host);
```

Map Graylog stream names to `stream` at ingest. Keep high-cardinality IDs (`request_id`, `user_id`) in `attributes`, not the sort key. For Kafka-based ingest, use the [Kafka table engine](https://clickhouse.com/docs/en/engines/table-engines/integrations/kafka) pattern from [kafka to ClickHouse example](https://www.tinybird.co/blog/kafka-to-clickhouse-example).

## Pipeline forwarder sidecar

When producers cannot change, insert a forwarder after Graylog stream processing:

```text
Input ──► Graylog streams/extractors ──► OpenSearch index
                              │
                              └──► HTTP/Kafka forwarder ──► ClickHouse
```

The sidecar should emit the **post-extractor** shape so Graylog dashboards and ClickHouse rollups share field names. Log the stream ID and index set name on every row for parity checks.

## Archive replay from index backups

Teams that export Graylog index snapshots to S3 for compliance can replay into ClickHouse with a scheduled loader. Treat archive replay as **batch backfill**, not live ingest. Loader lag of hours is normal.

Use [S3Queue](https://clickhouse.com/docs/en/engines/table-engines/integrations/s3queue) or a worker that reads NDJSON exports, maps extractors to columns, and inserts in 10k-row batches. Deduplicate on `(stream, event_time, host, cityHash64(message))` if replays overlap.

## Tinybird on the Graylog pipeline

Tinybird is managed ClickHouse with ingestion paths that fit Graylog dual-write:

1. **Events API** for app-level GELF mirrors (1K+ req/sec per token, NDJSON rows)
2. **Kafka connector** when the sidecar publishes to a topic instead of HTTP
3. **Pipes** to publish rollups as HTTP endpoints per [build real-time APIs on ClickHouse](https://www.tinybird.co/blog/build-real-time-apis-clickhouse-tinybird)
4. **JWT fixed_params** for tenant-scoped log explorers in product UIs
5. **Branches** to test new extractor mappings before production forwarder rollout

Example Pipe for error rate by stream:

```sql
NODE stream_error_rate
SQL >
    SELECT
        stream,
        countIf(level IN ('error', 'critical', 'alert')) AS errors,
        count() AS total,
        errors / total AS error_rate
    FROM graylog_logs
    WHERE event_time >= now() - INTERVAL 15 MINUTE
    GROUP BY stream
    ORDER BY error_rate DESC

TYPE endpoint
```

Canva reports 3.6 PB processed per month and 54 ms p99 query latency on Tinybird's product page. Resend reports 100 TB per month and 62 ms p90 query latency without relying on cache. Tinybird is SOC 2 Type II certified.

## 5 operational mistakes on Graylog ClickHouse integrations

### 1. Mirroring OpenSearch mapping into ClickHouse

OpenSearch dynamic mappings favor search flexibility. ClickHouse needs a stable DDL contract.

**Fix:** Define core columns up front. Park extras in `Map` or `JSON`. Fail on unknown fields in staging.

### 2. Sort key led by `request_id` or `_id`

Graylog encourages high-cardinality custom fields. ClickHouse sort keys must lead with bounded dimensions.

**Fix:** `ORDER BY (stream, level, event_time, host)`. Keep trace IDs in `attributes`.

### 3. Sidecar emits pre-extractor raw GELF

Graylog dashboards use post-pipeline fields. ClickHouse rollups on raw GELF diverge.

**Fix:** Forward after streams and extractors. Include `stream` and `facility` on every row.

### 4. Archive replay without dedupe keys

Index re-exports overlap windows. Inserts duplicate rows and inflates error rates.

**Fix:** Dedupe on hash of `(stream, event_time, host, message)` or use ReplacingMergeTree with version column.

### 5. Two KPI definitions and no parity job

Graylog alerts fire on stream conditions. Product teams query ClickHouse rollups. Filters differ.

**Fix:** One definition document per KPI with Graylog stream rule and SQL form. Nightly parity query with alert on greater than 1% divergence.

## What the integration comes down to

Graylog integration with ClickHouse is not rip-and-replace. Graylog stays the system of record for stream rules, extractors, and operator search inside OpenSearch retention. ClickHouse holds long retention, high-cardinality SQL, joins to product data, and customer-facing analytics APIs.

Use live dual-write for facts, pipeline sidecars when producers cannot change, and archive replay for compliance backfill. Write down which system owns each KPI before you wire the second destination.

## Frequently Asked Questions (FAQs)

### Does Graylog natively write to ClickHouse?

No. Graylog indexes into OpenSearch. ClickHouse ingest uses dual-write at producers, a pipeline forwarder, or batch loaders from archived indices.

### Should ClickHouse store the full GELF envelope?

Store typed columns for fields you aggregate. Keep the envelope in `raw_line` or `attributes` only if compliance requires it.

### Can I replace OpenSearch with ClickHouse for Graylog search?

Not as a drop-in backend. Graylog requires OpenSearch for its UI and stream engine. ClickHouse complements OpenSearch for SQL analytics and long retention.

### What is the best path when apps already send GELF?

Dual-write GELF to Graylog and NDJSON to ClickHouse or Tinybird at the producer. Simplest contract, lowest pipeline lag.

### How do index set rotations affect ClickHouse?

Rotations in Graylog do not automatically sync to ClickHouse. Plan TTL and partition drops in ClickHouse independently.

### When should teams add Tinybird on top of ClickHouse?

When you want managed ingestion, SQL APIs, and schema workflows rather than operating ClickHouse clusters and auth layers yourself.

{% cta
  title="SQL APIs on your Graylog log pipeline"
  text="Tinybird is managed ClickHouse with the Events API, Kafka connector, and sub-second SQL endpoints. Keep Graylog for ops; ship analytics from the same events."
  button={href: "https://cloud.tinybird.co/signup", target: "_blank", text: "Try Tinybird free"}
/%}