Monitor your queries validation status¶

When you use Tinybird, we manage ClickHouse upgrades for you. We select stable versions based on our internal testing and only upgrade when we're confident it won't impact your queries. As part of this process, we test all queries you've run in the past 7 days against the upcoming ClickHouse version.

In Tinybird, we work mainly with three versions at the same time:

• Edge. This version has the latest features, and it's the version we use as a testbench.
• Stable. Most of our clusters are on this version; it's the one we keep as the main reference. We don't promote a version from Edge to Stable until we're sure it has no regressions and we've tested it on clusters with varying workloads.
• Deprecated. When we promote a version to Stable, the previous ones become deprecated. We aim to have the fewest clusters possible on deprecated versions; these are typically for very specific use cases.

This means that, to ensure all clusters are running fully supported versions, we aim to upgrade them regularly. Some of the queries that you run might break in newer versions. For this reason, we run the validator, a tool that tests your recent queries against the next version (typically Edge) to ensure the upgrade won't break them.

When we find queries that would break, we add them to the Service Data Source query_validator_log. In this Data Source, we provide you with information about the query, such as:

• The originating Workspace and Pipe.
• The full query text.
• The error message produced in the new ClickHouse version.
• The new ClickHouse version used for testing.

You can use this information to fix the issues and ensure your queries won't break when we upgrade your cluster.

Monitoring for broken queries¶

You can query the query_validator_log like this:

SELECT
    stable_version,
    version,
    query_last_execution,
    workspace,
    pipe_name,
    query,
    error_code,
    error,
    fix_suggestion
FROM
    (
        SELECT *
        FROM query_validator_log
        WHERE error != '' AND query_id LIKE 'MV_%'
        ORDER BY run_validation DESC
        LIMIT 1 BY query_id
        UNION ALL
        SELECT *
        FROM query_validator_log
        WHERE error != '' AND query_id NOT LIKE 'MV_%'
        ORDER BY run_validation DESC
        LIMIT 1 BY workspace, pipe_name
    )

This will yield the queries that need fixing, along with a possible fix suggestion in fix_suggestion. Queries extracted from the definition of Materialized Views will not have meaningful information in the workspace and pipe_name fields, so you need to recognize the Materialized View from the query field (this will be improved soon).

The queries returned in this Data Source correspond to the full queries that we send to ClickHouse, which are composed from all the nodes you create in your Pipes.

Fixing queries¶

Some queries might be easy to fix with the provided fix_suggestion, or with the error field. You can check that your fix worked by looking up your Pipe in the Data Source the next day, and checking that the error_code and error fields are empty. We plan to provide better ways to test queries soon. You can also contact support if you need help fixing your queries.