QUERY_WAS_CANCELLED_BY_CLIENT ClickHouse error¶

This error occurs when a client application cancels a running query. It's common when clients implement timeout mechanisms, user cancellation, or connection management.

The QUERY_WAS_CANCELLED_BY_CLIENT error in ClickHouse (and Tinybird) happens when a client application explicitly cancels a running query. This is different from system-initiated cancellations and typically occurs when clients implement timeout mechanisms, user cancellation requests, connection management, or when the client application terminates unexpectedly.

What causes this error¶

You'll typically see it when:

Client application implements query timeout
User manually cancels a query through the client
Client connection is closed unexpectedly
Application implements automatic cancellation for long-running queries
Client-side resource constraints force cancellation
Network issues cause client disconnection
Application shutdown while queries are running
Client implements retry logic that cancels previous attempts

This error indicates client-side cancellation rather than server-side issues. Check your application's query management logic.

Example errors¶

Fails: client timeout implementation

SELECT COUNT(DISTINCT user_id) FROM events
WHERE timestamp >= '2020-01-01'
-- Error: Query was cancelled by client (timeout)

Fails: user cancellation

-- User cancels query in client application
SELECT * FROM large_table ORDER BY timestamp
-- Error: Query was cancelled by client

Fails: connection management

-- Client closes connection while query is running
SELECT
    user_id,
    COUNT(*) as event_count
FROM events
GROUP BY user_id
-- Error: Query was cancelled by client

Fails: application shutdown

-- Application shuts down during query execution
SELECT * FROM events WHERE timestamp > '2024-01-01'
-- Error: Query was cancelled by client

How to fix it¶

Check client timeout settings¶

Verify client-side timeout configurations:

Check client settings

-- In your client application, check timeout settings
-- Example for Python clickhouse-driver:
-- client = Client(host='host', port=9000, settings={'max_execution_time': 300})

Review connection management¶

Ensure proper connection handling:

Connection management

-- Implement proper connection handling in your application
-- Example pseudo-code:
--
-- try:
--     with get_connection() as conn:
--         result = conn.execute(query)
--         return result
-- except ConnectionError:
--     # Handle connection issues
--     pass

Implement proper error handling¶

Add cancellation handling to your application:

Error handling

-- In your application, handle cancellation gracefully
-- Example pseudo-code:
--
-- try:
--     result = execute_query(query)
--     return result
-- except QueryWasCancelledByClient:
--     # Handle client cancellation
--     logger.warning("Query cancelled by client")
--     return None

Check client application logs¶

Review client-side logs for cancellation reasons:

Client logging

-- Enable detailed logging in your client application
-- Example for Python:
-- import logging
-- logging.basicConfig(level=logging.DEBUG)

Common patterns and solutions¶

Client timeout management¶

Implement proper timeout handling:

Timeout configuration

-- Set appropriate timeouts in your client
-- Example for Python clickhouse-driver:
from clickhouse_driver import Client

client = Client(
    host='your-host',
    port=9000,
    database='your_database',
    settings={
        'max_execution_time': 600,  -- 10 minutes
        'max_memory_usage': 10000000000,  -- 10GB
        'max_bytes_before_external_group_by': 1000000000  -- 1GB
    }
)

Connection pooling¶

Use connection pooling to manage connections:

Connection pooling

-- Implement connection pooling in your application
-- Example pseudo-code:
--
-- class ConnectionPool:
--     def __init__(self, max_connections=10):
--         self.max_connections = max_connections
--         self.connections = []
--
--     def get_connection(self):
--         if self.connections:
--             return self.connections.pop()
--         return create_new_connection()
--
--     def return_connection(self, conn):
--         if len(self.connections) < self.max_connections:
--             self.connections.append(conn)
--         else:
--             conn.close()

Query retry logic¶

Implement retry mechanisms for cancelled queries:

Retry logic

-- Add retry logic for client cancellations
-- Example pseudo-code:
--
-- max_retries = 3
-- base_delay = 1 second
--
-- for attempt in range(max_retries):
--     try:
--         result = execute_query(query)
--         return result
--     except QueryWasCancelledByClient:
--         if attempt < max_retries - 1:
--             delay = base_delay * (2 ** attempt)
--             time.sleep(delay)
--             continue
--         else:
--             raise

Graceful shutdown¶

Handle application shutdown properly:

Graceful shutdown

-- Implement graceful shutdown in your application
-- Example pseudo-code:
--
-- import signal
--
-- def shutdown_handler(signum, frame):
--     # Cancel all running queries
--     cancel_all_queries()
--     # Close connections gracefully
--     close_all_connections()
--     sys.exit(0)
--
-- signal.signal(signal.SIGINT, shutdown_handler)
-- signal.signal(signal.SIGTERM, shutdown_handler)

Tinybird-specific notes¶

In Tinybird, QUERY_WAS_CANCELLED_BY_CLIENT errors often occur when:

API clients implement timeout mechanisms
User cancels queries in the UI
External applications close connections
Network issues cause client disconnection
Rate limiting forces client cancellation

To debug in Tinybird:

Check client timeout settings
Review network connectivity
Monitor API usage patterns
Check for rate limiting issues

In Tinybird, ensure your API clients have appropriate timeout settings and retry logic.

Best practices¶

Client configuration¶

Set appropriate timeout values for different query types
Implement proper connection pooling
Use connection health checks
Monitor client-side performance metrics

Error handling¶

Implement graceful cancellation handling
Add retry logic with exponential backoff
Log cancellation reasons for debugging
Provide user feedback for cancelled operations

Connection management¶

Use connection pooling to reuse connections
Implement connection health monitoring
Handle connection failures gracefully
Implement proper cleanup on shutdown

Configuration options¶

Client timeout settings¶

Client timeout configuration

-- Set various timeout parameters in your client
-- Example for Python clickhouse-driver:
client = Client(
    host='host',
    port=9000,
    settings={
        'max_execution_time': 300,  -- 5 minutes
        'max_query_size': 1000000000,  -- 1GB
        'max_memory_usage': 8000000000,  -- 8GB
        'max_bytes_before_external_group_by': 1000000000  -- 1GB
    }
)

Connection settings¶

Connection configuration

-- Configure connection parameters
-- Example for Python clickhouse-driver:
client = Client(
    host='host',
    port=9000,
    database='database',
    user='user',
    password='password',
    settings={
        'connect_timeout': 10,  -- 10 seconds
        'send_receive_timeout': 300,  -- 5 minutes
        'sync_request_timeout': 300  -- 5 minutes
    }
)

Query settings¶

Query-specific settings

-- Use settings for specific queries
-- Example for Python:
result = client.execute(
    "SELECT * FROM large_table SETTINGS max_execution_time = 600",
    settings={'max_execution_time': 600}
)

Alternative solutions¶

Asynchronous query execution¶

Use async patterns to avoid blocking:

Async execution

-- Use async/await patterns in your application
-- Example pseudo-code:
--
-- import asyncio
--
-- async def execute_query_async(query):
--     loop = asyncio.get_event_loop()
--     return await loop.run_in_executor(None, execute_query, query)
--
-- async def main():
--     tasks = []
--     for query in queries:
--         task = asyncio.create_task(execute_query_async(query))
--         tasks.append(task)
--
--     results = await asyncio.gather(*tasks, return_exceptions=True)
--     return results

Query queuing¶

Implement query queuing for better control:

Query queue

-- Implement query queuing in your application
-- Example pseudo-code:
--
-- from queue import Queue
-- import threading
--
-- class QueryQueue:
--     def __init__(self):
--         self.queue = Queue()
--         self.worker_thread = threading.Thread(target=self._worker)
--         self.worker_thread.start()
--
--     def _worker(self):
--         while True:
--             query = self.queue.get()
--             if query is None:
--                 break
--             try:
--                 result = execute_query(query)
--                 query.set_result(result)
--             except Exception as e:
--                 query.set_exception(e)
--             finally:
--                 self.queue.task_done()

Health monitoring¶

Implement health checks for connections:

Health monitoring

-- Add health checks to your connection management
-- Example pseudo-code:
--
-- def check_connection_health(connection):
--     try:
--         # Simple health check query
--         connection.execute("SELECT 1")
--         return True
--     except Exception:
--         return False
--
-- def get_healthy_connection():
--     for conn in connection_pool:
--         if check_connection_health(conn):
--             return conn
--     return create_new_connection()

Monitoring and prevention¶

Client performance tracking¶

Performance monitoring

-- Track client-side performance metrics
-- Example pseudo-code:
--
-- import time
--
-- def execute_query_with_monitoring(query):
--     start_time = time.time()
--     try:
--         result = execute_query(query)
--         duration = time.time() - start_time
--         log_metric('query_success', duration)
--         return result
--     except QueryWasCancelledByClient:
--         duration = time.time() - start_time
--         log_metric('query_cancelled_by_client', duration)
--         raise

Cancellation analysis¶

Cancellation tracking

-- Track cancellation patterns in your application
-- Example pseudo-code:
--
-- class QueryTracker:
--     def __init__(self):
--         self.cancellations = []
--
--     def track_cancellation(self, query, reason, duration):
--         self.cancellations.append({
--             'query': query,
--             'reason': reason,
--             'duration': duration,
--             'timestamp': time.time()
--         })
--
--     def get_cancellation_stats(self):
--         return {
--             'total_cancellations': len(self.cancellations),
--             'avg_duration': sum(c['duration'] for c in self.cancellations) / len(self.cancellations),
--             'reasons': {c['reason']: sum(1 for x in self.cancellations if x['reason'] == c['reason']) for c in self.cancellations}
--         }