Service Data Sources¶
In addition to the data sources you upload, Tinybird provides other “Service Data Sources” that allow you to inspect what’s going on in your account.
For example, you can get real time stats about API calls or a log of every operation over your Data Sources.
It’s the same as using system tables in any database but in this case contains information about the service usage.
Note: Service Data Sources can’t be used in Materialized View queries
These are the available Data Sources:
tinybird.snapshot_views¶
Contains stats about your snapshot views
Field |
Type |
Description |
---|---|---|
|
|
View creation date. |
|
|
Snaphot id. It’s the same id other users can see in the URL (https://ui.tinybird.co/snapshot/:snapshot_id) |
|
|
Visit count. |
tinybird.pipe_stats_rt¶
Contains information about all requests made to your API Endpoints in real time.
This Data Source has a TTL of 7 days. If you need to query data older than 7 days you must use the aggregated by day data available at tinybird.pipe_stats.
Field |
Type |
Description |
---|---|---|
|
|
API call start date and time. |
|
|
Pipe Id as returned in our Pipes API ( |
|
|
Pipe name as returned in our Pipes API ( |
|
|
API call duration in seconds. |
|
|
API call read data in bytes. |
|
|
API call rows read. |
|
|
Rows returned by the API call. |
|
|
URL ( |
|
|
|
|
|
API call identifier returned in |
|
|
API call token identifer used. |
|
|
API call token name used. |
|
|
API call returned status code. |
tinybird.pipe_stats¶
Aggregates the request stats in tinybird.pipe_stats_rt by day.
Field |
Type |
Description |
---|---|---|
|
|
Request date and time. |
|
|
Pipe Id as returned in our Pipes API. |
|
|
Name of the pipe. |
|
|
Request count. |
|
|
Number of requests with error. |
|
|
Average duration state in milliseconds (see Querying _state columns). |
|
|
0.9, 0.95 and 0.99 quantiles state. Time in milliseconds (see Querying _state columns). |
|
|
Total bytes read. |
|
|
Total rows read. |
tinybird.bi_stats_rt¶
Contains information about all requests to your BI Connector interface in real time.
This Data Source has a TTL of 7 days. If you need to query data older than 7 days you must use the aggregated by day data available at tinybird.bi_stats.
Field |
Type |
Description |
---|---|---|
|
|
Query start timestamp. |
|
|
Executed query. |
|
|
Normalized executed query. This is the pattern of the query, without literals. Useful to analyze usage patterns. |
|
|
Error code, if any. |
|
|
Error description, if any. Empty otherwise. |
|
|
Query duration in milliseconds. |
|
|
Read rows. |
|
|
Read bytes. |
|
|
Total rows returned. |
|
|
Total bytes returned. |
tinybird.bi_stats¶
Aggregates the stats in tinybird.bi_stats_rt by day.
Field |
Type |
Description |
---|---|---|
|
|
Stats date. |
|
|
Database identifier. |
|
|
Normalized executed query. This is the pattern of the query, without literals. Useful to analyze usage patterns. |
|
|
Requests count. |
|
|
Error count. |
|
|
Average duration state in milliseconds (see Querying _state columns). |
|
|
0.9, 0.95 and 0.99 quantiles state. Time in milliseconds (see Querying _state columns). |
|
|
Total bytes read. |
|
|
Total rows read. |
|
|
Total bytes returned state (see Querying _state columns). |
|
|
Total rows returned state (see Querying _state columns). |
tinybird.block_log¶
The Data Source contains details about how we ingested data into your Data Sources. We do ingestion in chunks.
For instance, when we ingest from a URL, if it is possible, we split the download in several requests resulting in different ingestion blocks. The same happens when the data upload happens with a multipart request: we ingest the data in several chunks as the data is arriving.
You can use this Service Data Source to spot problematic parts of your data.
Field |
Type |
Description |
---|---|---|
|
|
Date and time of the block ingestion. |
|
|
Id of the import operation. |
|
|
Id of the job that ingested the block of data, if it was ingested by URL. In this case, |
|
|
Id of the request that performed the operation. In this case, |
|
|
Either the URL or |
|
|
Block identifier. You can cross this with the |
|
|
|
|
|
Data Source consistent id. |
|
|
Data Source name when the block was ingested. |
|
|
The starting byte of the block, if the ingestion was split, where this block started. |
|
|
If split, the ending byte of the block. |
|
|
How many rows it ingested. |
|
|
Either if we used the native block parser or we had to fallback to row by row parsing. |
|
|
If any, how many rows went into the quarantine Data Source. |
|
|
If any, how many empty lines were skipped. |
|
|
How many bytes the block had. |
|
|
How long it took in seconds. |
|
|
Detailed message in case of error. |
tinybird.datasources_ops_log¶
Contains all operations performed to your Data Sources. We track the following operations:
Event |
Description |
---|---|
|
A Data Source is created. |
|
Append operation. |
|
Append operation using our High-frequency Ingestion API. |
|
Append operation using our Kafka Connector. |
|
A replace operation took place in the Data Source. |
|
A delete operation took place in the Data Source. |
|
A truncate operation took place in the Data Source. |
|
The Data Source was renamed. |
|
A populate operation was queued for execution. |
|
A finished populate operation (up to 8 hours after it started). |
|
A copy operation took place in the Data Source. |
Materializations are logged with same event_type
and operation_id
as the operation that triggers them. You can track the materialization pipe with pipe_id
and pipe_name
.
We log all of them with extra information in this Data Source.
Field |
Type |
Description |
---|---|---|
|
|
Date and time when the operation started. |
|
|
Operation being logged. |
|
|
Groups rows affected by the same operation. Useful for checking materializations triggered by an append operation. |
|
|
Id of your Data Source. The Data Source id is consistent after renaming operations. You should use the id when you want to track name changes. |
|
|
Name of your Data Source when the operation happened. |
|
|
|
|
|
How much time the operation took in seconds. |
|
|
Detailed error message if the result was error. |
|
|
Id of the import operation, if data has been ingested using one of the following operations: |
|
|
Id of the job that performed the operation, if any. If data has been ingested, |
|
|
Id of the request that performed the operation. If data has been ingested, |
|
|
How many rows the operations affected. This depends on |
|
|
How many rows went into the quarantine Data Source, if any. |
|
|
List of blocks ids that we used for the operation. See the tinybird.block_log Service Data Source for more details. |
|
|
We store key-value pairs with extra information for some of the operations. For the |
|
|
Read bytes in the operation. |
|
|
Read rows in the operation. |
|
|
Written rows in the operation. |
|
|
Written bytes in the operation. |
|
|
Quarantined rows in the operation. |
|
|
Quarantined bytes in the operation. |
|
|
If present, materialization pipe id as returned in our Pipes API. |
|
|
If present, materialization pipe name as returned in our Pipes API. |
tinybird.endpoint_errors¶
It provides the last 30 days errors of your published endpoints. We log all of them with extra information in this Data Source.
Field |
Type |
Description |
---|---|---|
|
|
Date and time when the API call started. |
|
|
The id of the request that performed the operation. |
|
|
If present, pipe id as returned in our Pipes API. |
|
|
If present, pipe name as returned in our Pipes API. |
|
|
URL query params included in the request. |
|
|
URL pathname. |
|
|
HTTP error code. |
|
|
Error message. |
tinybird.kafka_ops_log¶
Contains all operations performed to your Kafka Data Sources during the last 30 days
Field |
Type |
Description |
---|---|---|
|
|
Date and time when the operation took place. |
|
|
Id of your Data Source. The Data Source id is consistent after renaming operations. You should use the id when you want to track name changes. |
|
|
Kafka topic. |
|
|
Partition number, or |
|
|
‘info’ for regular messages, ‘warning’ for issues related to the user’s Kafka cluster, deserialization or materialized views, and ‘error’ for other issues. |
|
|
Number of messages behind for the partition. This is the difference between the high-water mark and the last commit offset. |
|
|
Messages processed for a topic and partition. |
|
|
Amount of bytes processed. |
|
|
Messages ingested for a topic and partition. |
|
|
Information in the case of warnings or errors. Empty otherwise. |
tinybird.datasources_storage¶
Contains stats about your Data Sources storage. We log max values per hour, the same we use to calculate your storage consumption.
Field |
Type |
Description |
---|---|---|
|
|
Id of your Data Source. The Data Source id is consistent after renaming operations. You should use the id when you want to track name changes. |
|
|
Name of your Data Source. |
|
|
When storage was tracked. By hour. |
|
|
Max number of bytes the Data Source has, not including quarantine. |
|
|
Max number of rows the Data Source has, not including quarantine. |
|
|
Max number of bytes the Data Source has in quarantine. |
|
|
Max number of rows the Data Source has in quarantine. |
Querying _state
columns¶
Several of the Service Data Sources include columns suffixed with _state
. This suffix identifies columns with values that are in an intermediate aggregated state. When reading these columns, you need to merge the intermediate states to get the final value.
To do this, wrap the column in the original aggregation function and apply the -Merge
combinator.
Learn more about the -Merge
combinator here.
For example, to finalize the value of the avg_duration_state
column, you use the avgMerge
function:
SELECT
date,
avgMerge(avg_duration_state) avg_time,
quantilesTimingMerge(0.9, 0.95, 0.99)(quantile_timing_state) quantiles_timing_in_ms_array
FROM tinybird.pipe_stats
where pipe_id = 'PIPE_ID'
group by date