May 25, 2022

You can now see how much data your endpoints scan in real-time

Break down every API request by scan size (rows or bytes) and token.
Juan Madurga
Senior Backend Engineer

What’s changed?

We have enriched our pipe_stats_rt service Data Source by including the data processed for every request and the token information, as well as other information related to the actual request. The new columns are:

  • pipe_name: name of the requested pipe
  • read_bytes: bytes processed for the request
  • read_rows: number of rows read in the request
  • request_id: identifies the request
  • token: the id of the token
  • token_name: the name of the token

Also, the service Data Source pipe_stats (with aggregations per day) has been extended by providing two new columns:

  • pipe_name: name of the requested pipe
  • read_bytes_sum: bytes processed for the request
  • read_rows_sum: number of rows read in the request

QueryAPI requests are included too. Just use the query_api value to filter by pipe_id or pipe_name.

Finally, since we’ve improved pipe_stats_rt, we figured we’d offer an easy way to explore that performance in the Tinybird UI. If you want to compare all your published Pipes, in the dashboard we have included the average and the total processed data:

And, if you want to check just one Pipe in detail, you’ll now find a graph showing the average processed data over time on the API endpoint’s information page. Use this graph to check usage in real-time and compare performance across time periods.

Why does this matter?

First of all, the upgrade to pipe_stats_rt means you can now create service Endpoints to monitor the amount of data processed on your Workspace. This is big news.

In addition:

  • You can identify which endpoints and requests are consuming the most data, so you can optimize them.
  • You can spot tokens that may have been leaked.
  • You can find endpoint parameters that consume comparatively more data.
  • You can detect changes in endpoints by looking at the scan size.

Get Started with the guide

Ready to optimize your endpoints? Some of these examples are available in the guide we have prepared for analyzing the performance and consumption of your API Endpoints.

Do you like this post?

Related posts

How to monitor Tinybird using Datadog with vector.dev
Changelog #19: Adding new columns in high-frequency ingestion and more
Tinybird
Team
Mar 08, 2022
Publish SQL-based endpoints on NGINX log analysis
Changelog #18: High-frequency ingestion, handling NDJSON files and more product enhancements
Tinybird
Team
Feb 01, 2022
Tinybird Changelog: New User Experience for Data Exploration
Introducing a redesigned Tinybird UI for better developer productivity
Tiny Updates: Drag and Drop to Rearrange Nodes in a Pipe
Changelog: BigQuery connector, CLI improvements, Pagination on endpoints and more
Tinybird
Team
Mar 04, 2021
A big performance boost, adding columns and more
Tinybird
Team
May 31, 2021
A new dashboard for Tinybird Analytics

Build fast data products, faster.

Try Tinybird and bring your data sources together and enable engineers to build with data in minutes. No credit card required, free to get started.
Need more? Contact sales for Enterprise support.