Jun 20, 2024

Jobs log: Improved o11y for background jobs

Now it’s easier than ever to build production-level observability of your Tinybird data.
Alejandro Martín
Product Manager

Today we’re launching the jobs_log a new Service Data Source to dramatically improve insight into background jobs. As with any Service Data Source, just query it like a table (SELECT * FROM tinybird.jobs_log) to start analyzing background jobs in your Tinybird Workspace.

Tinybird is the real-time analytics platform for user-facing applications. Tinybird keeps data fast and fresh by working incrementally on data as it arrives. However, there are still various uses for long-running background tasks. You might want to create scheduled, static snapshots of a dataset using Copy Pipes, or backfill an incremental materialized view with historical data. These run as background jobs.

Until today, visibility into jobs was only possible via the Jobs API, which has limitations on the time window, log retention, and level of details. The Jobs API will remain available and is useful as a simple way to look up the status of a single job. For production-level monitoring, the jobs_log bring major improvements to job observability:

  • Detailed job insights: Monitor background job runs, gather statistics for specific job types, track ongoing jobs, identify errored jobs, and analyze queue timings.
  • Extended Log Retention: Access logs of background jobs for up to 12 months.
  • Unlimited Records: There is no limit to the number of records in the log, so all jobs and events will be tracked and available for the entire retention period.
  • Centralized Monitoring: Available in all Tinybird Workspaces as tinybird.jobs_log. For Organization Admins, you can monitor jobs across all Organization Workspaces using organization.jobs_log.

How to use it

The jobs_log can be used like any other Data Source. In a Pipe or Playground, write a SQL query using tinybird.jobs_log as the table in your FROM clause. As it’s just a Data Source, you can use this to publish API Endpoint and build Charts over your observability data.

Here’s an example of querying the jobs_log to build a time series of job status for the past week:

Just like any other Data Source, you can query jobs_log in a Pipe to create API Endpoints and Charts. Here's a quick visualization over our status time series using Tinybird Charts:

And here’s an example showing how an Organization Admin can use the organization.jobs_log to determine the Workspace creating the most jobs:

And you can visualize this as a top 5 bar list using Tinybird Charts like so:

If you're not yet a customer, you can get started with Tinybird today, with no credit card required. Check out our documentation (including advanced use cases we've employed with some of our customers) to learn more about the jobs_log, and join our Slack community if you have any questions!

Do you like this post?

Related posts

Gain Visibility Over Your Asynchronous Operations
A new dashboard for Tinybird Analytics
Iterating terabyte-sized ClickHouse tables in production
Operational Analytics in Real Time with Tinybird and Retool
Tinybird is out of beta and open to everyone
How to monitor Tinybird using Datadog with vector.dev
Publish SQL-based endpoints on NGINX log analysis
Changelog: Snowflake connector, cancellable jobs and more
Mar 23, 2021
Analyzing our own Nginx logs with ClickHouse on Tinybird
More Data, More Apps: Improving data ingestion in Tinybird

Build fast data products, faster.

Try Tinybird and bring your data sources together and enable engineers to build with data in minutes. No credit card required, free to get started.
Need more? Contact sales for Enterprise support.