Aug 26, 2022

We've improved notifications for ingestion issues

A little boost to ingestion observability in Tinybird.
Rafa Moreno
Frontend Engineer

To do anything valuable with Tinybird, you first have to get data into it, whether from a third-party source like Kafka or S3, or using the Tinybird Events API. Because data ingestion is the analytical foundation of everything we offer, we’re hyper-focused on making sure it’s a reliable, observable process.

While you have access to all of the raw observability data through Tinybird’s Service Data Sources, we realized a simple feature might help improve basic observability...

You now get an email for data ingestion issues

If ingestion fails completely, resulting in data loss you’ll get an email like the one below letting you know about it.

Example email you'll receive if ingestion fails.

Likewise, you’ll get a similar email if data ends up in quarantine:

Example email you'll receive if ingestion sends rows to quarantine.

If you want to manage who gets these emails, you can do so in the Member configuration of the Workspace:

You can adjust email notification settings in the Member Configuration of the Workspace. Admins get notified by default.

Add your own observability layer with datasources_ops_log

Every Tinybird workspace includes the Service Data Source datasources_ops_log with information that you can use to build your own observability functions.

The email alerts you now get are built on top of this datasource, so feel free to expand these alerts into your own observability platforms whether that’s email, Slack, Datadog, or… anything.

Related: How to monitor Tinybird with Datadog using

For example, here’s a simple snippet of SQL to get, for the workspace, all append events in the last week that sent rows to quarantine:

As always, you can easily publish these results as an Endpoint that your observability services can call.

Let us know if this helps!

If you have any questions about these notifications, or ideas about how we can improve observability features, please let us know in Slack.

Do you like this post?

Related posts

Using custom Kafka headers for advanced message processing
More Data, More Apps: Improving data ingestion in Tinybird
How to build a real-time fraud detection system
Iterating terabyte-sized ClickHouse tables in production
Tinybird connects with Confluent for real-time streaming analytics at scale
Jul 18, 2023
Simplifying event sourcing with scheduled data snapshots in Tinybird
Designing and implementing a weather data API
A new way to create intermediate Data Sources in Tinybird
Jun 15, 2023
Building an enterprise-grade real-time analytics platform
Automating customer usage alerts with Tinybird and Make

Build fast data products, faster.

Try Tinybird and bring your data sources together and enable engineers to build with data in minutes. No credit card required, free to get started.
Need more? Contact sales for Enterprise support.