Ingest from the Events API¶

Intermediate

In this guide, you'll learn how to use the Tinybird Events API to ingest thousands of JSON messages per second.

Setup: create the target Data Source¶

Firstly, you need to create an NDJSON Data Source. You can use the API, or simply drag & drop a file on the UI.

Even though you can add new columns later on, you have to upload an initial file. The Data Source will be created and ordered based upon those initial values.

As an example, upload this NDJSON file:

Ingest from the browser: JavaScript¶

Ingesting from the browser simply requires making a POST request like this. Ensure you put your own Auth token and change the name of the target Data Source to the one you created, as explained above. Remember that publishing your admin token on a public web is a security vulnerability. We highly recommend that you create a new Auth token that restricts access granularity.

Ingest from the backend: Python¶

Ingesting from the backend is as simple as from the browser. Here is a Python snippet. As before, you'll need to set your Auth token and Data Source name.

Ingest from the command line: curl¶

For completeness, here is the curl snippet that sends two events in the same request.

Adding new columns from the UI¶

As you add extra information in the form of new JSON fields, the UI will prompt you to include those new columns on the Data Source.

As an example, we'll send a new event with an extra field:

After this, we can go to the UI's Data Source modal. We'll be asked if we want to add the new column.

Here, we'll be able to select the desired columns and adjust the types.

After we confirm the addition of the column, new events will populate it.

Error handling and retries¶

Read more about the possible responses returned by the Events API.

When sending data to Tinybird using the Events API, you can choose to 'fire and forget' by simply sending a POST request and ignoring the response. This is a reasonable choice for non-critical data, such as tracking page hits if you're building Web Analytics, where some level of loss is acceptable.

However, if you're sending data where you cannot tolerate events being missed, you must implement some error handling & retry logic in your application.

This guide will provide some best practices and recommendations for handling error codes returned by the Events API.

Wait for acknowledgement¶

When you send data to the Events API, you will usually recieve a HTTP202 response, which indicates that the request was successful. However, it's important to note that this response only indicates that the Events API successfully accepted your HTTP request. It does not confirm that the data has been committed into the underlying database.

Using the wait parameter with your request will ask the Events API to wait for acknowledgement that the data you sent has been committed into the underlying database.

If you use the wait parameter, you will recieve a HTTP200 response that confirms data has been committed.

To use this, your Events API request should include wait as a query parameter, with a value of true.

For example:

https://api.tinybird.co/v0/events?wait=true

It is good practice to log your requests to, and responses from, the Events API. This will help to give you visibility into any failures for reporting or recovery.

When to retry¶

Failures are indicated by a HTTP4xx or HTTP5xx response.

It's recommended to only implement automatic retries for HTTP5xx responses, which indicate that a retry might be successful.

HTTP4xx responses should be logged and investigated, as they often indicate issues which cannot be resolved by simply retrying with the same request.

For HTTP2 clients, you may recieve the 0x07 GOAWAY error. This indicates that there are too many alive connections. It is safe to recreate the connection and retry these errors.

How to retry¶

You should aim to retry any requests that fail with a HTTP5xx response.

In general, you should retry these requests 3-5 times. If the failure persists beyond these retries, you should log the failure, and attempt to store the data in a buffer to resend later (for example, this could be in Kafka, or a file in S3).

It's recommended to use an exponential backoff between retries. This means that, after a retry fails, you should increase the amount of time you wait before sending the next retry. If the issue causing the failure is transient, this gives you a better chance of a successful retry. You should be careful when calculating backoff timings, so that you do not run into memory limits on your application.