High-frequency Ingestion

Intermediate

In this guide, we’ll learn how to ingest JSONs at thousands of requests per second to Tinybird using v0/events. Several events can be ingested in the same request.

Feature preview:

High-frequency ingestion is currently a feature preview and it’s considered experimental.

Setup: create the target Data Source

Firstly, you need to create an NDJSON Data Source. You can use the API, or simply drag & drop a file on the UI.

Even though you can add new columns later on, you have to upload an initial file. The Data Source will be created and ordered based upon those initial values.

As an example, upload this NDJSON file:

Ingest from the browser: JavaScript

Ingesting from the browser simply requires making a POST request like this. Ensure you put your own Auth token and change the name of the target Data Source to the one you created, as explained above. Remember that publishing your admin token on a public web is a security vulnerability. We highly recommend that you create a new Auth token that restricts access granularity.

Check your URL

const url is the corresponding URL for your region: https://api.us-east.tinybird.co/ for US accounts or https://api.tinybird.co/ for the rest of the world accounts.

Ingest from the backend: Python

Ingesting from the backend is as simple as from the browser. Here is a Python snippet. As before, you’ll need to set your Auth token and Data Source name.

Ingest from the command line: curl

For completeness, here is the curl snippet that sends two events in the same request.

Adding new columns from the UI

As you add extra information in the form of new JSON fields, the UI will prompt you to include those new columns on the Data Source.

As an example, we’ll send a new event with an extra field:

After this, we can go to the UI’s Data Source modal. We’ll be asked if we want to add the new column.

data sources ui

Here, we’ll be able to select the desired columns and adjust the types.

data sources ui

After we confirm the addition of the column, new events will populate it.

data sources ui