Is there a recommended ingestion method?¶
For developers that already have 3rd party tools like Kafka, it is simple to integrate with Tinybird — docs and live coding session example. For those that don’t, we recommend the Events API for the real-time ingestion of JSON data and the Datasources API for CSV, JSON, and Parquet.
Does Tinybird use Kafka’s schema registry to infer the schema of incoming data?¶
Yes (for Kafka streams). To speed up your development, Tinybird infers the schema, but you are still able to change the data type if you wish.
What happens during ingestion if the schema does not match the data source?¶
In general, when Tinybird detects rows that don’t fit the data source schema, they are automatically sent to the quarantine data source. This way, the whole ingestion process doesn’t fail, and you can review quarantined rows later or perform operations on them in real-time using pipes.
In the case of new columns, Tinybird can detects the new columns and asks if you want to add them to the data source.
Any best practices for ingesting JSON with different schemas?¶
There is always a schema, whether on ingestion or on consumption, and we recommend setting it as soon as possible. However, you can define the schema with the CLI and, using JSONPaths, ingest the whole JSON in a column and then query it using JSONExtract functions.
Do Materialised Views affect usage?¶
We do not charge for one-time populates on Materialised Views, which means that when you create a new Materialised View, you can populate it with all existing data without any cost.
However, on-going incremental writes to Materialised Views do count towards your Tinybird usage.
Can I change the sorting key of a Data Source?¶
Currently, you can only change the sorting keys of Data Sources via the CLI. If you have already created a Data Source, you must pull the Data Source down using
tb pull and modify the
.datasource file. You’ll need to drop the existing Data Source and recreate it from the modified file.
Changing the sorting key for a Kafka Data Source is currently not supported.