Testing Pipe parameters¶

You can now easily test your Pipes with custom parameters from inside the UI.

Tinybird Pipes allow you to extend your SQL with templating, making it possible to use conditional logic, or add dynamic parameters. When you add dynamic parameters, you often want to test the execution of the query with different parameter values.

Previously, this would usually require publishing the query as an API Endpoint, and then modifying the URL query parameters.

With this new feature, you can now supply custom values and test query execution without leaving the UI.

Happy new year!¶

Before we kick into the first changelog of 2024, let us wish you a Happy New Year!

Looking back, 2023 has been an explosive year for Tinybird and our users. Together, we served over 45 billion API requests to power some amazing applications. In our creator community, our friends at Dub.co tracked over 18 million link clicks, while OpenStatus hit 1 million requests per day!

We can’t wait to see what people will build with Tinybird in 2024 ❤️

Compression details¶

You can now view the detailed breakdown of compression statistics per-column for every Data Source.

Tinybird automatically compresses ingested data, achieving significant reductions in data storage. The exact compression ratio depends on the data and column type, but until now, it hasn’t been as easy as it should be to identify opportunities to optimize storage. With this new feature, you can now see the compression ratio for each column, and exactly how much storage the column consumes, for every Data Source. You can access this in the UI on the Schema tab of the Data Source details page.

Viewer role for read-only access¶

A new ‘Viewer’ role has been added for read-only permissions on a Workspace.

The new ‘Viewer’ role allows for users to join a Workspace in a read-only capacity, being able to view queries and their results, but unable to edit resources. We had several requests from teams that wanted to give users access to Tinybird for visibility, but without a need for them to make any changes.

AWS EU Central is now public!¶

Tinybird is now available in the AWS eu-central-1 (Frankfurt, Germany) region for all Free and Pro users!

In December last year, we launched our first public AWS region in the US East, and promised that we’d follow up with an EU region shortly. Well, here it is! This complements our availability in GCP, giving Tinybird an EU and US region in both AWS and GCP.

Other notable changes ¶

• v2.1.0 of the CLI fixes several issues, improves performance of various operations and adds some new safety checks for dangerous commands
• Fix tb auth unexpected behavior when switching between Workspaces in different cloud providers

CLI 2.0.0¶

The CLI 2.0.0 release brings some important breaking changes: consistent error codes, and the removal of prefixes and the tb pipe create command.

You can find the release in pypi.

if(failure) return 1;¶

All CLI commands now return status code 1 on failure.

Several CLI commands didn't return 1 on failure, which meant that they could silently fail while reporting success. This added some friction to building great CI processes, so we're ensuring that, going forward, all commands are consistent in their return values.

Removed prefixes¶

All prefix commands have been removed from the CLI.

We first announced the deprecation of prefixes in October. Since then, we've been in contact with all active prefix users to help them migrate off.

Prefixes were the original way to manage different projects in Tinybird, before it was possible to create multiple Workspaces. Using Workspaces has been the preferred workflow since their release, but prefixes continued to be supported for legacy users. From this release, we are saying goodbye to prefixes for good.

Removed tb pipe create¶

The tb pipe create has been removed from the CLI.

This command was deprecated & hidden from the CLI some time ago, as it added no benefit over simply creating a .pipe file yourself. Also, no one was using it 🤷‍♀️

AWS US East is now public!¶

We are excited to announce that Tinybird is now available on Amazon Web Services (AWS) for self-service Free and Pro plans. Customers can now choose to build real-time data products with Tinybird in AWS US East (us-east-1), with EU regions coming soon.

Read more.

EXPLAIN goes GA¶

Every query Node within a Pipe now has an Explain link, giving you access to the full EXPLAIN plan of your query.

Tinybird already exposes a lot of information to help engineers tweak and optimize their queries, like showing you how much data was processed, automatically detecting full scan queries and providing the full observability log. However, when you want to tease out the last few percentage points of optimization, nothing beats the EXPLAIN plan. The EXPLAIN plan shows exactly how the underlying database is building and executing the query.

This feature is now GA available within the Tinybird UI, and should be expanded to the API and CLI soon.

Other notable changes ¶

• Added Map column type for NDJSON ingest
• Improved error messages when calling the wrong API base URL for your Auth Token
• Improved documentation for region-specific API base URLs

Updated summary page for Materialized Views¶

Published Materialized Views now have a summary page showing various details and metrics, such as their processed data, duration and errors, as well as a visual display of the Materialized Views data flow lineage.

When you publish an API Endpoint in Tinybird, you can access a summary page via the UI to see the API Endpoint’s requests, latency, errors and access its logs. It’s a great way for an engineer to get quick insights into an API Endpoint. However, when publishing a Materialized View, these insights were only available by analyzing the data in the Services Data Source, which takes a little more effort. The new Materialized View summary page makes the most important stats available immediately for quick monitoring.

Other notable changes ¶

• The Tokens page has been updated to show more details and now supports bulk actions and filters
• -SimpleState has been removed from the automatic query optimizer for Materialized Views. You can still use this modifier when manually defining Materialized View queries.
• When using Versions, Guest users are now more restricted in read-only Environments
• Improved some UI elements where long resource names could break formatting
• Improved handling of optional API parameters in auto-generated OpenAPI specs
• On the Data Source details page, renamed the Graph tab to Data flow to match the side nav
• Pipe queries that includes full scans over a Data Source continue to be a common anti-pattern that affect performance of API Endpoints, we’ve improved the design & added more information to the tooltip when we detect a query with a full scan
• Lots of improved error messages, modals and tooltips

Full screen query editor¶

You can now expand the Node query editor into a full screen view, making it easier to work with longer queries.

Tinybird Pipes allow you to break your queries down into small, chainable chunks that are much easier to build, share and maintain. However, there is a balance: no one wants 100 single line queries chained together. There’s no rules on how long your queries should be, or how many Nodes you must use, but we’ve found many users still have queries that are somewhere between 10-50 lines in a single Node. The way Nodes are displayed today means that these longer queries are cut off and require scrolling, which can disrupt the development flow. The new full screen editor gives you much more space to work in, can be split horizontally or vertically, and should make longer queries much easier to work with.

New sidebar navigation¶

The sidebar has been redesigned to provide better organization of your Tinybird resources.

As data projects grow, the amount of Pipes and Data Sources can become overwhelming. In the previous design, it could become hard to find the resources you wanted as there was too much noise present in the sidebar. Resources are now collapsed under their grouping (e.g., Pipes, Data Sources) and can be expanded into a table view in the main UI area. This new view also includes search and stats, making it easier to filter for the resources you care about right now.

Other notable changes ¶

• Better handling for query templating syntax errors, with error messages that now point to the Pipe and Node that contains the error
• Fixed a bug when sending Int types over the Events API where the field name and JSONPath don’t match
• The rows_before_limit_at_least field in an API Endpoint’s JSON response is now marked as optional in the OpenAPI spec

CLI 1.0.0 Stable¶

The Tinybird CLI is an essential tool for our customers. It has been the preferred way to interact with Tinybird APIs for many of our customers. With our recent Versions release, it has become the indispensable “command center” for version control and CI/CD operations. The importance of the CLI in our users’ daily operations cannot be overstated, and we recognize that bugs and instability in the tool can limit their productivity in our platform.

We are excited to announce our release of a stable version of the Tinybird CLI. The stable version is live on PyPi, and you can upgrade with pip:

pip install tinybird-cli==1.0.0

Other notable changes ¶

• Query parameters are now fully supported when using the Query API
• Lists can now be used as default values in the templating Array function
• You can no longer use an Endpoint Pipe for a Copy operation, use Copy Pipes instead
• Auto-generated OpenAPI specs now also include POST endpoints
• The TOO_MANY_SIMULTANEOUS_QUERIES error has been replaced with a more useful & actionable error message

Versions¶

The way you work with data is getting an upgrade. Iterating real-time data pipelines is very, very hard. And we’ve learned through trial, error, and long hours of customer support the patterns and practices that make it easier. Now, we’re enforcing those patterns, because the same techniques that software engineering teams use to collaborate on and deploy software products must also apply to real-time data products.

Read more.

Playground¶

The Playground provides a scratchpad for exploring and prototyping new queries. When an Environment is marked as read-only, the Playground can be used to safely write ad-hoc queries without requiring a new development Environment or impacting production.

EXPLAINs¶

EXPLAIN queries are now supported, allowing you to view the detailed query execution plan for your Pipe queries.

Command palette with CMD+K¶

A new command palette is accessible via the cmd+k hotkey. Use this to access a brand new search engine for your Tinybird resources, and navigate through the Tinybird UI using only the keyboard.

Region selector¶

The region selector has been extended. You can now select a cloud provider, see more region options, and request new regions without contacting support.

Deprecations 🚨¶

Python 3.7¶

Python 3.7 has been officially deprecated and is no longer supported by the Tinybird CLI.

Prefixes¶

Prefixes have been discouraged since the introduction of Workspaces. They are now officially deprecated and will no longer be supported. Most users moved away from prefixes a long time ago, so the impact should be minimal, but if you are still using prefixes and need support, please reach out in the Slack community!

Other notable changes ¶

• Ingesting CSVs now supports GZIP compression. Files must have the .gz extension.
• The date and timestamp reserved words are now allowed as column names when using the Snowflake connector
• Fixed a bug that caused node names not to re-render after the node name was changed
• Improved the error message when pushing a Copy Pipe from the CLI that contained invalid SQL
• Introduced a banner to remind users to save changes when modifying Auth Tokens
• Added the ability to duplicate an Auth Token

S3 connector¶

To bridge the gap between Amazon S3 and real-time applications, we’ve launched the Amazon S3 Connector. This first-party connector simplifies the process of ingesting data from files in Amazon S3 into Tinybird. Configured with either the UI and CLI, Tinybird can connect directly to your S3 bucket, discover files and ingest them into a Data Source, so you can publish APIs from your files.

Other notable changes ¶

• Accessing observability data from internal Service Data Sources is now free of charge
• Brand new Workspace settings modal, now has much more information and better organized for clarity
• Bug Fix - sometimes the UI would show the wrong chart for the BigQuery connector
• Bug Fix - BigQuery previews would sometimes fail
• Bug Fix - changing the name of Pipe to use the same name as another Pipe stopped displaying an error in the UI
• Bug Fix - some edge cases where Copy Pipes would not detect compatible schemas with a Data Source
• Quality of Life - the Token API /delete endpoint can now delete tokens by name or ID, not just by the token content
• Quality of Life - the Tinybird region (EU/US) has been added to the Auth Token payload, so you can now use tb auth from the CLI and it will auto-detect which region to authenticate against

After two back to back launch weeks, we took some time off from publishing a changelog to catch our breath - but we’re back! And we have some amazing things to share with you.

In June, we shipped some major new features, some QOL improvements, and over 100 minor bug fixes, tech debt resolutions, and security upgrades.

Copy Pipes¶

Copy Pipes let you sink the results of a Pipe into a Data Source either on demand or on a schedule. You can use them to create point-in-time snapshots, consolidate the results of CDC, finalize deduplication and more. Read the announcement here.

Other notable fixes and improvements ¶

• Although you might not notice it directly, we rebuilt a core piece of our architecture that handles the high-availability & failover capabilities for our Events API and the API Endpoints you publish. The old architecture was still letting us achieve over 99.9% availability, but this change will see us into the future. You can always monitor our availability at status.tinybird.co
• The Tinybird VS Code extension has had a huge upgrade.
• Our ClickHouse team has been working hard to improve support for Parallel Replicas which will make Tinybird scale effortlessly into a trillion-row future.
• When using the Kafka connector, you can now choose to include any Kafka headers along with the event message.
• Any API Endpoint can now be explored in the Time Series UI
• We added support for Map types, making it easier to ingest data with complex dictionary-style data.
• The email notifications about Workspace consumption have been improved, to make it easier to be alerted about changing in your Workspace without needing to log in.
• We’ve improved a wide range of error messages in the platform to be more consistent & helpful, particularly around Kafka & Parquet files.
• For our Enterprise customers with dedicated infrastructure, we’ve merged support for isolated Kafka consumers to allow to for more secure & predictable access to disparate Kafka clusters.
• Tinybird’s color palette has been updated! Our green & blue hues in the UI have been updated with richer colors that improve contrast and make the UI more accessible.
• You can now drag & drop your Tinybird resource files (.pipe, .datasource, etc.) onto the UI to create them without the CLI.
• From the CLI, you can also delete Tinybird resources using the resource files, e.g. tb pipe rm my_pipe.pipe.

Here is the 3 things you can't miss¶

• We released the new native Snowflake connector which provides a very similar experience to the Google BigQuery one.
• Newly created columns in Snowflake or Bigquery are also added automatically to Tinybird connected Data Sources.
• You can generate mock data directly from your Tinybird dashboard using GPT, so no more excuses to try your new ideas before even having your data ready.

Other notable fixes and improvements ¶

• Made it easier to connect your Confluent Kafka Streams using the Confluent connector.
• Data Copy operations are now atomic which ensures data is copied when it should be copied.
• Improved error feedback on the CLI when failing creating a new Workspace and when pushing a Pipe with an empty SQL node.
• Ensured that the OpenAPI schema we generate when creating an API Endpoint is now valid as an OpenAPI 3.0 schema.
• We added the option to select advanced settings like the engine or the sorting key when creating a Data Source from the UI.
• Destructive actions in the UI now require an extra confirmation step. No more unintentional breaks.
• We did some usability improvements in the Auth Tokens page.
• We fixed some inconsistencies in the number of workspaces a Data Source is being shared with.
• We fixed the date filter in the Time Series public view.

Other notable fixes and improvements ¶

• Added a new API method to add/modify schedule operations to Copy Pipes.
• Improved UX when refreshing a token to avoid undesired changes.
• Fixed a bug that made datasource_ops_logs inaccessible from the UI under some circumstances.
• It is now possible to unshare a Data Source from the CLI.
• Fixed cURL issues in the Organizations Monitoring and API Endpoint sample usage.
• Fixed a bug that made the Data Sources modal total storage calculation wrong when having Shared Data Sources.

Notable fixes and improvements ¶

• Added support for UUID and Datetime Avro logical types in the Kafka Connector.
• Fixed a bug that made it impossible to delete a published endpoint node with the CLI.
• Fixed the node fullscreen view within the Pipe editor.
• Added a link to the BigQuery Connector guide from the UI.
• Added Table metadata to the BigQuery Connector UI so you can how big a table is before connecting it.
• Fixed a bug that prevented the endpoint view from being opened in split screen.
• Fixed the styles of the invite actions dropdown within the Settings modal.
• Renaming a node in the UI erroneously received a warning as it was already existing in the pipe. We’ve fixed this too.
• Fixed the links on the protips (how did we miss this one?)
• Reversed the order of the jobs on the BigQuery Data Source UI.
• Fixed the styles of the Data Source type change actions dropdowns.
• Scheduling metadata is now being sent via the Pipe API.
• The platform now runs on Python 3.11. You should expect better performance. The CLI is already compatible with it too.

Our new BigQuery connector is in GA¶

Google BigQuery users now have an easy and reliable method to bring their data to Tinybird where they can query and shape it using SQL, and publish it as low-latency APIs to be consumed by their applications.

Tinybird Organizations is available for all enterprise customers¶

Our newest feature for enterprise customers lets you monitor usage on multiple Workspaces and manage their projects across Tinybird in a single dashboard.

Scheduled Copy API is already available in private beta¶

Tinybird has always made it possible to ingest relentless streams of data, query and shape them using SQL, and publish them as high-concurrency, low-latency APIs for use in your applications. With Scheduled Copy, you can sink the results of your queries into another Tinybird Data Source.

The Tinybird x Vercel integration is live¶

Use it to sync your Tinybird Workspaces with your Vercel projects to build the real-time, serverless analytics you've always wanted.

The Tinybird Grafana plugin is here¶

Just build fast charts, faster.

The Scheduled Data Copy API is more powerful than you’d think¶

A few customers are already using the beta version of the Scheduled Data Copy API to generate data snapshots, do deduplication at scale, and perform other operational duties, such as copying the last month of data to a development Data Source, or moving data out of quarantine. If this rings a bell, let us know and we will give you access to the beta preview.

Other notable fixes and improvements ¶

• The guided tour can now be resumed if cancelled.
• It is possible for Organizations to authenticate with more than one email domain.
• When working on a pipe, you can now see the output of other nodes in split screen mode.
• Workspace guests can no longer share Data Sources with other Workspaces. Only Workspace admins can do this now.
• API Endpoint statistics can be analyzed within the Time Series UI with a single click.

Get a birds eye view of your global usage¶

Track your usage commitments and manage your Workspaces and Users in a single place. Also get global observability metrics for all your Workspaces at once. If you use the Enterprise edition of Tinybird, let us know and we will activate it for you :)

Other notable fixes and improvements ¶

• We’re including a confirmation modal when changing the TTL. We want to avoid unnecessary data loss.
• User tokens can be refreshed (finally!).
• You can now explore specific API Endpoint metrics in the Time Series directly from the API Endpoint page (just clicking on the metric’s name).
• We’ve fixed a bug that allowed you to edit a query in a node while the query was running.
• Improved the experience when the creation of a Materialized View from the UI was giving a timeout.

Extra ball! Supercharge your CLI workflows with tb diff¶

The diff command allows you to compare files in your local env with the remote files in your Tinybird workspace. Find out more in the CLI docs, and remember to update to the last CLI version before using it.

Start accelerating your BigQuery data now¶

Some of our users are already using the new BigQuery connector for their production applications. Ask for private access and get access to it:

• It’s configurable via the CLI and the UI.
• It allows you to maintain your data in-sync.
• It gives you a detailed log of every sync operation.
• Your data remains secure.
• It is fast.
• It comes at no extra cost.

We'd love to hear from you what connectors you will like to have available in Tinybird. Just let us know!

Other notable fixes and improvements ¶

• You can now specify the ENGINE settings in Kafka Data Sources allowing you to set your own Sorting Keys or other advanced parameters.
• We’ve added new functions to calculate differences between dates. Now you can calculate differences in seconds, minutes, hours and days.
• All users within a Workspace should be able to see the Workspace consumption stats.
• We’ve done some optimizations that should make the Data Sources List view load faster.

Enterprise clients are getting a new home (soon)¶

Companies with many Workspaces and many users are going to have a better way to manage their permissions and track their usage commitments with our new Organization UI. Interested in learning more? Let us know!

Ingestion UI enhancements¶

• We've added easier access to all supported ingestion formats.
• We’ve revamped the different ingestion options UI to make it easier to follow.
• We’ve added a shortcut for ingesting data streams from Confluent cloud.
• Users can now suggest new connectors through the UI.
• The new BigQuery connector is available in private beta. Let us know if you want to give it a try.

Other notable fixes and improvements ¶

• Changes in the Data Source schema when previewing a newly created Data Source were being ignored under some circumstances. It shouldn’t happen again.
• Fixed a problem with datasources_ops_log not being updated after re-pushing a Materialized View.
• Fixed an issue in the Auth tokens modal that prevented the Read / Write options to be correctly seen.
• Fixed a couple of bugs in the Data Flow view related to wrong rendering of Data Sources metadata.
• Fixed a bug that apparently allowed the creation of tokens with no scopes but raised a silent error.
• When using tb fmt to format your .datasource file we had a bug that removed the Data Source description. Now, the description is kept.

CLI¶

• New guided process to create a workspace using the CLI. Also improved inline help for workspace creation. Go try ‘tb workspace create –help’
• We’ve fixed an issue with the ‘tb fmt’ CLI command to format queries. If your pipe had versioning, it was removing the VERSION 0 line on formatting.

Data Sources and Data Sources UI¶

• It is now easier to inspect the columns of the Service Data Sources when working from the Pipe editor or Time Series editor.
• We are now displaying the pulse graph also for materializations coming from Kafka Data Sources.
• Users won’t be offered the option to delete Shared data sources under the "Clean up" section.
• Now you can modify existing JSONPaths in data source columns, or add new columns with JSONPaths using the data source alter API.
• Improved error messages when problems arise adding a column to the Data Source via the UI.
• The UUID type is now available to choose when defining a new Schema via the UI.

And more!¶

• We’ve introduced 30 days option in the Time Series granularity selector.
• Added support for arbitrary (non UTF-8) bytes in Kafka message keys when using Schema Registry.

SOC 2 Type II¶

Tinybird has successfully completed its SOC 2 Type II audit made by Ernst & Young Global Limited, affirming the effectiveness of our security processes and controls.

This is one of the many measures we are taking to ensure the continuous security, integrity and availability of the Tinybird platform. We will be audited again in early Q2 2023 to ensure our security procedures continue to evolve as we evolve our platform.

Learn about security and compliance at Tinybird.

Notable fixes / improvements ¶

• Tinybird Docs continue to improve! Revamped our Introduction to Tinybird along with a Main Concepts section, quickstarts for Tinybird’s CLI and UI, a revamped structure for Guides as well as this new Changelog that you are reading :-)

• We’ve updated the editor keyboard shortcuts and fixed some problems when using the shortcuts to query data from a node.

• We have completely revamped how Data Replace jobs work internally, which bring improved stability and reliability to the process.

• Added a new check that prevents the creation of a Materialized View that produces more than 10 partitions. This significantly reduces the risk of getting a TOO_MANY_PARTS exception at ingestion time when creating a Materialized View with a high cardinality partition key.

• Solved an issue that prevented deletion or renaming of Data Sources/Pipes with GUID format.

New Features¶

Tinybird’s BI Connector is a PostgreSQL-compatible interface to data in Tinybird.

All Data Sources and published Pipes created in Tinybird are available as standard Postgres tables when you connect a BI tool to Tinybird, such as Tableau, Apache SuperSet or Grafana.

Until recently, BI usage was not available to track. We have now made two new service datasources available that show stats about the BI Connector consumption: bi_stats_rt (real time) and bi_stats (aggregated per day and per query).

This information is also visible in a new information module in the dashboard for the BI Connector (only visible when active) and the Query API (visible to everyone).

Notable fixes / improvements ¶

• New guide with best practices and limitations around Tinybird’s BI connector.
• Another guide about how to ingest streaming data from Google Pub/Sub.
• Previously, when you were running a partial replace job you couldn’t tell from the response whether it was running a full replace or a partial one. We’ve added a replace_condition attribute in the job response.
• Fixed a bug where clicking a data source in a Pipe’s node would cause the page to scroll and annoyingly move the user away from the node they were editing.
• We added supported Clickhouse types to the documentation to avoid surprises :-)

New Features¶

For API endpoint requests as well as for calls to the Query API you can access the amount of read data and rows, duration of the API call and error information (if relevant) in the pipe_stats_rt Service Datasource, as well as what parameters were used on each request. Now, you can do the same thing to monitor ingest and materialization usage.

Read and written bytes and rows are now also available in the datasources_ops_log Service Datasource for every ingestion operation, whether that’s new data ingested, materializations or replaces.

And lastly, it’s now possible to track your total storage in the new datasources_storage Service Data Source.

Remember that these service Data Sources can be queried from your Pipes so you can easily create API Endpoints or Time Series visualizations to analyze your own usage.

Notable fixes / improvements ¶

• The Kafka connector now supports ingestion of messages encoded with non UTF-8 message keys. Until now, we were assuming that everyone would use UTF-8 keys in their messages but… apparently not!
• Improved performance for ingestion of large JSON and Parquet files, specifically for files with a large number of columns.
• Also fixed schema detection issues when importing large Parquet files from a URL. Now, we parse the whole file and use the actual schema defined in the footer, instead of trying to infer the schema as we do with other schemaless formats. As a result of this change, the whole import process will be more accurate and predictable.
• Added support for "bytes" type in Parquet files.
• Certain situations were causing materialized views populate jobs to appear stuck. Various improvements to progress tracking.
• If the underlying Clickhouse process was restarted while a Populate job was waiting for a query to finish, it would cause the job to get stuck and never finish. Now solved.
• populateview entries in the service datasource tinybird.datasources_ops_log were not being properly registered when pushing a pipe with force=true. They are now.
• When creating a Materialized View, the columns in the GROUP BY need to match those in the Sorting Key of the destination data source. Tinybird validates that this is the case when you create a materialized view, but the validation was being triggered incorrectly in some instances.
• Fixed an issue that was exposing some internal error information when some operations on the Events API failed. Now we return an error code and a generic "Internal Tinybird error" message without leaking any unnecessary internal information.
• Now, the CLI shows warnings with tips when uploading a materialization pipe. These warnings link to documentation where you can learn how to improve your queries. Right now, there are only a few tips, but we will be adding more.
• We've released a new homepage for Tinybird’s Documentation.

• The “pulse” graph, the chart that displays real-time ingestion stats for every data source, is now also available for Materialized Views. This helps get a sense of how many rows are being materialized at any given time.
• We’ve introduced brushing for date range selection in the Time Series UI.
• Previously, when adding new columns to existing data sources in the UI, rows not containing them used to go into quarantine. Now, when new columns are added, the suggested type by default is ‘Nullable(String)’.
• We’ve fixed an issue where a "Data Source not found" state would prevent you from closing the right panel for the Data Source preview view.
• Also fixed an issue that was causing the Data Source preview to close when hitting escape within the code editor.
• Made all buttons highlight states consistent across the platform.

Events API¶

While Kafka remains a popular way to ingest events data, sometimes our users just want a super simple way to send an events stream to Tinybird. And there’s nothing simpler than sending events through HTTPS via our events API. Just copy the snippet, run the script, and see your data instantly show up in the workspace for you to start querying.

CLI updates¶

One of our core beliefs at Tinybird is that you should be able to build with data like you build with code. And that means being able to run tests. We’re excited to announce our first few feature launches to enable testing in Tinybird data projects. In particular, the CLI has new commands and plus some added functionality, including:

• tb test to add tests to files
• New metrics about endpoint response times (max, min, mean, median and p90) on the pipe command tb pipe regression-test
• tb workspace clear now deletes all files in the workspace. Given that this command drops all the resources inside a project, please use it with care!
• tb pipe publish to change which node of the Pipe is published as an endpoint.
• tb check to verify query syntax.You can find all the latest changes in the command-line updates.

ClickHouse¶

Our work to allow decimal values in seconds (for example, max_execution_time=0.5) was merged.

Data Source descriptions¶

In a large workspace that contains many Data Sources, you may want more information than just the name of the Data Source. Documentation matters. Now you can add a description to a Data Source like you already do with pipes, nodes and endpoints.

This feature is available through the UI and the CLI. Any new descriptions will propagate to shared sources.

Early beta support for Parquet files¶

At Tinybird we aim to capture and transform large amounts of data whatever the origin of the data or format. In addition to CSV and NDJSON, we're working on accepting Parquet format files. Parquet is an open-source, column-oriented data file format designed for efficient data storage and retrieval. It is also commonly used as an interchange format between data tools.

Our team is now testing ingesting data to Tinybird from Parquet. After further testing, this new format for ingestion will be included in our docs. If you'd like you join the beta reach out to us on Slack

CLI updates!¶

tb push --subset - We've added the tb push --subset option to be used with --populate so you can populate using only a subset of the data of between 0% and 10%. Now you can quickly validate a materialized view with just a subset of your total dataset. You can check everything is working with a single month’s worth of data even if you have several years’ worth of data.

Data Source description - We've added the option of adding a description for a Data Source, as you already could for pipes, thereby improving the documentation of your data project.

Endpoints from materialized Data Sources - We've fixed code so that nodes whose type is materialized can no longer be published as endpoints. Logically the endpoint should depend on the target Data Source of the materialized node, not the node itself.

Check out the latest command-line updates in the changelog.

ClickHouse improvements¶

groupSortedArray - This new aggregation function was added to ClickHouse. groupSortedArray(n)(param1) returns an array with the n first values from a field param1 sorted by itself. groupSortedArray(n)(param1, param2) returns an array with the n first values from a field param1 sorted by param2 (field or expression). This aggregation function is useful if, for example, you have a materialized view with the two most recent values in a field.

ASOF join - The ASOF join performance improvement was included in Tinybird. This join was improved by the ClickHouse community to be twice as fast.