
## Here is the 3 things you can't miss
- We released the new native [Snowflake connector](https://www.tinybird.co/docs/ingest/snowflake.html) which provides a very similar experience to the [Google BigQuery one](https://www.tinybird.co/docs/ingest/bigquery.html).
- Newly created columns in Snowflake or Bigquery are also added automatically to Tinybird connected Data Sources.
- You can [generate mock data directly from your Tinybird dashboard using GPT](https://www.tinybird.co/blog-posts/generate-mock-data-schemas-with-gpt), so no more excuses to try your new ideas before even having your data ready.

## Other notable fixes and improvements
- Made it easier to connect your Confluent Kafka Streams using the Confluent connector.
- [Data Copy operations](https://www.tinybird.co/docs/api-reference/pipe-api.html#data-copy-header) are now atomic which ensures data is copied when it should be copied.
- Improved error feedback on the CLI when failing creating a new Workspace and when pushing a Pipe with an empty SQL node.
- Ensured that the OpenAPI schema we generate when creating an API Endpoint is now valid as an OpenAPI 3.0 schema.
- We added the option to select advanced settings like the engine or the sorting key when creating a Data Source from the UI.
- Destructive actions in the UI now require an extra confirmation step. No more unintentional breaks.
- We did some usability improvements in the Auth Tokens page.
- We fixed some inconsistencies in the number of workspaces a Data Source is being shared with.
- We fixed the date filter in the Time Series public view.


Massive update after a busy launch week #2


## Other notable fixes and improvements
- Added a new API method to add/modify schedule operations to Copy Pipes.
- Improved UX when refreshing a token to avoid undesired changes.
- Fixed a bug that made datasource_ops_logs inaccessible from the UI under some circumstances.
- It is now possible to unshare a Data Source from the CLI.
- Fixed cURL issues in the Organizations Monitoring and API Endpoint sample usage.
- Fixed a bug that made the Data Sources modal total storage calculation wrong when having Shared Data Sources.

Au revoir Recently Used


## Notable fixes and improvements
- Added support for UUID and Datetime Avro logical types in the Kafka Connector.
- Fixed a bug that made it impossible to delete a published endpoint node with the CLI.
- Fixed the node fullscreen view within the Pipe editor.
- Added a link to the BigQuery Connector guide from the UI.
- Added Table metadata to the BigQuery Connector UI so you can how big a table is before connecting it.
- Fixed a bug that prevented the endpoint view from being opened in split screen.
- Fixed the styles of the invite actions dropdown within the Settings modal.
- Renaming a node in the UI erroneously received a warning as it was already existing in the pipe. We’ve fixed this too.
- Fixed the links on the protips (how did we miss this one?)
- Reversed the order of the jobs on the BigQuery Data Source UI.
- Fixed the styles of the Data Source type change actions dropdowns.
- Scheduling metadata is now being sent via the Pipe API.
- The platform now runs on Python 3.11. You should expect better performance. The CLI is already compatible with it too.

Tinyfixes all around


### Our new BigQuery connector is in GA
Google BigQuery users now have an easy and reliable method to [bring their data to Tinybird](https://www.tinybird.co/blog-posts/real-time-applications-with-bigquery-connector) where they can query and shape it using SQL, and publish it as low-latency APIs to be consumed by their applications. 

### Tinybird Organizations is available for all enterprise customers
Our newest feature for enterprise customers lets you monitor usage on multiple Workspaces and manage their projects across Tinybird in [a single dashboard](https://www.tinybird.co/blog-posts/enterprise-grade-real-time-analytics).

### Scheduled Copy API is already available in private beta
Tinybird has always made it possible to ingest relentless streams of data, query and shape them using SQL, and publish them as high-concurrency, low-latency APIs for use in your applications. With [Scheduled Copy](https://www.tinybird.co/blog-posts/scheduled-copy-api-launch), you can sink the results of your queries into another Tinybird Data Source.

### The Tinybird x Vercel integration is live
[Use it](https://www.tinybird.co/blog-posts/serverless-realtime-analytics-vercel-marketplace) to sync your Tinybird Workspaces with your Vercel projects to build the real-time, serverless analytics you've always wanted.

### The Tinybird Grafana plugin is here
Just build fast charts, [faster](https://www.tinybird.co/blog-posts/tinybird-grafana-plugin-launch).

Launch week’s hangover!


## The Scheduled Data Copy API is more powerful than you’d think
A few customers are already using the beta version of the Scheduled Data Copy API to generate data snapshots, do deduplication at scale, and perform other operational duties, such as copying the last month of data to a development Data Source, or moving data out of quarantine. If this rings a bell, [let us know](mailto:hi@tinybird.co) and we will give you access to the beta preview.

## Other notable fixes and improvements
- The guided tour can now be resumed if cancelled.
- It is possible for Organizations to authenticate with more than one email domain.
- When working on a pipe, you can now see the output of other nodes in split screen mode.
- Workspace guests can no longer share Data Sources with other Workspaces. Only Workspace admins can do this now.
- API Endpoint statistics can be analyzed within the Time Series UI with a single click.

A new way to copy data within Tinybird is in beta preview


## Get a birds eye view of your global usage
Track your usage commitments and manage your Workspaces and Users in a single place. Also get global observability metrics for all your Workspaces at once. If you use the Enterprise edition of Tinybird, [let us know](mailto:hi@tinybird.co) and we will activate it for you :)

## Other notable fixes and improvements
- We’re including a confirmation modal when changing the TTL. We want to avoid unnecessary data loss.
- User tokens can be refreshed (finally!).
- You can now explore specific API Endpoint metrics in the Time Series directly from the API Endpoint page (just clicking on the metric’s name).
- We’ve fixed a bug that allowed you to edit a query in a node while the query was running.
- Improved the experience when the creation of a Materialized View from the UI was giving a timeout.

## Extra ball! Supercharge your CLI workflows with tb diff
The `diff` command allows you to compare files in your local env with the remote files in your Tinybird workspace. Find out more in the [CLI docs](https://www.tinybird.co/docs/cli.html#diff), and remember to update to the last CLI version before using it.




Welcome home Organizations


## Start accelerating your BigQuery data now
Some of our users are already using the new BigQuery connector for their production applications. [Ask for private access](mailto:hi@tinybird.co) and get access to it:

- It’s configurable via the CLI and the UI.
- It allows you to maintain your data in-sync.
- It gives you a detailed log of every sync operation.
- Your data remains secure.
- It is fast.
- It comes at no extra cost.

We'd love to hear from you what connectors you will like to have available in Tinybird. [Just let us know!](https://docs.google.com/forms/d/e/1FAIpQLSeTvrblrUP7SSLtHLQTL7Xeo45T16nk8RLN7J11YL3Jc8E0AA/viewform)

## Other notable fixes and improvements

- You can now specify the ENGINE settings in Kafka Data Sources allowing you to set your own Sorting Keys or other advanced parameters.
- We’ve added [new functions](https://www.tinybird.co/docs/cli/advanced-templates.html#template-functions-title) to calculate differences between dates. Now you can calculate differences in seconds, minutes, hours and days.
- All users within a Workspace should be able to see the Workspace consumption stats.
- We’ve done some optimizations that should make the Data Sources List view load faster.

## Enterprise clients are getting a new home (soon)
Companies with many Workspaces and many users are going to have a better way to manage their permissions and track their usage commitments with our new Organization UI. Interested in learning more? [Let us know!](mailto:hi@tinybird.co)




New BigQuery connector in Private Beta


## Ingestion UI enhancements

- We've added easier access to all supported ingestion formats.
- We’ve revamped the different ingestion options UI to make it easier to follow.
- We’ve added a shortcut for ingesting data streams from Confluent cloud.
- Users can now suggest new connectors through the UI.
- The new BigQuery connector is available in private beta. [Let us know](mailto:hi@tinybird.co) if you want to give it a try.

## Other notable fixes and improvements

- Changes in the Data Source schema when previewing a newly created Data Source were being ignored under some circumstances. It shouldn’t happen again.
- Fixed a problem with [datasources_ops_log](https://www.tinybird.co/docs/monitoring/service-datasources.html) not being updated after re-pushing a Materialized View.
- Fixed an issue in the Auth tokens modal that prevented the Read / Write options to be correctly seen.
- Fixed a couple of bugs in the Data Flow view related to wrong rendering of Data Sources metadata.
- Fixed a bug that apparently allowed the creation of tokens with no scopes but raised a silent error.
- When using `tb fmt` to format your .datasource file we had a bug that removed the Data Source description. Now, the description is kept.



Introducing the new Data Ingestion Experience


## CLI

- New guided process to create a workspace using the CLI. Also improved inline help for workspace creation. Go try ‘tb workspace create –help’
- We’ve fixed an issue with the ‘tb fmt’ CLI command to format queries. If your pipe had versioning, it was removing the VERSION 0 line on formatting.


## Data Sources and Data Sources UI

- It is now easier to inspect the columns of the [Service Data Sources](https://www.tinybird.co/docs/monitoring/service-datasources.html) when working from the Pipe editor or Time Series editor.
- We are now displaying the pulse graph also for materializations coming from Kafka Data Sources.
- Users won’t be offered the option to delete Shared data sources under the "Clean up" section.
- Now you can modify existing JSONPaths in data source columns, or add new columns with JSONPaths using the [data source alter API](https://www.tinybird.co/docs/api-reference/datasource-api.html#post--v0-datasources-(.+)-alter).
- Improved error messages when problems arise adding a column to the Data Source via the UI.
- The UUID type is now available to choose when defining a new Schema via the UI.


## And more!

- We’ve introduced 30 days option in the Time Series granularity selector. 
- Added support for arbitrary (non UTF-8) bytes in Kafka message keys when using Schema Registry. 


Notable fixes and improvements


## SOC 2 Type II

Tinybird has successfully completed its SOC 2 Type II audit made by [Ernst & Young Global Limited](https://www.ey.com/en_gl/consulting/service-organization-controls-reporting), affirming the effectiveness of our security processes and controls.

This is one of the many measures we are taking to ensure the continuous security, integrity and availability of the Tinybird platform. We will be audited again in early Q2 2023 to ensure our security procedures continue to evolve as we evolve our platform.

[Learn about security and compliance at Tinybird](https://trust.tinybird.co/).

## Notable fixes / improvements

- [Tinybird Docs](https://www.tinybird.co/docs) continue to improve! Revamped our [Introduction to Tinybird](https://www.tinybird.co/docs/index.html) along with a [Main Concepts](https://www.tinybird.co/docs/main-concepts.html) section, quickstarts for Tinybird’s [CLI](https://www.tinybird.co/docs/quick-start-cli.html) and [UI](https://www.tinybird.co/docs/quick-start-ui.html), a revamped structure for Guides as well as this new Changelog that you are reading :-)

- We’ve updated the editor keyboard shortcuts and fixed some problems when using the shortcuts to query data from a node. 

![Tinybird Shortcuts](/assets/changelog/shortcuts.jpeg)

- We have completely revamped how Data Replace jobs work internally, which bring improved stability and reliability to the process. 

- Added a new check that prevents the creation of a Materialized View that produces more than 10 partitions. This significantly reduces the risk of getting a TOO_MANY_PARTS exception at ingestion time when creating a Materialized View with a high cardinality partition key.

- Solved an issue that prevented deletion or renaming of Data Sources/Pipes with GUID format.



We are SOC 2 Type II compliant


## New Features

Tinybird’s BI Connector is a PostgreSQL-compatible interface to data in Tinybird.

All Data Sources and published Pipes created in Tinybird are available as standard Postgres tables when you connect a BI tool to Tinybird, such as Tableau, Apache SuperSet or Grafana.

Until recently, BI usage was not available to track. We have now made two new service datasources available that show stats about the BI Connector consumption: [bi_stats_rt](https://www.tinybird.co/docs/monitoring/service-datasources.html#tinybird-bi-stats-rt) (real time) and [bi_stats](https://www.tinybird.co/docs/monitoring/service-datasources.html#tinybird-bi-stats) (aggregated per day and per query). 

This information is also visible in a new information module in the dashboard for the BI Connector (only visible when active) and the Query API (visible to everyone).

## Notable fixes / improvements

- New guide with [best practices and limitations around Tinybird’s BI connector](https://www.tinybird.co/docs/guides/best-practices-bi-connector.html).
- Another guide about [how to ingest streaming data from Google Pub/Sub](https://www.tinybird.co/docs/guides/ingest-from-google-pubsub.html).
- Previously, when you were running a partial replace job you couldn’t tell from the response whether it was running a full replace or a partial one. We’ve added a `replace_condition` attribute in the job response. 
- Fixed a bug where clicking a data source in a Pipe’s node would cause the page to scroll and annoyingly move the user away from the node they were editing.
- We added supported [Clickhouse types](https://www.tinybird.co/docs/api-reference/datasource-api.html#create-from-schema) to the documentation to avoid surprises :-)


Observability for the BI connector


## New Features

For API endpoint requests as well as for calls to the Query API you can access the amount of read data and rows, duration of the API call and error information (if relevant) in the [pipe_stats_rt](https://www.tinybird.co/docs/monitoring/service-datasources.html#tinybird-pipe-stats-rt) Service Datasource, as well as what parameters were used on each request. Now, you can do the same thing to monitor ingest and materialization usage.

Read and written bytes and rows are now also available in the [datasources_ops_log](https://www.tinybird.co/docs/monitoring/service-datasources.html#tinybird-datasources-ops-log) Service Datasource for every ingestion operation, whether that’s new data ingested, materializations or replaces.

And lastly, it’s now possible to track your total storage in the new [datasources_storage Service Data Source](https://www.tinybird.co/docs/monitoring/service-datasources.html?highlight=service%20data%20source#tinybird-datasources-storage).

Remember that these service Data Sources can be queried from your Pipes so you can easily create API Endpoints or Time Series visualizations to analyze your own usage.

## Notable fixes / improvements

- The Kafka connector now supports ingestion of messages encoded with non UTF-8 message keys. Until now, we were assuming that everyone would use UTF-8 keys in their messages but… apparently not!
- Improved performance for ingestion of large JSON and Parquet files, specifically for files with a large number of columns.
- Also fixed schema detection issues when importing large Parquet files from a URL. Now, we parse the whole file and use the actual schema [defined in the footer](https://parquet.apache.org/docs/file-format/), instead of trying to infer the schema as we do with other schemaless formats. As a result of this change, the whole import process will be more accurate and predictable. 
- Added support for "bytes" type in Parquet files. 
- Certain situations were causing materialized views populate jobs to appear stuck. Various improvements to progress tracking.
- If the underlying Clickhouse process was restarted while a Populate job was waiting for a query to finish, it would cause the job to get stuck and never finish. Now solved.  
- `populateview` entries in the service datasource `tinybird.datasources_ops_log` were not being properly registered when pushing a pipe with `force=true`. They are now.
- When creating a Materialized View, the columns in the GROUP BY need to match those in the Sorting Key of the destination data source. Tinybird validates that this is the case when you create a materialized view, but the validation was being triggered incorrectly in some instances.
- Fixed an issue that was exposing some internal error information when some operations on the Events API failed. Now we return an error code and a generic "Internal Tinybird error" message without leaking any unnecessary internal information.
- Now, the CLI shows warnings with tips when uploading a materialization pipe. These warnings link to documentation where you can learn how to improve your queries. Right now, there are only a few tips, but we will be adding more.
- We've released a new homepage for [Tinybird’s Documentation](https://www.tinybird.co/docs). 


Increased observability capabilities in Service Data Sources


- The “pulse” graph, the chart that displays real-time ingestion stats for every data source, is now also available for Materialized Views. This helps get a sense of how many rows are being materialized at any given time.
- We’ve introduced brushing for date range selection in the Time Series UI. 
- Previously, when adding new columns to existing data sources in the UI, rows not containing them used to go into quarantine. Now, when new columns are added, the suggested type by default is ‘Nullable(String)’.
- We’ve fixed an issue where a "Data Source not found" state would prevent you from closing the right panel for the Data Source preview view.
- Also fixed an issue that was causing the Data Source preview to close when hitting escape within the code editor.
- Made all buttons highlight states consistent across the platform.

![Buttons highlight states](/assets/changelog/buttons-highlight-states.png)






## Events API

While Kafka remains a popular way to ingest events data, sometimes our users just want a super simple way to send an events stream to Tinybird. And there’s nothing simpler than sending events through HTTPS via our events API. Just copy the snippet, run the script, and see your data instantly show up in the workspace for you to start querying.

## CLI updates

One of our core beliefs at Tinybird is that you should be able to build with data like you build with code. And that means being able to run tests. We’re excited to announce our first few feature launches to enable testing in Tinybird data projects. In particular, the CLI has new commands and plus some added functionality, including:

- `tb test` to add tests to files
- New metrics about endpoint response times (max, min, mean, median and p90) on the pipe command `tb pipe regression-test`
- `tb workspace clear` now deletes all files in the workspace. Given that this command drops all the resources inside a project, please use it with care!
- `tb pipe publish` to change which node of the Pipe is published as an endpoint.
- `tb check` to verify query syntax.You can find all the latest changes in the [command-line updates](https://pypi.org/project/tinybird-cli/).

## ClickHouse

Our [work](https://github.com/ClickHouse/ClickHouse/pull/37187) to allow decimal values in seconds (for example, `max_execution_time=0.5`) was merged.


Quickstart on loading events to Tinybird


## Data Source descriptions

In a large workspace that contains many Data Sources, you may want more information than just the name of the Data Source. Documentation matters. Now you can add a description to a Data Source like you already do with pipes, nodes and endpoints.

This feature is available through the UI and the CLI. Any new descriptions will propagate to shared sources.

## Early beta support for Parquet files

At Tinybird we aim to capture and transform large amounts of data whatever the origin of the data or format. In addition to CSV and NDJSON, we're working on accepting Parquet format files. Parquet is an open-source, column-oriented data file format designed for efficient data storage and retrieval. It is also commonly used as an interchange format between data tools.

Our team is now testing ingesting data to Tinybird from Parquet. After further testing, this new format for ingestion will be included in our docs. If you'd like you join the beta reach out to us on [Slack](https://join.slack.com/t/tinybird-community/shared_invite/zt-yi4hb0ht-IXn9iVuewXIs3QXVqKS~NQ)

## CLI updates!

**tb push --subset** - We've added the `tb push --subset` option to be used with `--populate` so you can populate using only a subset of the data of between 0% and 10%. Now you can quickly validate a materialized view with just a subset of your total dataset. You can check everything is working with a single month’s worth of data even if you have several years’ worth of data.

**Data Source description** - We've added the option of adding a description for a Data Source, as you already could for pipes, thereby improving the documentation of your data project.

**Endpoints from materialized Data Sources** - We've fixed code so that nodes whose type is materialized can no longer be published as endpoints. Logically the endpoint should depend on the target Data Source of the materialized node, not the node itself.

Check out the latest [command-line updates](https://pypi.org/project/tinybird-cli/) in the changelog.

## ClickHouse improvements

[**groupSortedArray**](https://github.com/ClickHouse/ClickHouse/pull/34055) - This new aggregation function was added to ClickHouse. `groupSortedArray(n)(param1)` returns an array with the n first values from a field param1 sorted by itself. `groupSortedArray(n)(param1, param2)` returns an array with the n first values from a field param1 sorted by param2 (field or expression). This aggregation function is useful if, for example, you have a materialized view with the two most recent values in a field.

**ASOF join** - The [ASOF join performance improvement](https://github.com/ClickHouse/ClickHouse/pull/35525) was included in Tinybird. This join was improved by the ClickHouse community to be twice as fast.
