
## Start accelerating your BigQuery data now
Some of our users are already using the new BigQuery connector for their production applications. [Ask for private access](mailto:hi@tinybird.co) and get access to it:

- It’s configurable via the CLI and the UI.
- It allows you to maintain your data in-sync.
- It gives you a detailed log of every sync operation.
- Your data remains secure.
- It is fast.
- It comes at no extra cost.

We'd love to hear from you what connectors you will like to have available in Tinybird. [Just let us know!](https://docs.google.com/forms/d/e/1FAIpQLSeTvrblrUP7SSLtHLQTL7Xeo45T16nk8RLN7J11YL3Jc8E0AA/viewform)

## Other notable fixes and improvements

- You can now specify the ENGINE settings in Kafka Data Sources allowing you to set your own Sorting Keys or other advanced parameters.
- We’ve added [new functions](https://www.tinybird.co/docs/cli/advanced-templates.html#template-functions-title) to calculate differences between dates. Now you can calculate differences in seconds, minutes, hours and days.
- All users within a Workspace should be able to see the Workspace consumption stats.
- We’ve done some optimizations that should make the Data Sources List view load faster.

## Enterprise clients are getting a new home (soon)
Companies with many Workspaces and many users are going to have a better way to manage their permissions and track their usage commitments with our new Organization UI. Interested in learning more? [Let us know!](mailto:hi@tinybird.co)




New BigQuery connector in Private Beta


## Ingestion UI enhancements

- We've added easier access to all supported ingestion formats.
- We’ve revamped the different ingestion options UI to make it easier to follow.
- We’ve added a shortcut for ingesting data streams from Confluent cloud.
- Users can now suggest new connectors through the UI.
- The new BigQuery connector is available in private beta. [Let us know](mailto:hi@tinybird.co) if you want to give it a try.

## Other notable fixes and improvements

- Changes in the Data Source schema when previewing a newly created Data Source were being ignored under some circumstances. It shouldn’t happen again.
- Fixed a problem with [datasources_ops_log](https://www.tinybird.co/docs/monitoring/service-datasources.html) not being updated after re-pushing a Materialized View.
- Fixed an issue in the Auth tokens modal that prevented the Read / Write options to be correctly seen.
- Fixed a couple of bugs in the Data Flow view related to wrong rendering of Data Sources metadata.
- Fixed a bug that apparently allowed the creation of tokens with no scopes but raised a silent error.
- When using `tb fmt` to format your .datasource file we had a bug that removed the Data Source description. Now, the description is kept.



Introducing the new Data Ingestion Experience


## CLI

- New guided process to create a workspace using the CLI. Also improved inline help for workspace creation. Go try ‘tb workspace create –help’
- We’ve fixed an issue with the ‘tb fmt’ CLI command to format queries. If your pipe had versioning, it was removing the VERSION 0 line on formatting.


## Data Sources and Data Sources UI

- It is now easier to inspect the columns of the [Service Data Sources](https://www.tinybird.co/docs/monitoring/service-datasources.html) when working from the Pipe editor or Time Series editor.
- We are now displaying the pulse graph also for materializations coming from Kafka Data Sources.
- Users won’t be offered the option to delete Shared data sources under the "Clean up" section.
- Now you can modify existing JSONPaths in data source columns, or add new columns with JSONPaths using the [data source alter API](https://www.tinybird.co/docs/api-reference/datasource-api.html#post--v0-datasources-(.+)-alter).
- Improved error messages when problems arise adding a column to the Data Source via the UI.
- The UUID type is now available to choose when defining a new Schema via the UI.


## And more!

- We’ve introduced 30 days option in the Time Series granularity selector. 
- Added support for arbitrary (non UTF-8) bytes in Kafka message keys when using Schema Registry. 


Notable fixes and improvements


## SOC 2 Type II

Tinybird has successfully completed its SOC 2 Type II audit made by [Ernst & Young Global Limited](https://www.ey.com/en_gl/consulting/service-organization-controls-reporting), affirming the effectiveness of our security processes and controls.

This is one of the many measures we are taking to ensure the continuous security, integrity and availability of the Tinybird platform. We will be audited again in early Q2 2023 to ensure our security procedures continue to evolve as we evolve our platform.

[Learn about security and compliance at Tinybird](https://trust.tinybird.co/).

## Notable fixes / improvements

- [Tinybird Docs](https://www.tinybird.co/docs) continue to improve! Revamped our [Introduction to Tinybird](https://www.tinybird.co/docs/index.html) along with a [Main Concepts](https://www.tinybird.co/docs/main-concepts.html) section, quickstarts for Tinybird’s [CLI](https://www.tinybird.co/docs/quick-start-cli.html) and [UI](https://www.tinybird.co/docs/quick-start-ui.html), a revamped structure for Guides as well as this new Changelog that you are reading :-)

- We’ve updated the editor keyboard shortcuts and fixed some problems when using the shortcuts to query data from a node. 

![Tinybird Shortcuts](/assets/changelog/shortcuts.jpeg)

- We have completely revamped how Data Replace jobs work internally, which bring improved stability and reliability to the process. 

- Added a new check that prevents the creation of a Materialized View that produces more than 10 partitions. This significantly reduces the risk of getting a TOO_MANY_PARTS exception at ingestion time when creating a Materialized View with a high cardinality partition key.

- Solved an issue that prevented deletion or renaming of Data Sources/Pipes with GUID format.



We are SOC 2 Type II compliant


## New Features

Tinybird’s BI Connector is a PostgreSQL-compatible interface to data in Tinybird.

All Data Sources and published Pipes created in Tinybird are available as standard Postgres tables when you connect a BI tool to Tinybird, such as Tableau, Apache SuperSet or Grafana.

Until recently, BI usage was not available to track. We have now made two new service datasources available that show stats about the BI Connector consumption: [bi_stats_rt](https://www.tinybird.co/docs/monitoring/service-datasources.html#tinybird-bi-stats-rt) (real time) and [bi_stats](https://www.tinybird.co/docs/monitoring/service-datasources.html#tinybird-bi-stats) (aggregated per day and per query). 

This information is also visible in a new information module in the dashboard for the BI Connector (only visible when active) and the Query API (visible to everyone).

## Notable fixes / improvements

- New guide with [best practices and limitations around Tinybird’s BI connector](https://www.tinybird.co/docs/guides/best-practices-bi-connector.html).
- Another guide about [how to ingest streaming data from Google Pub/Sub](https://www.tinybird.co/docs/guides/ingest-from-google-pubsub.html).
- Previously, when you were running a partial replace job you couldn’t tell from the response whether it was running a full replace or a partial one. We’ve added a `replace_condition` attribute in the job response. 
- Fixed a bug where clicking a data source in a Pipe’s node would cause the page to scroll and annoyingly move the user away from the node they were editing.
- We added supported [Clickhouse types](https://www.tinybird.co/docs/api-reference/datasource-api.html#create-from-schema) to the documentation to avoid surprises :-)


Observability for the BI connector


## New Features

For API endpoint requests as well as for calls to the Query API you can access the amount of read data and rows, duration of the API call and error information (if relevant) in the [pipe_stats_rt](https://www.tinybird.co/docs/monitoring/service-datasources.html#tinybird-pipe-stats-rt) Service Datasource, as well as what parameters were used on each request. Now, you can do the same thing to monitor ingest and materialization usage.

Read and written bytes and rows are now also available in the [datasources_ops_log](https://www.tinybird.co/docs/monitoring/service-datasources.html#tinybird-datasources-ops-log) Service Datasource for every ingestion operation, whether that’s new data ingested, materializations or replaces.

And lastly, it’s now possible to track your total storage in the new [datasources_storage Service Data Source](https://www.tinybird.co/docs/monitoring/service-datasources.html?highlight=service%20data%20source#tinybird-datasources-storage).

Remember that these service Data Sources can be queried from your Pipes so you can easily create API Endpoints or Time Series visualizations to analyze your own usage.

## Notable fixes / improvements

- The Kafka connector now supports ingestion of messages encoded with non UTF-8 message keys. Until now, we were assuming that everyone would use UTF-8 keys in their messages but… apparently not!
- Improved performance for ingestion of large JSON and Parquet files, specifically for files with a large number of columns.
- Also fixed schema detection issues when importing large Parquet files from a URL. Now, we parse the whole file and use the actual schema [defined in the footer](https://parquet.apache.org/docs/file-format/), instead of trying to infer the schema as we do with other schemaless formats. As a result of this change, the whole import process will be more accurate and predictable. 
- Added support for "bytes" type in Parquet files. 
- Certain situations were causing materialized views populate jobs to appear stuck. Various improvements to progress tracking.
- If the underlying Clickhouse process was restarted while a Populate job was waiting for a query to finish, it would cause the job to get stuck and never finish. Now solved.  
- `populateview` entries in the service datasource `tinybird.datasources_ops_log` were not being properly registered when pushing a pipe with `force=true`. They are now.
- When creating a Materialized View, the columns in the GROUP BY need to match those in the Sorting Key of the destination data source. Tinybird validates that this is the case when you create a materialized view, but the validation was being triggered incorrectly in some instances.
- Fixed an issue that was exposing some internal error information when some operations on the Events API failed. Now we return an error code and a generic "Internal Tinybird error" message without leaking any unnecessary internal information.
- Now, the CLI shows warnings with tips when uploading a materialization pipe. These warnings link to documentation where you can learn how to improve your queries. Right now, there are only a few tips, but we will be adding more.
- We've released a new homepage for [Tinybird’s Documentation](https://www.tinybird.co/docs). 


Increased observability capabilities in Service Data Sources


- The “pulse” graph, the chart that displays real-time ingestion stats for every data source, is now also available for Materialized Views. This helps get a sense of how many rows are being materialized at any given time.
- We’ve introduced brushing for date range selection in the Time Series UI. 
- Previously, when adding new columns to existing data sources in the UI, rows not containing them used to go into quarantine. Now, when new columns are added, the suggested type by default is ‘Nullable(String)’.
- We’ve fixed an issue where a "Data Source not found" state would prevent you from closing the right panel for the Data Source preview view.
- Also fixed an issue that was causing the Data Source preview to close when hitting escape within the code editor.
- Made all buttons highlight states consistent across the platform.

![Buttons highlight states](/assets/changelog/buttons-highlight-states.png)






## Events API

While Kafka remains a popular way to ingest events data, sometimes our users just want a super simple way to send an events stream to Tinybird. And there’s nothing simpler than sending events through HTTPS via our events API. Just copy the snippet, run the script, and see your data instantly show up in the workspace for you to start querying.

## CLI updates

One of our core beliefs at Tinybird is that you should be able to build with data like you build with code. And that means being able to run tests. We’re excited to announce our first few feature launches to enable testing in Tinybird data projects. In particular, the CLI has new commands and plus some added functionality, including:

- `tb test` to add tests to files
- New metrics about endpoint response times (max, min, mean, median and p90) on the pipe command `tb pipe regression-test`
- `tb workspace clear` now deletes all files in the workspace. Given that this command drops all the resources inside a project, please use it with care!
- `tb pipe publish` to change which node of the Pipe is published as an endpoint.
- `tb check` to verify query syntax.You can find all the latest changes in the [command-line updates](https://pypi.org/project/tinybird-cli/).

## ClickHouse

Our [work](https://github.com/ClickHouse/ClickHouse/pull/37187) to allow decimal values in seconds (for example, `max_execution_time=0.5`) was merged.


Quickstart on loading events to Tinybird


## Data Source descriptions

In a large workspace that contains many Data Sources, you may want more information than just the name of the Data Source. Documentation matters. Now you can add a description to a Data Source like you already do with pipes, nodes and endpoints.

This feature is available through the UI and the CLI. Any new descriptions will propagate to shared sources.

## Early beta support for Parquet files

At Tinybird we aim to capture and transform large amounts of data whatever the origin of the data or format. In addition to CSV and NDJSON, we're working on accepting Parquet format files. Parquet is an open-source, column-oriented data file format designed for efficient data storage and retrieval. It is also commonly used as an interchange format between data tools.

Our team is now testing ingesting data to Tinybird from Parquet. After further testing, this new format for ingestion will be included in our docs. If you'd like you join the beta reach out to us on [Slack](https://join.slack.com/t/tinybird-community/shared_invite/zt-yi4hb0ht-IXn9iVuewXIs3QXVqKS~NQ)

## CLI updates!

**tb push --subset** - We've added the `tb push --subset` option to be used with `--populate` so you can populate using only a subset of the data of between 0% and 10%. Now you can quickly validate a materialized view with just a subset of your total dataset. You can check everything is working with a single month’s worth of data even if you have several years’ worth of data.

**Data Source description** - We've added the option of adding a description for a Data Source, as you already could for pipes, thereby improving the documentation of your data project.

**Endpoints from materialized Data Sources** - We've fixed code so that nodes whose type is materialized can no longer be published as endpoints. Logically the endpoint should depend on the target Data Source of the materialized node, not the node itself.

Check out the latest [command-line updates](https://pypi.org/project/tinybird-cli/) in the changelog.

## ClickHouse improvements

[**groupSortedArray**](https://github.com/ClickHouse/ClickHouse/pull/34055) - This new aggregation function was added to ClickHouse. `groupSortedArray(n)(param1)` returns an array with the n first values from a field param1 sorted by itself. `groupSortedArray(n)(param1, param2)` returns an array with the n first values from a field param1 sorted by param2 (field or expression). This aggregation function is useful if, for example, you have a materialized view with the two most recent values in a field.

**ASOF join** - The [ASOF join performance improvement](https://github.com/ClickHouse/ClickHouse/pull/35525) was included in Tinybird. This join was improved by the ClickHouse community to be twice as fast.
