Ingest from Google Pub/Sub

Intermediate

In this guide you'll learn how to send data from Google Pub/Sub to Tinybird.

Introduction

Tinybird is a Google Cloud partner & we are adding deeper integrations with Google Cloud services all the time.

Google Pub/Sub is often used as a messaging middleware that decouples event stream sources from the end destination. Pub/Sub streams are often consumed by Google's DataFlow which can send events on to destinations such as BigQuery, BigTable or Google Cloud Storage.

This DataFlow pattern works with Tinybird too, however, Pub/Sub also has a feature called Push subscriptions which can forward messages directly from Pub/Sub to Tinybird.

Push Messages From Pub/Sub To Tinybird

In the following section we are going to show you how to integrate Pub/Sub with Tinybird using push subscriptions.

Step 1. Create a Pub/Sub topic

Start by creating a topic in Google Pub/Sub. You can find the detailed documentation from Google here.

Create a topic in Google Pub/Sub

Step 2. Create a push subscription

Next, create a Push subscription in Pub/Sub.

The Delivery Type should be set to Push & set the Endpoint URL field to use the Tinybird Events API.

You must include the token parameter in your Endpoint URL with a token copied from your Workspace.

https://api.tinybird.co/v0/events?name=events_demo&token=<your auth token>``

You don't need to create the Data Source in advance, it will automatically be created for you.

Add Push subscription to Pub/Sub topic

Step 3. Send sample messages

Generate and send some sample messages, using this script for example.

Pub/Sub dashboard with metrics

Step 4. Decode message data

You'll start to see message arrive in the Data Source in Tinybird.

By default, a Pub/Sub Push subscription encodes the message body in base64.

Raw Pub/Sub messages

To decode the message body, you can use the base64Decode() function. You can then start to extract values from the JSON.

Below is an example Pipe that decodes the message body, extracts values from the JSON message and writes to a Materialized View.

Materialized view with denormalizaiton

And that's it!