Ingest from Google Pub/Sub¶
Intermediate
In this guide you'll learn how to send data from Google Pub/Sub to Tinybird.
Introduction¶
Tinybird is a Google Cloud partner & we are adding deeper integrations with Google Cloud services all the time.
Google Pub/Sub is often used as a messaging middleware that decouples event stream sources from the end destination. Pub/Sub streams are often consumed by Google's DataFlow which can send events on to destinations such as BigQuery, BigTable or Google Cloud Storage.
This DataFlow pattern works with Tinybird too, however, Pub/Sub also has a feature called Push subscriptions which can forward messages directly from Pub/Sub to Tinybird.
Push Messages From Pub/Sub To Tinybird¶
In the following section we are going to show you how to integrate Pub/Sub with Tinybird using push subscriptions.
Step 1. Create a Pub/Sub topic¶
Start by creating a topic in Google Pub/Sub. You can find the detailed documentation from Google here.
Step 2. Create a push subscription¶
Next, create a Push subscription in Pub/Sub.
The Delivery Type should be set to Push & set the Endpoint URL field to use the Tinybird Events API.
You must include the token
parameter in your Endpoint URL with a token copied from your Workspace.
https://api.tinybird.co/v0/events?name=events_demo&token=<your auth token>``
You don't need to create the Data Source in advance, it will automatically be created for you.
Step 3. Send sample messages¶
Generate and send some sample messages, using this script for example.
Step 4. Decode message data¶
You'll start to see message arrive in the Data Source in Tinybird.
By default, a Pub/Sub Push subscription encodes the message body in base64.
To decode the message body, you can use the base64Decode() function. You can then start to extract values from the JSON.
Below is an example Pipe that decodes the message body, extracts values from the JSON message and writes to a Materialized View.
And that's it!