Sep 20, 2024

Introducing the Tinybird DynamoDB Connector in public beta

Today we announce the Tinybird DynamoDB Connector in public beta, enabling those who use the popular AWS database to quickly ingest and sync their DynamoDB tables to Tinybird. The DynamoDB Connector is available in all Workspaces in AWS regions.
Cameron Archer
Product Marketing Manager

Today, we release the Tinybird DynamoDB Connector in public beta. The DynamoDB Connector is a fully managed data source integration for importing data from Amazon DynamoDB to Tinybird. With the DynamoDB Connector, developers can quickly sync data from the popular AWS database into Tinybird, which is better suited for real-time, OLAP workloads.

The DynamoDB Connector is now available in public beta on all plans across all supported AWS regions. For detailed instructions on how to use the DynamoDB Connector, please refer to the documentation.

For more information on how the DynamoDB Connector works, keep reading or watch the screencast below.

Connecting DynamoDB to Tinybird

The Tinybird DynamoDB Connector gives developers using Amazon's DynamoDB a fully-managed solution to move real-time analytics use cases off of DynamoDB and onto Tinybird without writing any additional code or maintaining any external ETLs.

While DynamoDB can be a powerful real-time database for the right type of workload, it can struggle under more intense analytical workloads. Customers like Canva have transitioned analytics previously run on DynamoDB to Tinybird for better performance, and the DynamoDB Connector now makes this transition much simpler.

Canva captured telemetry into DynamoDB in an attempt to build user-facing analytics, but the approach failed because DynamoDB struggled to handle Canva's aggregations, and Canva found it complex to design the right schemas. They'd spend several months in the design phase trying to find the optimal schema and couldn't afford to redesign these schemas anytime they wanted to evolve or explore new use cases.

Read the full story

Backfills and streaming upserts

The DynamoDB Connector syncs DynamoDB to Tinybird in two parts: 1) an initial backfill, and 2) continuous streaming updates. Tinybird transparently manages both processes.

For initial backfills, Tinybird uses DynamoDB's Point-in-time Recovery (PITR) feature to export a snapshot of the DynamoDB instance to files in Amazon S3, which are ingested into the Tinybird Data Source when the connection is first created.

To stay in sync with changes to DynamoDB tables, Tinybird automatically sets up and subscribes to DynamoDB Streams, using a Change Data Capture (CDC) approach to handle upserts and maintain a stateful copy of your DynamoDB tables in Tinybird.

Latency for this continuous sync is no more than 5 seconds under normal loads.

Security through AWS IAM Roles and Policies

As with Tinybird's Amazon S3 Connector, the DynamoDB Connector uses the AWS Identity and Access Management (IAM) service to securely manage permissions between Tinybird and DynamoDB.

When you create DynamoDB connection, Tinybird will automatically generate the JSON snippets needed to create the IAM Role and Access Policy that allow Tinybird to read from your DynamoDB tables. All you need to do is add your DynamoDB table name and S3 bucket, then copy and paste.

Out-of-the-box ingestion observability

As with every Tinybird Data Source Connector, observability for the DynamoDB Connector is available out of the box using the datasource_ops_log Service Data Source. All write operations from DynamoDB to Tinybird are logged in this automatically generated Data Source, and you can query and analyze the logs free of charge to build observability APIs and dashboards.

Pricing for the DynamoDB Connector

Pricing for the DynamoDB Connector is based on data processing and data storage.

The initial write operation to backfill data and the continuous CDC-based syncs will incur write processing costs at the standard $0.07/GB.

The data stored in your DynamoDB Data Source will incur a monthly charge of $0.34/GB compressed. The precise amount of compression will depend on your schema, but the typical compression ratio for Tinybird Data Sources varies between 3.5x and 10x.

In addition, you'll need to consider the costs incurred on AWS, specifically for any storage and egress costs to handle snapshotting and continuous syncs. These will depend on your data size, region, etc.

A plug-and-play transition for Rockset customers

Real-time database Rockset recently announced its acquisition by OpenAI and its intention to sunset the service by the end of the month. Many Rockset customers used Rockset's DynamoDB connector to quickly ingest tables from DynamoDB to Rockset.

The Tinybird DynamoDB Connector provides an experience familiar to former Rockset customers, enabling a seamless transition to Tinybird's real-time data platform.

For more information about migrating from Rockset to Tinybird, check out the migration guide.

Using the DynamoDB Connector

Connecting Tinybird to DynamoDB is straightforward. First, create a new Data Source and select the DynamoDB Connector.

A screenshot of the the Tinybird UI show data source connectors to many different data tools including DynamoDB
You can create a DynamoDB Data Source in the Tinybird UI

Then, follow the guide to create the Access Policy and Role in the AWS IAM Service. Tinybird will automatically generate the JSON snippets for each. All you need to do is copy and paste into the AWS IAM console and supply the DynamoDB table name and S3 bucket you want to use for backfill.

A screenshot of JSON policies to connect Tinybird to DynamoDB using IAM
Tinybird automatically generates IAM policies to authorized Tinybird to read from DynamoDB.

Once you've established a secure connection between Tinybird and your DynamoDB instance, choose the DynamoDB table and an S3 bucket to support the snapshot and initial load. Tinybird will analyze the DynamoDB table and generate a preview with inferred schema. You can modify the schema as necessary before creating the Data Source.

A screenshot of DynamoDB data in a Tinybird table
Tinybird will analyze the data in your DynamoDB tables and automatically generate schema. You can adjust the schema as necessary to fit your needs.

Once the Data Source has been created, Tinybird will perform an initial load, backfilling existing data from DynamoDB into the Tinybird Data Source. Once the initial load is complete, Tinybird will automatically sync changes from the DynamoDB table into Tinybird, transparently handling upserts to maintain an accurate state.

A screenshot of a Tinybird DynamoDB Data Source table
DynamoDB ingestion happens in both backfill and streaming upsert. All write operations are logged in a Service Data Source for observability.

You can also use the Tinybird CLI to set up a connection to DynamoDB. Simply use the command below and follow the guided prompts to create the connection:

Get started with the DynamoDB Connector

The DynamoDB Connector is now available in public beta for Workspaces in all supported AWS regions. To get started, create a new Data Source and follow the guided steps to sync your DynamoDB tables with Tinybird.

For more details about the DynamoDB Connector, refer to the documentation.

As always, if you have any questions, please join our Slack Community.

New to Tinybird?

If you're currently working with DynamoDB and looking for a real-time analytics database to scale your performance, try Tinybird. The Build plan is free, with no credit card requirement and no time limit. Sign up today to unify data from DynamoDB and other sources, shape it with SQL, and publish your queries as scalable REST API Endpoints that you can integrate into your application.

And if you're coming from Rockset, check out our Rockset Migration Guide for more information on making the transition from your existing Rockset environment to Tinybird's purpose-built real-time data platform.

Do you like this post?

Related posts

3 ways to run real-time analytics on AWS with DynamoDB
Tinybird expands self-service real-time analytics to AWS
Tinybird has joined the AWS ISV Accelerate Program

Tinybird

Team

Jun 05, 2024
Simplifying event sourcing with scheduled data snapshots in Tinybird
Run analytics on files in Amazon S3 the easy way

Tinybird

Team

Jul 20, 2023
Tinybird connects with Confluent for real-time streaming analytics at scale

Tinybird

Team

Jul 18, 2023
Migrating from Rockset? See how Tinybird features compare
Low-latency APIs over your BigQuery datasets
Changelog: New API endpoints pages for easier integration, upgraded version of ClickHouse and more
Building an enterprise-grade real-time analytics platform

Build fast data products, faster.

Try Tinybird and bring your data sources together and enable engineers to build with data in minutes. No credit card required, free to get started.
Need more? Contact sales for Enterprise support.