Infra to work with billions of rows. Tooling to make it feel easy

https://api.tinybird.co/v0/pipes/total_views.json?start_date=2024-01-01&end_date=2024-12-31&token=<JWT_TOKEN>

>> curl \
-X POST 'https://api.tinybird.co/v0/events?name=analytics_events' \
-H "Authorization: Bearer $TB_TOKEN" \
| jq .
{
"timestamp": "2022-10-27T11:43:02.099Z",
"transaction_id": "8d1e1533-6071-4b10-9cda-b8429c1c7a67",
"name": "BobbyDrake",
"email": "bobby.drake@pressure.io",
"age": 42,
"passport_number": 3847665,
"flight_from": "Barcelona",
"flight_to": "London",
"extra_bags": 1,
"flight_class": "economy",
"priority_boarding": false,
"meal_choice": "vegetarian",
"seat_number": "15D",
"airline": "Red Balloon"
}

>> tb infra add
Running against Tinybird Cloud: Workspace example_workspace
Enter name: example
Enter host: https://tinybird.example.com
» Adding infrastructure 'example' in Tinybird...
✓ Infrastructure 'example' added
» Required environment variables:
TB_INFRA_TOKEN=example_token
TB_INFRA_WORKSPACE=example_workspace
TB_INFRA_ORGANIZATION=example_organization
TB_INFRA_USER=user@example.com

# Run Tinybird locally with our Docker image,
# and deploy your APIs on localhost.
# Validate your entire build on your machine.

>> curl https://tinybird.co | sh
Installing Tinybird CLI...

Make sure to add `/Users/user_name/.local/bin` to your PATH. To use installed tools, run `export PATH="/Users/user_name/.local/bin:$PATH"` or `uv tool update-shell`.

Tinybird local has been successfully installed, start using it with `tb login` and create a new project with `tb create`.
>> export PATH="/Users/user_name/.local/bin:$PATH"
>> tb login
» Opening browser for authentication...

Workspace: worksp01
User: user_name@your_email_domain.com
Host: https://api.europe-west2.gcp.tinybird.co

✓ Authentication successful!
>> tb local start
» Starting Tinybird Local...
* Waiting for Tinybird Local to be ready...
✓ Tinybird Local is ready!

tinybird/ |- datasources/ | |- events.datasource | |- metadata.datasource |- endpoints/ | |- top_users.pipe | |- latest_data.pipe | |- fastest_api_ever.pipe |- materializations/ | |- metadata_latest_mv.pipe |- fixtures/ | |- mock_events.ndjson |- tests/ | |- top_users.yaml | |- latest_data.yaml | |- fastest_api_ever.yaml |- README.md |- .cursorrules

# Live schema migration

SCHEMA >
`timestamp` DateTime,
`user_id` String,
`user_id` UUID,
`event` String,
+++`payload` JSON

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYear(date)"
---ENGINE_SORTING_KEY "date, user_id, event, extra_data"
+++ENGINE_SORTING_KEY "user_id, date, event, extra_data"
ENGINE_TTL "date + toIntervalDay(90)"

# Query service data source

SELECT *
FROM tinybird.jobs_log
WHERE job_id = {{String(job_id)}}

# OTLP configuration

receivers:
otlp:
protocols:
grpc:
http:

processors:
batch:
timeout: 10s

exporters:
tinybird:
endpoint: ${OTEL_TINYBIRD_API_HOST}
token: ${OTEL_TINYBIRD_TOKEN}

service:
pipelines:
metrics:
receivers: [otlp]
processors: [batch]
exporters: [tinybird]
traces:
receivers: [otlp]
processors: [batch]
exporters: [tinybird]

# Vibe code

>> tb create --prompt "I'm building a logs explorer. Create a data source to store my logs payload sorted by timestamp and an API endpoint to return the percentage of error logs over a supplied time range."

>> tb mock --rows 1000 --prompt "Create logs with exactly 5% errors using timestamps within the last 7 days"

>> tb test create error_pct --prompt "Make sure there are 5% errors"

>> tb --cloud deploy

>> curl \
-G '$TB_HOST/v0/pipes/error_pct.json' \
-H 'Authorization: Bearer $TB_TOKEN' \
-d 'start_date=2025-01-01' \
-d 'end_date=2025-02-01'