May 02, 2021

Analyzing our own Nginx logs with ClickHouse on Tinybird

How we analyze our own logs in real-time using our own product.
Xoel LĂłpez
Founder at TheirStack

In a recent post we walked through the steps required to use Tinybird to analyze Nginx logs. In this one we’ll share how we are using our own product internally to analyze our own Nginx traffic in real-time and automatically, and how you can do it too.

To follow along, you’ll need 3 things:

  • A Tinybird account. Sign in if you don’t have one yet.
  • The code from this GitHub repo: it contains a Tinybird project Data source configurations and Pipes that will work out of the box.
  • tbtail, a tool to stream your logs to Tinybird, installed in the machines where your Nginx logs are generated.

With that, you can quickly get a sense of what’s going on with all your traffic in real-time, as well as examples for you to start slicing and dicing your logs in however other ways you want.

We’re still in private beta, so after you sign in with Google, Github or Microsoft authentication, your account won’t be active. Email us at hi@tinybird.co and we’ll activate it for you.

Run these commands to clone the data project locally, replicate it on Tinybird and have everything set and ready for tbtail to start sending logs to your account:

The last thing is installing tbtail. The details on how to do it are in its repo. We published deb and bin packages, and you can also compile it from source.

After it’s installed, you just need to run this command in the machine where the Nginx logs are created to send logs to Tinybird automatically.

Then, if you go to your dashboard, you’ll see a new pipe called query_grouped_requests. It defines an endpoint that lets you query your aggregated data. In the Pipe, you can click on the green “View API” button on the top right corner of the page to see its live documentation.

Do you like this post?

Related posts

Publish SQL-based endpoints on NGINX log analysis
A new dashboard for Tinybird Analytics
Operational Analytics in Real Time with Tinybird and Retool
Improved Support for Replacing or Appending Data
More Data, More Apps: Improving data ingestion in Tinybird
Jobs log: Improved o11y for background jobs
Changelog #21: Quickstart on loading events to Tinybird
Tinybird connects with Confluent for real-time streaming analytics at scale

Tinybird

Team

Jul 18, 2023
Gain Visibility Over Your Asynchronous Operations
Changelog: BigQuery connector, CLI improvements, Pagination on endpoints and more

Tinybird

Team

Mar 04, 2021

Build fast data products, faster.

Try Tinybird and bring your data sources together and enable engineers to build with data in minutes. No credit card required, free to get started.
Need more? Contact sales for Enterprise support.