May 02, 2021

Analyzing our own Nginx logs with ClickHouse on Tinybird

How we analyze our own logs in real-time using our own product.
Xoel López
Founder at TheirStack

In a recent post we walked through the steps required to use Tinybird to analyze Nginx logs. In this one we’ll share how we are using our own product internally to analyze our own Nginx traffic in real-time and automatically, and how you can do it too.

To follow along, you’ll need 3 things:

  • A Tinybird account. Sign in if you don’t have one yet.
  • The code from this GitHub repo: it contains a Tinybird project Data source configurations and Pipes that will work out of the box.
  • tbtail, a tool to stream your logs to Tinybird, installed in the machines where your Nginx logs are generated.

With that, you can quickly get a sense of what’s going on with all your traffic in real-time, as well as examples for you to start slicing and dicing your logs in however other ways you want.

We’re still in private beta, so after you sign in with Google, Github or Microsoft authentication, your account won’t be active. Email us at hi@tinybird.co and we’ll activate it for you.

Run these commands to clone the data project locally, replicate it on Tinybird and have everything set and ready for tbtail to start sending logs to your account:

The last thing is installing tbtail. The details on how to do it are in its repo. We published deb and bin packages, and you can also compile it from source.

After it’s installed, you just need to run this command in the machine where the Nginx logs are created to send logs to Tinybird automatically.

Then, if you go to your dashboard, you’ll see a new pipe called query_grouped_requests. It defines an endpoint that lets you query your aggregated data. In the Pipe, you can click on the green “View API” button on the top right corner of the page to see its live documentation.

Do you like this post?

Related posts

More Data, More Apps: Improving data ingestion in Tinybird
Low-code analytics with James Devonport of UserLoop
Tinybird
Team
May 05, 2023
Modern data management with real-time Change Data Capture
You can now explore and analyze time series data in Tinybird
To the limits of SQL... and beyond
Export data from Tinybird to Amazon S3 with the S3 Sink
Tinybird
Team
Mar 21, 2024
8 example projects to master real-time data engineering
Killing the ProcessPoolExecutor
Tinybird
Team
Aug 09, 2023
The Data Journey: Unlocking data for the right now
Tinybird is SOC 2 Type II compliant
Tinybird
Team
Dec 19, 2022

Build fast data products, faster.

Try Tinybird and bring your data sources together and enable engineers to build with data in minutes. No credit card required, free to get started.
Need more? Contact sales for Enterprise support.