Real‑time Data Crash Course

Chapter 1

Let's talk (just a bit) about modern hardware

Understand modern storage and compute for real-time, big data. Focus on latency vs. throughput and CPU vs. memory speed for faster sequential access.

Chapter 2

Choose the right database

Choose the right OLAP database. Learn the power of columnar storage over row-based for superior compression and vectorization.

Chapter 3

Master data sourcing and ingestion

Data ingestion essentials: why structured data is faster, master batching data, using open formats, and distinguishing event vs. dimension tables.

Chapter 4

Store information efficiently

Optimize storage with a tight schema. Leverage compression and sort data by access pattern for efficient range-based indexes.

Chapter 5

Write effective queries

Learn the fundamentals of good SQL. Optimize query performance and parallelization. Query less data (filters/indexes) and use efficient, filtered joins.

Chapter 6

Take advantage of views

Learn to use materialized views to pre-aggregate data, saving CPU and money. Learn to use incremental views for faster real-time updates.

Chapter 7

Productize your data

Expose data via a scalable, tested API. Ensure quality with data quality and regression tests. Learn top benchmark performance using statistical concurrency models.

Chapter 8

Scale your data infrastructure

Learn how to scale your cluster. Vertical first, then horizontal. Use sharding for query speed and replication for redundancy. Implement proper load balancing.