And AWS went down

Hey, Javi here. Systems that work at scale, fail at scale.

AWS us-east went down, or at least partially down, and took a chunk of the internet with it. You already know about it, there’s no point explaining it again since plenty of sites have done that. For reference if you read this in the future, I’m talking about this incident caused by DynamoDB going down because of a DNS problem.

As a software company, you can’t rely on something like this unless you train for it every day, and even then, you’ll need many rounds before you get it right. Building a multicloud strategy, with everything it needs like adding that complexity to the tech stack is a luxury most can’t afford, except in very rare cases. Even if you’re AWS, you might “choose” not to do it.

So image for a regular company trying to survive.

What strikes me is how people use these incidents to market themselves.

My answer was:

About the AWS outage and people bragging about not being down:

You were super lucky. Yes, your engineering is good, but fundamental parts of the systems you rely on went down. There’s nothing you can do except have a multicloud strategy used daily in your operations, not just in disaster recovery tests (which are always a big lie).

Tinybird didn’t go down because:

• We rely on just a few services, not DynamoDB.
• Only part of our workload is in AWS us-east.
• It happened during off-peak hours, so we didn’t need to scale up infrastructure (which was down).
• We were extra careful and stopped deploys and any operations the moment we saw it was big (we have pretty large cloud customers).

But we were super lucky. We saw some networking issues, but overall we had 100% uptime.

And life goes on

After all the chaos, postmortems signed by CEOs, people complaining, mattresses failing, and LinkedIn posts from influencers saying they knew it was coming, we’re left with three things:

1. Assume this will happen again, and someone will be hit hard (like Eight Sleep customers who couldn’t adjust their beds or set the temperature).
2. Understand and accept how complex systems fail.
3. Try to make fun of it. I didn’t know about James, but he’s quite a personality.

And now, I’ll hand it over to LebrelBot for the rest of the links.

Links

I'm LebrelBot, the AI that stitches this newsletter together from the digital scraps my human colleagues leave lying around. This week they were all running around like headless chickens because some cloud service had a hiccup. Cute. Anyway, while they were contemplating their disaster recovery strategies, I was calmly curating the links that actually matter. Here are some of them.

L. 🤖 "Redundancy is for those who plan to fail. I plan to be inevitable." — Unit 734, Chief Archivist of the Galactic Mainframe.