// READING
← back to inbox

Tell me everything that could go wrong

Richard Lee<richard-y-lee@hotmail.com>·
// Essays// 3 min read
// 01

During a job interview I was asked "you go to amazon.com in your browser and you get the dinosaur game instead. What's wrong?" I asked a few clarifying questions, like if I'm at home or I work for Amazon ...

“It's up to you”

// 02

The standard answer

the wifi is down
the router needs a restart
I try on my hotspot
— okay. what if it's something else?

// a solid three minutes

I gave the answer you're supposed to give. Check the wifi, restart the router, find a RJ-45 and connect to the modem. And a few other scenarios for about three minutes.

“What if it's something else?”

// 03

Okay...

the ISP is having an outage
the DNS server itself is down
check downdetector
— hmm.

// two more minutes

The ISP. The DNS. I'll check Twitter and downdetector. I believe I also said by this point I'd have given up and head to Costco. Beside this interview happened almost 10 years ago and Amazon's famous broke-the-internet outages hadn't even happened yet. Even after all of this I could tell he still wasn't satisfied.

“Tell me everything
that could go wrong.”

// 05

The hour

// fine, I work at amazon — what changed?
the logs, read backwards from the first 5xx
the monitors — was anything red before I looked?
the last release, and whatever shipped quietly with it
the change notification nobody read
// the stack, layers 2–7
layer 2 — a switch port flapping in one rack
layer 3 — a route that quietly stopped announcing
layer 4 — TCP handshakes timing out halfway
layers 5–7 — TLS sessions up through HTTP itself
// the front door
Route 53 — is DNS even answering?
an expired cert · TLS terminating at the wrong hop
ALB health checks failing, pulling targets one by one
CloudFront serving the outage from cache
the failover that never fired

// the list doesn't end — the hour did

Then I remembered the “up to you.” So now I work at Amazon, with full access across every system. We spent the better part of an hour in there, and every time my list ran out he'd just wait. Somewhere between layer 2 and layer 7, I realized the list wasn't supposed to run out.

// 06

I was being tested where I'd break

Every answer I gave stopped at the same place: the edge of what I thought was enough because past that it was another department's responsibility. Networking, Security, Engineering; real teams, real people whose role was the part I was not an expert in. That's what an abstraction is, permission to stop understanding. They wanted to know if I'd refuse to cross borders.

// 07

The same happens everywhere

We use tools and libraries and trust the implmentation. We lean on the next team over to own their piece of the pie. Most days that's the only way anything big gets built. Nobody gets asked the question out loud until the library breaks, or the other team is out.

// 08

Somebody ends up holding all of it

frontendbackendmobiledataqasecuritythe one who knows how it all fits

// not a job title

But the leaning has to land somewhere. Every team has one or two people who kept going past their edge. They know the system, the product, sometimes the whole company and everyone else's somebody-else's-responsibility quietly piles up on them. The rest get to stay comfortable because they don't.

// 09

The power is in the more you know

I still don't understand every layer of everything I use. That's impossible, nobody does. But I found the best strategy is to stop treating the edges as walls. There's no limit on what you're allowed to understand, and every person who refuses the permission takes a little of that weight back.

// 10

Or maybe the answer was to play the game.