Designing for millions: lessons from scaling a real-time API

The morning it happened, I was making coffee. A small tool I’d built to broadcast live scores had been picked up by a popular account overnight, and by 9 a.m. it was fielding more traffic in a minute than it had seen in its entire life. Nothing was technically down; it was just slow in a way that felt personal. That day taught me more about systems than a year of reading ever had.

The thing nobody tells you about scale is that it doesn’t arrive as a single problem. It arrives as a sequence of small, smug assumptions failing one after another, each one perfectly reasonable until the moment it isn’t.

The first wall is the database, not the app

I had spent my optimization energy in the wrong place. The application code was fine; Go was barely breaking a sweat. The bottleneck was a single unindexed column being asked the same question tens of thousands of times a second. The fix took thirty seconds. Finding it took three hours, because I’d never learned to read an execution plan under pressure.

If you take one thing from this post: before you reach for another replica or a bigger box, open the query analyzer. The database almost always knows exactly what’s wrong, and it will tell you for free.

Cache like you actually mean it

The naive cache makes everything faster right up until a popular key expires and a thousand requests stampede the database at once. The fix is to collapse those duplicate requests into a single in-flight load, which is what Go’s singleflight package does in about a dozen lines:

// Get returns a value, collapsing concurrent misses
// for the same key into one trip to the database.
func Get(ctx context.Context, key string) ([]byte, error) {
    if v, ok := cache.Load(key); ok {
        return v.([]byte), nil
    }
    v, err, _ := group.Do(key, func() (any, error) {
        return loadFromDB(ctx, key)
    })
    if err != nil {
        return nil, err
    }
    cache.Store(key, v)
    return v.([]byte), nil
}

The goal of a cache isn’t to be fast. It’s to protect the thing behind it from ever being asked the same question twice.

Backpressure is a feature, not an apology

My instinct was to accept every request and try heroically to serve it. That’s how you turn a slow service into a dead one. The healthier posture is to decide, early and explicitly, what you will refuse. A request rejected in two milliseconds with a clear 429 is a far kinder answer than one that hangs for thirty seconds and then times out anyway.

Once I started treating rejection as a first-class outcome, the whole system got calmer. A few things made the biggest difference:

• A bounded queue in front of expensive work, so load sheds predictably instead of catastrophically.
• Per-client rate limits, so one enthusiastic user can’t starve everyone else.
• Timeouts on every outbound call, because a service is only as reliable as its slowest dependency.
• Health checks that fail fast, so the load balancer routes around trouble before users feel it.

What I’d never do again

I’d never again confuse “it works on my machine” with “it works.” Local development is a single user with infinite patience and zero latency: the three conditions least like production. The bugs that matter live in the gap between them, and the only way to find them is to put real load on the thing before the internet does it for you.

The little scoreboard still runs today, quietly, on hardware that costs less than lunch. It’s not impressive to look at. But it stays up when it’s busy, and after that one chaotic morning, I’ve learned not to take that for granted.