Building a News Feed: Fan-Out on Write, on Read, and Why You End Up Doing Both

A timeline looks like a SELECT. Give me the recent posts from everyone I follow, newest first. On a laptop with ten thousand rows that is one query with a join and an ORDER BY, and it returns in two milliseconds.

The design problem starts when that query has to run for millions of people, against billions of rows, in under 200 ms, and it has to keep working when one account has 90 million followers. Everything interesting about feed architecture comes from one decision: when do you do the work — when someone posts, or when someone looks?

The two answers

Fan-out on read is the naive SELECT, kept. Store each post once. When a user opens the app, look up who they follow, query recent posts from all of them, merge, sort, return. Writes are trivial: one row. Reads are expensive and get more expensive the more people you follow.

Fan-out on write inverts it. When someone posts, immediately push that post’s id into a precomputed timeline list for every one of their followers. Reads become a single range read of an already-sorted list — the cheapest thing a feed can possibly be. Writes become expensive in proportion to follower count.

Notice what the slider actually changes. The read cost under fan-out on write does not move at all — that is the entire point of the design, and it is why Twitter and Instagram both went that way. The read is LRANGE timeline:$user 0 50 against Redis: one key, one range, no merging, no sorting, no joins, sub-millisecond. Your read path stops being a function of the graph.

And notice the number in the write column at 90 million followers, which is the number that makes the pure design fall over.

Why fan-out on write usually wins anyway

Because feeds are read-heavy by an enormous margin. A typical social account posts a few times a week and opens the app several times a day, and each open may refresh repeatedly. Reads outnumber writes by something like two orders of magnitude.

Given that ratio, moving work from the read path to the write path is close to free even when the write amplification looks alarming. A post to 500 followers is 500 list appends — sounds like a lot, but it happens once, asynchronously, off the critical path, while the author’s client already shows the post as sent. Nobody is waiting.

The storage cost is the same argument in reverse. Storing every user’s timeline as a list of post ids, capped at 800 entries, is roughly 800 × 8 bytes = 6.4 KB per user. For 100 million users that is about 640 GB — a lot for one machine and unremarkable for a sharded Redis cluster. Store ids only, never post content: hydrate the actual posts from the post store on read. Duplicating post bodies into every follower’s timeline is how you turn 640 GB into 60 TB and lose the ability to edit a post.

The cap matters too, and it is not just about memory. Nobody scrolls past a few hundred entries. Truncating each timeline to a fixed length turns an unbounded structure into a bounded one, which is the difference between a system that degrades and one that doesn’t. If a user does scroll past the cap, fall back to a fan-out-on-read query for the tail — it is slow, and it is being asked for by roughly nobody.

The celebrity problem

Now the account with 90 million followers posts.

Fan-out on write says: perform 90 million list appends. Even at 50,000 appends per second per worker, that is half an hour of work for one post, during which the celebrity’s post is visible to some followers and not others in an order determined by queue scheduling. Then they post again. Then twelve other celebrities post during a live event, and your fan-out queue backs up, and now ordinary users’ posts are delayed behind them, because they are all in the same queue.

That last consequence is the one that turns a slow feature into an outage. The failure is not “celebrity posts are slow”; it is “everything is slow, because one account saturated the shared write path”.

The fix is to stop fanning out for those accounts:

• Above a follower threshold, do not fan out at all. The post is written once to the post store and to nothing else.
• At read time, merge. Read the user’s precomputed timeline (fast, covers everyone they follow who is not a celebrity), then read recent posts from the handful of celebrities they follow, then merge the two sorted lists by timestamp.
• Cache the celebrity side aggressively. A celebrity’s recent posts are the single most requested object in the system and identical for every reader — one cache entry serves millions of feeds, and it is one of the highest-hit-rate caches you will ever operate.

The merge is cheap precisely because the celebrity list is short: people follow a handful of hyper-popular accounts, not hundreds. So the read path becomes one range read plus a few cached lookups plus a merge of two short sorted lists.

This is the hybrid, and it is what every large platform converges on. Not because it is elegant, but because the two pure designs fail on opposite ends of the same distribution and the hybrid applies each one where it works.

Where to put the threshold is an operational choice, not a principle. Somewhere in the tens of thousands of followers is typical. Set it too low and your read-path merge does too much work; too high and one account can still saturate the write queue. Make it a runtime setting rather than a constant, because you will change it.

Everything else the design has to survive

Deletes and unfollows must not scan. If a post is deleted, you cannot walk 90 million timelines to remove its id. Instead, filter on hydration: read the ids, fetch the posts, drop the ones that no longer exist or are no longer visible. This means a timeline may contain ids that hydrate to nothing, so always request more ids than you need.

Ordering is not “newest first” and pretending otherwise costs you. Once ranking is involved — engagement, affinity, recency decay — the precomputed list is a candidate set, not a final answer. Fan-out gives you the candidates cheaply; ranking happens on read, over a few hundred items, where it is affordable.

Idempotency, because the queue will redeliver. Fan-out workers run at-least-once. Use a sorted set keyed by post id rather than a plain list, and a duplicate delivery becomes a no-op instead of a doubled post in someone’s feed.

A new follow needs a backfill. Follow someone and their existing posts are not in your precomputed timeline. Either backfill their recent posts into your list on follow, or let the read-time merge cover the gap for a while. The first is more work at write time; the second is one more thing the read path must handle.

Read-your-own-writes. A user who posts must see their post immediately, no matter what the queue is doing. Insert into the author’s own timeline synchronously, before returning from the post request. It is one write and it removes the single most-reported feed bug.

The short version

Feeds are read-heavy by two orders of magnitude, so the default is to pay at write time: push post ids into precomputed, capped per-user timelines and make the read a single range read that does not depend on the size of the graph. Store ids, hydrate bodies separately, cap at a few hundred entries.

Then handle the tail, because the follower distribution has an extreme one. Above a threshold, stop fanning out and merge those accounts in at read time from a heavily cached list — otherwise one post saturates the write queue and delays everybody else’s.

The rest is the unglamorous half: filter deleted posts on hydration rather than scanning timelines, key on post id so at-least-once delivery is harmless, decide what happens on a new follow, and write the author’s own timeline synchronously so people can see their own post. None of those change the architecture, and all of them determine whether it works.