Designing a news feed: fan-out on write, fan-out on read, and what a 5-million-follower account breaks Advanced

The problem

A generic social app: users follow other users, post short text, and open a home feed showing the newest posts from everyone they follow. The naive query joins follows with posts, sorts by time, and takes 20 rows. It works at 10,000 users.

At 20 million daily users, the same query runs thousands of times a second over hundreds of followees each. Precomputing every feed moves the cost to writes, until one account with 5 million followers posts.

Where should the work happen: when a post is written, when a feed is read, or both?

What to choose? (summary first)

Situation	Choose	Condition
Typical authors (below the threshold)	Fan-out on write into capped Redis timelines	Feed reads far outnumber posts; timelines fit in RAM
Authors above a follower threshold (say 10,000)	Fan-out on read for their posts, merged at read time	Each reader follows only a few such accounts
Small app, or a chronological admin view	Pure pull from the database	An index on (author_id, created_at) keeps the query fast
Followers inactive for 30+ days	No timeline; rebuild on next login	Saves fan-out writes and RAM
Pages beyond the timeline cap	Pull from the posts store, or end the feed	Few users scroll past ~800 posts

Load estimation (stated assumptions)

These numbers are illustrative assumptions, not data from a real product.

Assumption	Value
Registered / daily active users	200 million / 20 million
New posts per day	4 million
Mean followers per posted item	200 (median far lower; a few accounts have millions)
Feed opens per active user per day	10
Peak-to-average factor	3x
Post row (text, metadata, indexes; media elsewhere)	~1 KB

Quantity	Calculation	Result
Posts	4,000,000 / 86,400	~46/s, peak ~140/s
Timeline inserts (fan-out on write)	4M × 200 = 800M/day	~9,300/s, peak ~27,800/s
Feed reads	200M / 86,400	~2,300/s, peak ~6,900/s
Post storage	4M × 1 KB	~4 GB/day, ~1.5 TB/year
Follow edges	200M × 100 following (assumed)	20 billion edges

The decisive number is write amplification: one post becomes 200 timeline writes on average. Writes are cheap individually; the problem is the tail of the follower distribution.

Core concepts

	Fan-out on write (push)	Fan-out on read (pull)
Work at post time	One insert per follower	One insert
Work at read time	Read one precomputed list	Query every followee, merge, sort
Read latency	Low and flat	Grows with number of followees
Cost of a 5M-follower post	5M writes	Nothing extra
Freshness	Delayed by queue lag	Immediate

The data model has three parts:

• posts(post_id, author_id, created_at, body, deleted_at): the source of truth.
• follows(follower_id, followee_id), indexed both ways: “who do I follow” and “who follows me”.
• tl:{user_id}: a Redis sorted set per user. Member = post_id, score = created_at in milliseconds. It stores ids only, never content.

How it works

One post becomes 300 timeline writes; a feed read is one ZRANGE plus a 20-id hydrate

Enlarge diagram

Post API → Posts storeINSERT post

◀ Prev▶ PlayNext ▶↺ RestartStep 1/6

Author A posts. The Post API inserts post 7390…999 into the posts store; this is the only durable write.

State after each step:

1. The post exists only in the posts store. No follower can see it yet.
2. The event is queued after commit (see transactional outbox). Fan-out delay starts now.
3. The worker holds one page of follower ids. Large accounts need thousands of pages.
4. 300 timelines contain the new id, each trimmed to 800 entries. This is where write amplification lands.
5. B gets ids in order in one command: ZRANGE tl:B 0 19 REV.
6. B gets content. Edits and deletions take effect here because timelines hold only ids.

The Redis costs, from the official docs: ZADD is O(log N) per member, ZRANGE and ZREMRANGEBYRANK are O(log N + M), where M is the number of returned or removed members. ZREVRANGE still works but is deprecated since 6.2; use ZRANGE ... REV.

The celebrity problem and the hybrid

With a 10,000-follower threshold, the post path checks the author:

Loading diagram…

1. A typical author pays 200 writes per post; readers pay nothing extra.
2. A 5M-follower author pays one write. Every reader following them pays one extra lookup on posts:{author}.
3. posts:{author} of a large account is now a hot key: millions of readers hit one shard. Cache it in process for 1–5 seconds.

Keep hysteresis on the threshold (switch to pull at 10,000, back to push below 8,000) so an account near the line does not flap.

Queue and worker fan-out

Do not fan out inside the request. Split it in two stages:

1. PostCreated keyed by author_id (ordering per author, see message keys).
2. A splitter pages followers and emits FanoutBatch {post_id, 1,000 follower ids} keyed by batch, spreading one large post over every partition.

Large authors get a separate lane, so their batches never queue in front of ordinary posts. Monitor consumer lag as time, per lane (see lag monitoring).

Operational details

• Pagination: use a cursor (score, post_id) of the last item, not a rank. Ranks shift whenever a new post arrives. See keyset pagination.
• Tie-break: query ZRANGE tl:B <score> -inf BYSCORE REV LIMIT 0 21 (inclusive), then skip members ≥ the cursor’s post_id. Equal scores come back in reverse lexicographic order, so ids need fixed width.
• Score precision: scores are doubles, exact only up to 2^53. A 64-bit snowflake id as score loses precision; use milliseconds.
• Delete: mark deleted_at in the posts store. Hydration drops it; do not fan out a delete to 5M timelines.
• Edit: invalidate the post content cache only; timelines are unaffected.
• Unfollow: an async job ZREMs that author’s recent post ids from the follower’s timeline. Block must filter at read time, not eventually.
• Follow: backfill the author’s last 20 posts into the follower’s timeline.
• Ranking is out of scope. It plugs in after retrieval: the timeline supplies ~800 candidates, a ranker reorders them, and the cursor must then refer to a ranked snapshot.

Real-world example

Timeline memory: measured on Redis 7.4.9

800 entries, 19-digit post ids, millisecond scores. Default zset-max-listpack-entries is 128, so the timeline is a skiplist: MEMORY USAGE ... SAMPLES 0 = 84,048 bytes (~105 bytes per entry). With the threshold raised to 1,024 it stays a listpack: 16,432 bytes.

For 20 million active users that is ~1.7 TB of RAM as skiplists, or ~330 GB as listpacks. Listpack inserts shift a contiguous array, so benchmark write cost before raising the threshold.

The commands behind those numbers (run against examples/docker-compose.yml, Redis on port 6380):

> ZCARD tl:test                      # after 1,000 ZADDs
1000
> ZREMRANGEBYRANK tl:test 0 -801     # keep the newest 800
200
> OBJECT ENCODING tl:test
skiplist
> MEMORY USAGE tl:test SAMPLES 0
84048
> ZADD p 9007199254740993 a          # 2^53 + 1
1
> ZSCORE p a
9007199254740992

The default MEMORY USAGE samples only 5 nested values and reported 79,392 for the same key; use SAMPLES 0 for capacity planning.

🎮 Try it

• Scenario: a celebrity posts: a 5M-follower account backs up the fan-out queue for 40 minutes.

Trade-offs

Pros

• Feed reads are one ZRANGE plus a 20-id hydrate, independent of followee count.

• Fan-out batches are idempotent: re-running ZADD with the same member and score changes nothing.

• The hybrid caps the worst post at threshold × 1 writes.

< slot="cons">

• Write amplification: 800M timeline writes per day for 4M posts.

• Freshness depends on queue lag; a backlog delays everyone behind it.

• Two read paths to maintain, plus threshold transitions and backfills.

• Timelines cost RAM (~0.3–1.7 TB here); inactive users must be excluded.

Cons

Production checklist

1. Write down the follower distribution (median, p99, max) before choosing push, pull or hybrid.
2. Set the hybrid threshold with hysteresis, and store the mode per author.
3. Split fan-out into author events and follower batches; give large authors their own lane.
4. Alert on fan-out lag in seconds per lane, not in message count.
5. Fan out only to followers active in the last 30 days; rebuild other timelines on login.
6. Cap timelines (ZREMRANGEBYRANK 0 -801) in the same pipeline as ZADD.
7. Paginate with (score, post_id) cursors; never pass a rank to the client.
8. Handle delete, edit, unfollow and block at hydration time, with async cleanup.

Common mistakes

Two mistakes appear in almost every first design.

Fanning out inside the request

✕ Common

for (const f of await followersOf(authorId)) await redis.zadd(`tl:${f}`, ts, postId);
return res.status(201).end();

✓ Prefer

await db.tx(async (t) => { await insertPost(t, post); await outbox(t, 'PostCreated', post); });
return res.status(201).end();

Why

The first version makes the author wait for 5M round trips and loses the remainder on a crash. The outbox commits the event with the post.

The second mistake looks harmless in tests because test data has no celebrities.

Keying fan-out batches by author_id

Every batch of a 5M-follower post lands on one partition, so one consumer does all 5,000 batches while others idle. Key the author event by author_id, but key the batches by batch number.

Misconception	Consequence	Do instead
Timelines store post content	Edits and deletes must touch millions of keys	Store ids, hydrate at read
Paging by rank (ZRANGE 20 39)	Duplicates when new posts arrive	Cursor (score, post_id)
One queue for all authors	A celebrity post delays everyone	Separate lane plus hybrid

Self-check

1. With a mean of 200 followers and 4M posts per day, how many timeline writes per second does the peak (3x) produce?
2. Why does a delete not need to be fanned out, but a block must be checked at read time?
3. An account crosses from 9,999 to 10,000 followers. What happens to its posts already in followers’ timelines, and to new ones?
4. Why does ZRANGE ... (score -inf BYSCORE REV (exclusive) lose posts?

Summary

Key takeaways

• Under the stated assumptions: 46 posts/s become ~9,300 timeline writes/s, against ~2,300 feed reads/s.
• Push makes reads one ZRANGE; pull makes posts one insert. The follower distribution decides.
• The hybrid pushes for ordinary authors and pulls for accounts above a threshold, with hysteresis.
• Timelines hold capped ids; content, deletes and edits are resolved at hydration.
• Fan-out runs in batches on a queue, with a separate lane and lag measured in seconds.

References

• ZADD — Redis Docs: O(log N) per member, update semantics, 2^53 score precision.
• ZRANGE — Redis Docs: REV, BYSCORE, LIMIT, tie ordering.
• ZREVRANGE — Redis Docs: deprecated as of 6.2.0.
• ZREMRANGEBYRANK — Redis Docs: negative ranks for trimming to the newest N.
• MEMORY USAGE — Redis Docs: why SAMPLES 0 matters for large keys.
• redis.conf for Redis 7.4: listpack thresholds for sorted sets.