CodeBrewerzChat looks like the easiest system in this series, because the transport is solved. A WebSocket gives you a persistent bidirectional connection, messages go up one and down another, and a working prototype is genuinely an afternoon.
Then you hit the three things the prototype hid. The recipient is not a person, it is a set of devices that disagree about what they have. The order of a conversation cannot be derived from any participant’s clock. And the small grey tick under a message is a factual claim that you have to be able to defend, in a UI where users have already decided what they think it means.
Nobody’s clock can order the conversation
Start with ordering, because it is the one that produces bug reports nobody can reproduce.
Ana's phone is four seconds fast
—
The obvious design is to stamp each message with the sender’s clock and sort by it. Set the skew to four seconds and read the transcript: Ana’s replies now carry timestamps earlier than the questions they answer, so Ben’s device renders the answer above the question. No error is raised anywhere. The conversation is simply nonsense, and it will be reported as “messages showing in the wrong order sometimes”, which is unreproducible on the developer’s correctly-synchronized laptop.
Four seconds is not an exotic amount of skew. Phones drift, users set clocks manually, timezone handling gets applied twice. A client timestamp is a claim by an untrusted party about a quantity you cannot verify, and it is also trivially forgeable — a client can place its message anywhere in history it likes.
The fix is a per-conversation sequence number assigned by the server on accept. One authority, monotonic by construction, immune to every clock on every device. Switch the control and the transcript is right no matter how wrong Ana’s watch is. Timestamps do not disappear — you still show “3:42 pm” — but they become display metadata rather than the ordering key.
Two refinements matter. The sequence is per conversation, not global, because a global counter is a single point of contention for your entire product. And the client needs its own local ordering for messages it has sent but the server has not yet acknowledged: optimistically render them at the bottom with a pending state, then reconcile when the sequence number arrives. That reconciliation occasionally moves a message, which is acceptable, and it is why the pending state should be visible.
The recipient is not a person
Three devices, one recipient, one tick to show
—
Ben has three devices in three different states, which is the normal case rather than an edge case. Every definition of “delivered” in the control row is defensible and they disagree with each other.
Try “every registered device”. It sounds like the rigorous choice and the tick never turns, because the tablet has been in a drawer for eight days and may never come back. Any definition that waits for every device is a definition that waits for the worst device.
“Any device” is what most products choose, and it is a reasonable lie: at least one thing Ben carries has the bytes. It quietly permits the case where the laptop he actually works on does not.
Underneath the tick, the fan-out is the real work. The server holds a per-device queue, because devices disconnect at different times and each needs to resume from a different point. A device reconnecting asks “give me everything after sequence N for this conversation” — which is the same cursor idea as the message log, applied per device. Push notifications are a separate path for devices with no live connection, and they are best-effort: the notification is a hint that something is waiting, never the message itself, because notification delivery is outside your control and frequently silently dropped.
And read receipts are a product decision that arrives disguised as an engineering one. They require the receiving client to report back after rendering, which means they can be disabled, delayed, or fabricated, and they carry social weight that no other tick does. Decide whether they are optional before you build them, because retrofitting a privacy toggle onto a receipt system is harder than it sounds.
Where the messages live
The storage shape follows from one observation: chat is written once, read in reverse chronological order, and almost never updated. That is close to the friendliest access pattern a database has.
Partition by conversation id, cluster by sequence descending. Loading a conversation is then a single contiguous read of the most recent N rows from one partition — no join, no scatter, no sort. Scrolling back is the same read with a cursor. This works on a sharded relational store or a wide-column one; the shape matters more than the product.
Group chat forces one decision. Either every member’s device queue gets its own copy of each message — fan-out on write, fast to read, expensive for a group of 500 — or there is one copy per conversation and each device tracks its own position — fan-out on read, cheap to store, more work per open. This is the same trade-off as the news feed, with one difference: group sizes in chat are bounded by a product rule, not by a celebrity’s follower count, so the hybrid is less necessary and a single copy per conversation is usually enough.
Media does not go through the message path at all. The client uploads to object storage directly and sends a reference; the message stays small, and the retry semantics of a 100 MB video stop being entangled with the retry semantics of a text message.
The parts nobody mentions
Typing indicators must not be durable. They are high-frequency, worthless after two seconds, and if they go through the same persistence path as messages they will be the majority of your write volume. Send them over the live connection only, and never store them.
Idempotency is on the client, not just the server. A message send that times out will be retried by the app, and without a client-generated message id the retry becomes a second message. Generate a UUID on the device at compose time and let the server deduplicate on it — the same key that makes the ledger safe.
Presence is more expensive than messaging. “Ana is online” changes constantly and must be pushed to everyone who can see it, so a user with 500 contacts generates 500 fan-outs per state change. Presence is the feature that takes down chat systems. Coarsen it — “active recently” rather than a live dot — and expire it aggressively.
Deletion has to mean something specific. “Delete for everyone” is a request to devices that may be offline for a week, so it is a tombstone that gets applied on reconnect, and you cannot promise it happened. Say what you actually guarantee.
End-to-end encryption removes your options. If the server cannot read the message, it cannot do server-side search, cannot generate a notification preview, and cannot transcode media. Every one of those becomes a client-side problem, on the weakest device the user owns. That is a defensible trade and it is not a feature flag.
The short version
Order the conversation with a per-conversation sequence number assigned by the server. Any design that sorts by the sender’s clock produces replies above questions the moment one device is a few seconds off, raises no error, and cannot be reproduced by the person debugging it.
The recipient is a set of devices in different states, so “delivered” needs a definition you choose deliberately — and any definition requiring every device is a tick that never turns. Keep per-device queues with a resume cursor, treat push notifications as a hint rather than a delivery mechanism, and decide the privacy shape of read receipts before you build them.
Store by conversation, clustered by sequence descending, so loading a chat is one contiguous read. Keep typing indicators and presence off the durable path entirely — they are the highest-volume, lowest-value data in the system, and presence is what will take you down. And give every message a client-generated id at compose time, because the retry that duplicates a message is not a hypothetical, it is what happens every time a phone changes network.