CodeBrewerz“Put a queue in front of it” is the most-given piece of architecture advice, and it is usually right. Producers and consumers stop having to be alive at the same time, a traffic spike becomes a backlog instead of an outage, and a slow downstream service stops being your problem to solve synchronously.
It is also the advice most likely to be followed with the wrong product. RabbitMQ and Kafka are both described as message brokers and are not the same category of thing — one is a router that destroys messages on delivery, the other is a file you append to and read with a cursor. Choosing between them by throughput benchmarks is choosing on the one axis that almost never decides it.
A queue divides work. A log retains it.
The distinction is one sentence, and it sounds like an implementation detail until you have lived on the wrong side of it. A queue treats delivery as a division of labour: a message goes to exactly one consumer in the group and is then removed. A log treats consumption as a cursor: consumers hold an offset into retained, ordered data, and reading changes nothing except your own position.
Add a second consumer, six months later
—
Consume a few messages as the billing service, then add the analytics service — first in queue mode, then in log mode. On the queue, the new consumer can only ever receive what billing has not already destroyed. On the log, it starts at offset zero and replays everything, including messages billing processed months before analytics was written.
That difference decides more of your architecture than throughput does. If a message is a command — “send this email”, “resize this image” — a queue is right and retention is pointless; the email either gets sent or it doesn’t. If a message is an event — “an order was placed” — you do not know today how many things will want to know that tomorrow, and a log means adding the fourth consumer is a config change rather than a redesign of the producer.
The log also gets you a debugging property that is hard to overvalue: when a consumer processes a batch wrongly, you can fix the code, reset the offset, and reprocess. On a queue those messages are gone and your recovery plan is a backfill script written under pressure.
Two costs, in fairness. Ordering on a log is per-partition, not global, so the ordering guarantee you actually get is “messages with the same key are in order” — and that only holds if you chose the key deliberately. And a log needs a retention policy: seven days of every event is a real storage bill and a real compliance question, and “keep everything forever” is a decision, not a default.
The guarantees, and the one that does not exist
Now the part every vendor page fudges.
Crash the consumer at each point in the handler
There is one message, one handler, and one crash. The only real choice is where the acknowledgement sits, and it has exactly two positions.
Acknowledge before the work and you have at-most-once. Crash between the ack and the side effect and the broker has already forgotten the message. Nobody is ever charged and nothing anywhere records that anything went wrong. Use this for data where loss is genuinely cheaper than duplication — telemetry, mostly.
Acknowledge after the work and you have at-least-once. Crash after charging the card and before the ack, and the broker’s visibility timeout expires and hands the same message to another consumer, which charges the card again. This is the default in every broker worth using, and it is the right default.
There is no third position. Any place you put the ack, a crash can land on the other side of it. The broker cannot give you exactly-once delivery, and the systems that advertise “exactly-once” are describing exactly-once processing within their own transactional boundary — Kafka’s version covers a consume-transform-produce cycle that stays inside Kafka, and stops existing the moment your handler charges a card or sends an email.
So the guarantee is not something you select. Set the handler to the idempotency-key version and run the same crash again: the redelivery still happens, the handler recognises the key as already applied, and the second attempt does nothing. That is what exactly-once means in practice — at-least-once delivery plus a handler that can absorb a repeat. The payments ledger post is the same idea from the other end, which is not a coincidence; it is the same problem.
The practical version: every message carries a stable, producer-generated identifier, the handler records that identifier in the same transaction as its effect, and a second arrival of a known identifier returns without re-running. If the effect is not transactional — an outbound email, a third-party charge — you need the idempotency key to be understood by the thing on the other end, which is why good payment APIs all take one.
The parts nobody mentions
Ordering is narrower than you think. A log orders within a partition, and the partition is chosen by the message key. Two events about the same order land in order only if they share a key. Change the partition count later and the key-to-partition mapping moves, so messages that used to be ordered against each other stop being — repartitioning is a data migration, not an autoscaling event.
One slow message blocks its whole partition. Ordering means the consumer cannot skip ahead, so a single poison message stalls every message behind it indefinitely. This is head-of-line blocking, and the fix is a dead letter queue with a retry limit — after N attempts, move it aside and continue. A DLQ nobody alerts on is a directory where messages go to be forgotten, so alert on its depth, not just on its existence.
Retries need backoff and jitter. Immediate retry against a struggling downstream service is a denial-of-service attack you wrote yourself. Exponential backoff with randomisation, always.
Consumer lag is the metric. Not throughput, not queue length in isolation — lag, the distance between the newest offset and your committed one, because it is the only number that answers “how far behind is reality?”. If lag is growing your consumers are permanently under-provisioned, and no amount of burst capacity fixes a rate mismatch.
The transaction across two systems does not exist. Writing to your database and publishing to the broker cannot be made atomic by trying harder. Either the database write lands and the publish fails, or the reverse. The standard answer is the transactional outbox: write the event into a table in the same transaction as the state change, and let a separate process read that table and publish. You have converted an impossible distributed transaction into a local one plus an at-least-once publisher, which you already know how to handle.
The queue that never drains
A queue absorbs a spike. It does not absorb a rate mismatch, and the difference is the one people discover during an incident.
If producers run at 1,000 messages a second and consumers process 800, the backlog grows by 200 a second forever. Adding consumers fixes it; adding queue capacity does not, it only moves the moment you notice. The tell is that queue depth is rising steadily rather than spiking and recovering, and the correct alert is on the slope, not the depth.
Which raises the question everyone defers: what should the producer do when the broker is full or unavailable? There are exactly three answers, and only three. Block — the producer waits, and your queue has just propagated backpressure into a synchronous request path, which is often correct and always needs a timeout. Drop — refuse the message and return an error, which is right for telemetry and wrong for orders. Buffer locally — write to disk and publish later, which turns every producer into a small broker with its own durability problem.
Pick one per message type and write it down, because the default is “block forever with no timeout”, and that is how a broker hiccup becomes a full outage of a service that was supposed to be decoupled from it.
The related trap is retention as an availability mechanism. Seven days of retention does not mean you have seven days to fix a broken consumer — it means that on day eight the broker starts deleting messages your consumer has not read, silently, with no error anywhere. A consumer that is down for a long weekend and a retention window of three days is data loss with a paper trail that says everything was healthy.
The short version
A queue divides work and destroys messages on delivery; a log retains them and hands each consumer group its own offset. Commands belong on a queue. Events belong on a log, because you do not yet know who will want them and replay is the difference between a config change and a backfill script.
Where you put the acknowledgement is the only guarantee you get to choose: before the work is at-most-once and loses messages, after the work is at-least-once and duplicates them. No broker can eliminate the gap, and “exactly-once” on a product page means exactly-once inside that product’s own transactions, which is not where your card charge lives.
So build the handler that makes duplicates harmless: a producer-generated identifier, recorded in the same transaction as the effect, checked before the work runs. Then key your partitions on purpose, cap retries into a dead letter queue you actually alert on, and watch consumer lag rather than throughput — it is the only number that tells you how far behind the truth your system currently is.