Video Streaming: You Are Not Serving a File, You Are Serving Six Thousand of Them

The naive version of video streaming is a file on a server and a <video> tag. It works, right up to the moment two things are true at once: someone uploads a 4K file, and someone else tries to watch it on a train.

Everything in a real streaming system exists to break that single file into pieces small enough to make decisions about. Different qualities, so the viewer’s connection can choose. Different time slices, so the choice can be revised every few seconds. The result is that one uploaded video becomes several thousand small files, and the interesting engineering is in producing them, choosing between them, and paying for them.

The upload is a job queue wearing a costume

Nothing about ingest should happen inside the request. The client uploads directly to object storage with a presigned URL — the bytes never touch your application servers, which is the difference between a service that handles large uploads and one that falls over on them. Multipart, so a dropped connection at 92% resumes rather than restarts.

Then the upload emits an event, and everything after that is a pipeline: validate and probe the container, generate thumbnails, extract audio, and encode the ladder. Each rendition is an independent job, which is what makes this scale — a ten-minute video split into one-minute chunks encoded across a hundred workers finishes in about the time it takes to encode one minute.

Toggle the rungs. The top one is typically more than half the storage bill for a video that most people will watch on a phone, and the bottom two cost almost nothing while deciding whether a viewer on a bad connection sees anything at all. This is the clearest cost-versus-reach lever in the system and it is usually set by whoever copied a config from a blog post.

Two operational points fall out of the pipeline shape. Encoding is the most expensive compute you own, so it belongs on preemptible instances with checkpointing — a job that dies at 80% and resumes is cheap, a job that dies and restarts is not. And publish progressively: make the 480p rendition playable the moment it exists rather than waiting for the whole ladder, or an upload is unwatchable for twenty minutes for no reason.

Playback is a control loop with a deadline

On the other side, the player is not choosing a quality. It is running a control loop whose only real objective is that the buffer never reaches zero.

Bandwidth collapses to a fifth for twelve seconds — a train entering a cutting, a phone handing off between towers. Set the policy to hold 1080p and watch the buffer fall monotonically to zero: every segment now takes longer to fetch than it takes to play, so the deficit compounds. The viewer gets a spinner in exchange for a resolution they could not have distinguished on that screen.

Now vary the segment duration, because that is the parameter people set arbitrarily. The player cannot change its mind mid-segment. With ten-second segments it is committed for up to ten seconds after conditions change, which is exactly the reaction time it needs during a drop. With two-second segments it reacts almost immediately — and a forty-minute video becomes 1,200 requests per viewer per rendition, with per-segment overhead on every one and a noticeably worse compression ratio, because each segment must start with a keyframe.

Four to six seconds is the usual compromise, and low-latency live streaming pushes it lower by breaking segments into partial chunks, accepting the overhead because the deadline is different.

The other thing the loop needs is a signal that is not just recent throughput. Throughput-only estimators are famously fooled by a burst at the start of a segment, so production players blend measured bandwidth with buffer level — plenty buffered, be optimistic; running low, step down hard and immediately. That asymmetry is deliberate: step down fast, step up slowly. A player that oscillates between rungs looks broken even when nothing is stalling.

Delivery is somebody else’s network

Video is where a CDN stops being an optimization. The files are large, immutable and requested from everywhere, which is the ideal cache object — and origin bandwidth for video at any real scale is not a bill you want to see.

Two properties make video caching unusually well-behaved. Segments never change, so they can be cached effectively forever with a content-addressed path; only the manifest is short-lived, and for video-on-demand it barely changes either. And popularity follows a brutal power law: a tiny fraction of your catalogue is nearly all of your traffic, so a small edge cache gets a very high hit ratio. The long tail misses, and misses are fine because nobody is watching them.

The manifest — an HLS .m3u8 or a DASH .mpd — is just a text file listing the available renditions and their segments. That is worth internalizing, because it means the “player protocol” is a playlist and a series of ordinary HTTP GETs. There is no video protocol on the wire. Everything adaptive is the client deciding which URL to fetch next, which is why this all works through firewalls, proxies and every CDN ever built.

Live is the same pieces with the slack removed

Everything above assumes video-on-demand, where you have all the time in the world between upload and first play. Live takes the same components and deletes that slack, and the differences are worth naming because “we’ll add live later” is a much larger sentence than it sounds.

Encoding is now real-time or it is nothing. A ladder that takes 4.5 core-minutes per minute of 4K source cannot be produced live on one machine, so live ladders are shorter, encoded in parallel on dedicated hardware, and often start from a single contribution feed that the broadcaster has already encoded once. You cannot retry a failed job; the moment is gone.

The manifest becomes a moving window rather than a fixed list, which means the player is polling for an updated playlist every few seconds and the segment naming has to be predictable. And glass-to-glass latency — camera to viewer — is the number the product cares about, built out of encode time, segment duration, the player’s target buffer and CDN propagation. Standard HLS with six-second segments and a three-segment buffer lands around twenty to thirty seconds behind live, which is fine for a concert and unusable for anything where the audience can hear the neighbors cheering first.

Getting under five seconds means partial segments published before the segment completes, a player buffer measured in hundreds of milliseconds, and giving up most of the safety margin the buffer was providing. That is the whole trade: latency is bought with buffer, and buffer is what was protecting you from the bandwidth drop in the figure above.

The parts nobody mentions

Seeking is a keyframe problem. A player can only start decoding at a keyframe, so a seek lands on the nearest one before your target and decodes forward. Keyframe interval is therefore a direct trade between seek accuracy and file size, and it must be aligned across renditions — if 720p and 1080p have keyframes in different places, the player cannot switch between them cleanly, and you get a visible glitch on every quality change.

Captions and audio are separate tracks, and they are not optional. Multiple audio languages, multiple caption files, all needing to stay in sync with segments that were encoded independently. Retrofitting this is worse than building it, because it changes the manifest structure.

Thumbnail scrubbing is its own artifact. The preview strip when you drag the scrub bar is a sprite sheet generated at upload time. It is a small job that nobody remembers until a designer asks for it.

Storage tiering is most of the savings. Ninety percent of a catalogue is watched almost never after its first month. Moving cold renditions to infrequent-access or archive storage is the second-largest cost lever after the ladder — with the caveat that retrieval from archive takes minutes, so the bottom rungs of cold videos should stay warm.

DRM changes the architecture, not a setting. Encrypted segments, a licence server, and per-device key exchange, with the practical consequence that you now support three DRM systems because the browsers do not agree. Know before you promise it.

The short version

The upload never touches your servers: presigned multipart straight to object storage, then an event, then a fan-out of independent encode jobs on preemptible workers with checkpointing. Publish rungs as they finish rather than waiting for the ladder.

The ladder itself is the biggest cost decision in the system. The top rung is often more than half the storage for a video watched mostly on phones; the bottom rungs are nearly free and decide whether someone on a weak connection can watch at all.

Playback is a control loop trying to keep the buffer above zero, and the segment duration is the reaction time you are buying — four to six seconds, keyframes aligned across renditions, step down fast and step back up slowly. Blend buffer level into the estimate, because throughput alone lies at the start of every segment.

Then let the CDN do the delivery. Segments are immutable, popularity is a power law, and the whole protocol is a text playlist plus ordinary HTTP GETs — which is precisely why it works everywhere.