Our subtitle pipeline worked. Real discs disagreed.

A few days ago we wrote about how a media server ought to handle subtitles: translate the text tracks that already exist, run local OCR on the image-based ones from DVD and Blu-ray, and fall back to listening to the dialogue when a file has nothing usable at all. Three routes, and a timed text track at the end of each.

The architecture was right. Then we pointed it at real media, and it fell over three times, never once in the place we were watching.

A disc that produced 984 subtitles and matched none of them

The first test was a PGS-only Blu-ray. The log said frameCount=1371 imageCount=0, the job failed on a cue-timing mismatch, and then something worse happened without a sound: automatic source selection shrugged, moved down the list, and translated the German from a French subtitle Quven itself had generated earlier. Two languages away from the original, and nobody was told.

There were three causes stacked on top of each other, which is why it looked like one big mysterious failure.

FFmpeg can’t work out the size of a PGS stream from the default probe, so the filter graph was built from the video dimensions, and the first correctly sized subtitle was rejected as a change in frame properties. A bigger probe window will fix that one.

Then the extracted images were blank. We’d been pulling the alpha plane, which would be the reasonable thing to do for an overlay format, except that the PGS decoder hands over paletted frames whose alpha plane measures a flat zero while the actual glyphs live in the colour planes. We had been extracting the one plane that held nothing.

The third one was the good one. Every image was extracted, every cue was known, and not a single pair matched. FFprobe reports the raw container timestamps; FFmpeg rebases its output to zero by subtracting the container start time. On this disc the container started 79 milliseconds in, so every image landed one start time earlier than the cue it belonged to, and the one-millisecond tolerance threw all of them away. Before the fix: 0 of 984. After: 984 of 984.

That same start time turned out to explain an offset we’d been blaming on VobSub for weeks. VobSub now pairs 73 of 73 by timestamp, and no longer has to guess by position.

One approach we tried and threw away, in case it looks tempting: using scene detection to drop the transparent clear frames PGS emits between subtitles. It will work right up until two consecutive cues contain the same text, which scores as no scene change, and one of them will silently disappear. We’d sooner have a pipeline that fails loudly than one that quietly loses a line of dialogue.

The recogniser was fine. We were chopping its sentences up.

Then the audio route. A film with no usable subtitle track went through local speech recognition and came back readable, timed correctly, and almost unwatchable, because the dialogue appeared roughly a word at a time. 408 of the 1593 cues held a single word. 113 were exactly 7.00 seconds long, which is a suspiciously round number and the kind of detail you should stop on.

The obvious suspect was the recogniser, or its segmentation flags. Was it? We ran whisper directly on the same audio, and it returned whole sentences. A ten-and-a-half-second "Come va?" came back as one span. We tested the flags we suspected too, and they produced byte-identical output, so that theory died as well.

The defect was in our code. Seven seconds is our maximum display time for a cue, and somewhere along the way that display cap had started being used as a boundary for the text as well. Any span longer than seven seconds was cut at every word. The cap belongs to the timing pass, and that’s where it must live, pinned by tests so it can’t wander back.

There’s a general lesson in there that we keep relearning. You should run the component on its own before you blame it.

Eight minutes with every core at 100%

The third problem wasn’t a bug. Transcribing a film took about eight minutes and pinned all 24 cores of the machine for the duration. It worked, and it also made the computer unusable for eight minutes, which isn’t something anyone would want running in the background, and it would only get worse on a longer film.

So recognition moved to the GPU: Metal on macOS, Vulkan on Windows and Linux. On the card we measured, the encode step went from 591 milliseconds to 15.7, and a two-minute clip finished in 2.26 seconds where it had taken 6.34. A two-hour film now takes roughly two and a half minutes, down from eight, and the CPU is mostly idle while it does.

Selection walks down a ladder (discrete GPU, integrated GPU, optimised CPU build, plain CPU build), so a machine with no usable adapter will still transcribe, just more slowly. The preference for the discrete card must be learned from the adapter list, because the enumeration comes back in driver order and would otherwise hand us whichever one happened to be first. And on the CPU path we now leave at least one processor free, so the machine stays responsive while it works.

The limit, stated plainly: this is measured on one NVIDIA card. AMD, Intel and Apple Silicon are implemented, and they’ll stay unverified until we’ve run them on real hardware. We’d sooner say that than let you find out.

What “good enough” actually looks like

We reviewed a complete translated episode line by line, all 740 cues, against the original. Around 18 were genuinely wrong, and of those maybe a dozen matter: grammar, an idiom that landed oddly, a continuity slip. That’s a usable subtitle. It couldn’t pass for a human translator, and we’d sooner write that number down than describe it as indistinguishable from professional work.

The audio route is more interesting. The translation itself came back clean, 1593 of 1593 cues with nothing empty and no leakage from the source language, and the translator actively repaired the recogniser’s mistakes along the way, quietly turning invented words back into real ones. What survives is the other class of error: text that’s perfectly grammatical and simply isn’t what was said. On that route the recogniser sets the ceiling, and no amount of work further down the chain will fix a sentence that was misheard confidently.

Why one half of this is free and the other cannot be

Generating a subtitle locally uses your processor, your graphics card and your electricity. Nobody sends us an invoice for that, so it’s free, and it will stay free.

Translation is a different shape. The text leaves the machine, and every film is thousands of cues going through a paid provider, on servers we rent, behind a job queue that has to survive restarts, plus the licences and the months of work that put it there. A single two-hour film is a real, countable cost, and it will be repeated every time somebody presses the button.

That’s the only reason it can sit in the paid tier. The local floor stays complete and free. This specific part has a bill attached, and pretending otherwise would just mean we’d have to cap it quietly later. If you want to try it before the plans open, that conversation can happen on the Discord.

For the practical version of all this, there’s a guide on translating subtitles with AI, and another on generating subtitles for a film that has none.