A media server that breaks on PGS subtitles isn’t ready for the living room

The subtitle menu is a good place to test how solid a media server’s playback really is. Open it on a Blu-ray remux, try to translate the only decent track, and you find out quickly how much of the pipeline behind it was improvised. That is how we found out about ours.

Every product page says “subtitles supported”. In practice, with a Blu-ray remux whose only decent track is PGS and someone who needs a translation, that checkbox can mean a buffering video, a subtitle stream with nothing in it, or a track that drifts halfway through the film.

In a real living room, subtitles are a chain of decisions. Which track is actually useful? Is it text or a set of images? Can the client draw it? Will the timing survive the trip? And what happens when the file has no usable subtitle track at all?

“Supports subtitles” on its own says very little. The useful question is which source reaches which client, and what the server does when that source isn’t plain text.

What looks like one feature turned out to be three systems that meet at the player, and we built them in the order the problems showed up: text-to-text translation for tracks that already exist, OCR for the image-based subtitles on Blu-ray and DVD, and local speech-to-text for a media file that has nothing usable to start from. All three produce a timed text track. The work is in getting there.

Text subtitles are not automatically the easy case

An embedded SubRip, WebVTT or ASS track looks like the easy case: read the cues, translate them, save a new external track, attach it to the player. The failures actually start before a translator has seen a single word, in choosing which track to read.

One file can hold several languages, a commentary track, a signs-only track, a forced track, a sparse track, and duplicate streams with metadata that lies. Picking the first subtitle stream automatically would be a poor policy, and it would fail on most discs. A forced track is worth having beside dialogue in the original language, and it’s still no substitute for the full dialogue track. A sparse signs track is a bad source for someone who asked to translate the whole film.

So the selection step has to look at every candidate, reject the unusable and forced-only ones when full dialogue was asked for, and prefer the densest track that qualifies. If you explicitly choose a bad source, it should fail where you can see it. Quietly switching to an unrelated track would make the problem impossible to debug and the feature impossible to trust.

Then comes the unglamorous part, which is jobs. A translation can run to thousands of cues, and if the server restarts in the middle it should pick the work back up. If two people ask for the same thing, the quota should be spent once. If the viewer cancels, a late result from a worker shouldn’t wander back in as a new subtitle track ten minutes later. And the progress indicator has to mean something. Selecting, extracting, translating, saving, attaching are the real stages, and the bar should follow them. A cheerful made-up percentage would tell you nothing.

In Quven’s desktop path, which is the one that has been verified, an eligible text source is split into durable translation slices, sent through the Cloud translation path behind the consent and entitlement checks, stored as a localised external subtitle, attached as the selected native track, and restored the next time the media is opened. It was smoke-tested with an embedded English SubRip track translated into French. None of that is visible to the viewer, and all of it is what separates a demo from something people can rely on.

PGS and VobSub are an image-processing problem, not a text-format problem

This is where many otherwise polished setups break.

PGS on Blu-ray and VobSub on DVD are bitmap formats. There’s no text in them to translate. Each subtitle you see is an image with timing attached, and the renderer has to understand the palette, the transparency, how long the image stays up and when it’s cleared, all before OCR even starts.

The obvious approach is to extract a few PNGs, run OCR and write an SRT. It works on a clean test file and fails on most real discs, for the reasons below.

For image subtitles to be reliable, the pipeline has to do a list of things properly. It must select the exact subtitle stream you asked for, because a probe can land on a different one. It resolves cue timing from the format’s own data. It renders the real bitmap. It ignores the transparent clear frames and state frames without losing a genuine cue. It keeps every frame paired with its timing. It recognises the text locally with the right language model. And it writes valid, monotonic WebVTT that doesn’t fall apart two hours into a long film.

The text that comes out then joins the same translation system a normal text track uses. That reuse matters. OCR shouldn’t turn into a second-class feature with its own progress model, its own permissions and its own way of saving things, just because the source arrived as pixels.

There’s a privacy boundary here too. Bitmap recognition happens on your own server. Only the text that comes out of it can enter the Cloud translation path, and only after the same consent and entitlement checks as any other subtitle translation. The OCR cache is keyed on the decoded cue content and the exact timings; the title and the TMDb identifier play no part in it, so two editions with different timing are never treated as the same thing just because they share a poster.

The current implementation has already rendered and restored a translated track that started life as VobSub, in the desktop workflow. That’s real evidence for the cached OCR-to-text translation and playback path. It isn’t a claim that every bitmap format, every operating-system package and every native client has shipped. Cold OCR from scratch, DVB, native payload validation per platform, signing and the native-device gates all still have to close around the engine.

Speech-to-text is the last fallback, not a replacement for good subtitles

There’s a third situation: a file with no usable embedded text, no usable bitmap output and no sidecar subtitle beside it.

That’s where local speech recognition comes in, as the last fallback and only that. A good authored subtitle has editorial timing, knows who’s speaking, and usually has better wording than anything a recogniser produces, so an existing track always comes first. Speech recognition is for the file that would otherwise leave the viewer with nothing.

The route is deliberately local:

selected audio stream → timestamp-preserving PCM → whisper.cpp → normalised WebVTT → optional translation

A few less obvious requirements hide behind that one line. The audio stream you select matters, because a film with several may carry dubbed dialogue, an audio description, a commentary or another language. The job belongs to the exact physical media file you picked, and it preserves the mapping between the container’s timestamps and the decoded PCM samples. Without that, a recogniser can produce perfectly readable words that slowly drift away from the people saying them.

What the recogniser produces can’t go straight into a subtitle file either. It needs checking and cleaning up: malformed timestamps rejected, cues clamped to the length of the media, order kept monotonic, empty output and special tokens removed, the obvious hallucinations over silence suppressed, lines split so they read well, and the WebVTT kept Unicode-safe. The temporary audio, the PCM and the recogniser’s working files have to disappear whether the job completes, fails, is cancelled or is recovered after a restart.

What you want out of it is one validated local subtitle. If a Cloud translation later fails or gets cancelled, that local subtitle is still there and still useful; it doesn’t get thrown away with the failed request. And if the source language and the requested language are the same, there’s no reason to call a Cloud translator at all.

Provisioning Whisper and Silero, deriving synchronised audio, normalising WebVTT, selecting the source and persisting the transcript are all implemented and tested in the current codebase. They shipped in Quven 1.1.0, with cross-client activity, package validation and real-media acceptance behind them. That’s a more useful thing to say than “coming soon”, and it stops short of claiming the work is in every published download.

The useful model is a source hierarchy

The product rule is simple, and it isn’t “always use AI”:

1. use an eligible authored text subtitle when one exists;
2. otherwise turn an eligible bitmap subtitle into text locally;
3. only then offer local speech recognition from the chosen audio stream.

Each level leaves behind a subtitle asset you can inspect; the player can list it, select it, store it and use it again. Each one can survive being cancelled and being recovered after a restart. And each one reports which stage it’s at, since a progress indicator without stages tells the viewer nothing.

That’s what a self-hosted media server needs in order to be trusted with a real, mixed library: remuxes, DVDs, Blu-rays, multilingual releases, forced tracks, old encodes and the occasional file with no subtitle source at all. For the practical player workflow, read how to translate subtitles with AI in Quven.