A reuse rate case study for an LLM workflow

I was asked to review the architecture of a machine translation pipeline for regulated technical directives, German to French. The documents are the kind where one term has one approved translation and a reviewer signs the result.

To review it I needed something to measure, so I wrote a small demonstrator: 6,600 lines of Python, standard library plus a PDF reader, runs offline in one command and prints a table. It translates badly on purpose. By default the model is a local lookup table. Its job is to measure three mechanisms, and to label everything it does not measure.

All figures below come from a public document: Swissgrid's minimum technical requirements for connecting converter-based energy storage systems, version 1.1, 36 pages, published in German and in French. No client document appears in this post.

The design

Three decisions carry the rest.

Segment per sentence, not per paragraph. A paragraph almost never repeats between two versions of a directive. A sentence does. Everything about reuse follows from this.

Look up the memory by edit distance on the source string, not by embeddings. A sentence and its negation share a topic and a vocabulary, so a vector index tends to call them near-identical. Edit distance does not put them far apart either, but it never calls them the same: the match comes back fuzzy, below 100%, and a fuzzy match is always opened by a reviewer. In a regulated text, serving the translation of "requires approval" for "does not require approval" is the worst available failure, and the test suite has a case for exactly that pair. I did not run an embedding model to compare, so the first half of that argument is reasoning and not a measurement.

Label every segment. Each sentence leaves with a status and a reason, so the reviewer opens only what is flagged.

What reading the PDF costs

The longest segment is a table that PDF extraction poured into running text. That one row is the argument for working from the original Word file through a proper filter and not rebuilding structure from a PDF.

The number that depends on a setting nobody sees

To measure reuse on a real document I aligned the German with the published French, sentence by sentence, and loaded the pairs into the memory. Then I translated the German again and counted how many characters never reached the model.

The aligner gives each pair a confidence score. A threshold decides which pairs enter the memory. Same document, same memory, same code, only the threshold moves:

From 14% to 91% on one setting. Anyone quoting a reuse rate without the threshold behind it is quoting nothing.

At 0.0 the pipeline reuses 91% by accepting wrong pairs, and a wrong pair loaded into a memory comes back later as a wrong translation with a displayed score of 100%. The high number is the dangerous one.

The cliff between 0.5 and 0.7 has a dull cause.

202 of the 716 pairs score exactly 0.600. The aligner cannot rank them against each other, so it gives them all the same value. A threshold at 0.7 drops all of them at once and a threshold at 0.6 keeps all of them. My default landed just above the plateau by accident. The tool now prints its plateaus so the cliff is visible before someone picks a number.

The score is noisy in both directions. • Normen IEC; and • les normes de la CEI ; are a correct pair scored 0.32, because the acronym itself is translated and the surfaces diverge. A plainly wrong pair in the same document also scores 0.32. The score has never been calibrated against a human judgement on this corpus. That calibration is a concrete task with a cost, and it comes before any claim about reuse.

Reuse between two versions

The other reuse case is the pleasant one: version 2.3 of a document after version 2.2 has been translated. I built a synthetic pair where the second version differs from the first by six declared string replacements, listed in a changelog next to the files.

92% is a laboratory ceiling, since I wrote the second version myself. The number to present is the one from the real document, with its threshold.

A word search does not hold in German

The usual glossary check is a regular expression with word boundaries. German breaks it twice. It declines: \bPlangenehmigungsverfahren\b does not find Plangenehmigungsverfahrens, because the genitive s removes the boundary. And it compounds: the term can sit inside Plangenehmigungsverfahrensantrag, where no boundary will ever be.

I ran both checkers on the same segments, the naive regex and a morphological one that splits compounds and folds declensions.

The regex reports 52 on the synthetic sample, and one of those is a finding the morphological checker does not confirm. That is why it misses 5 and not 4. The second column is the more useful one. The glossary was written for a directive about plan approval. On a document about energy storage it recognises one compound and one term. A glossary does not transfer between documents, and neither does a hand-written compound lexicon. The 9% is measured on that text with that glossary. It is not a statement about German.

Measuring quality, with the caveats attached

Quality is measured in exactly one place: translate cold with a real model, then score against the published human translation.

Two changes separate the runs. The glossary and instructions moved into a cached system prompt, with only the per-segment terms in the user turn. And the first run leaked. On short headings the model sometimes returned the German unchanged followed by its own hesitation: 6.2.1.5 Validierung, then "Hmm, the segment is". A scorer sees that as a bad translation. A reviewer sees a document with model commentary inside it. The second run forbids commentary in the prompt and filters what still gets through.

It did not fix everything. Segment 648, a sentence fragment cut at a page break, still comes back in German in the second run. It is flagged, not hidden.

What these scores do not say is printed under them every time:

• 311 of 750 segments are scored, 41% of the document. A segment without a reference is not counted as a zero and enters no average.

• The reference is paired by an automatic aligner, so alignment error is inside the score.

• chrF and BLEU measure surface overlap with a single reference. A different correct translation scores low and a close wrong one scores high.

• Nobody has read the output. The numbers are automatic end to end.

What I would keep

A probabilistic component is easier to trust inside a deterministic frame. The model is one interface with three implementations: a local stub, an API, a self-hosted server. Nothing else in the pipeline knows which one is running, so the question of where the data may go stays a deployment decision.

Every number the tool prints is either measured by the tool or labelled as a demonstration, and the table always ends with what it does not say. That costs a few lines of code and it is the part of the review people remember.

Sources

• Popović, chrF: character n-gram F-score for automatic MT evaluation, WMT 2015

• Papineni et al., BLEU: a method for automatic evaluation of machine translation, ACL 2002

• OASIS, XLIFF 2.0, the review format the pipeline exports

• Measurements: my own runs of 5 November 2026 on the public document above

The demonstrator behind these measurements was built with AI coding tools. The text was proofread by AI.