How well does this actually work?

Text units translated

67

Coverage

100.0%

Source text left untranslated

0.0%

URLs and emails altered

0

What the harness checks

• No-op gate. Translate with a pass-through that returns each string unchanged, then compare the two .docx packages part by part at the XML level. Each part is compared as a canonicalized tree, so attribute order and whitespace noise do not count as changes, only real structural differences do. The result must be identical.
• Formatting gate. Translate for real, then run the same comparison with every text node blanked first. This answers the question that matters: did anything other than the words change? Styles, tables, numbering, headers, footers, colours and page breaks all have to survive it.
• Coverage and leakage. Read the output back and language-detect every text unit. Coverage is how much the engine reported translating; leakage is output text still detected as the source language. A tool can translate 100% of units and still leave source text sitting in the file, so these are measured separately.
• Do-not-translate gate. Harvest every URL and email address from the source and require each one to appear verbatim in the output. These have no valid localized form, so any change is a real defect rather than a judgement call.
• Opens cleanly. The translated file is opened with LibreOffice to confirm no repair prompt or load error.

Known limitations

• The published live run targets French. Other languages are tested by hand, not by this harness.
• Leakage is a review signal, not a hard gate. Short units are unreliable to language-detect, so the metric only judges units long enough to be confident and never fails one for uncertainty alone.
• Numbers and dates are deliberately excluded from the do-not-translate check, because localizing "1,234.56" to "1 234,56" in French is correct behaviour, not a violation.
• This measures fidelity, not fluency. There is no LLM-judge or COMET-style quality score here, so none of these numbers claim the translation reads well, only that the document survived it intact.

Use the Word (.docx) translator → Harness: eval/run_eval.py

What the harness checks

• No-op gate. A full-scope identity translation must leave every zip entry byte-for-byte identical except the shared-string table and the recalculation flag. This proves the engine never re-serializes what it did not translate.
• Scope gate. Translate one sheet and two columns with the header row skipped, then check the four promises the scope panel makes: only the selected cells changed, formulas and numbers and SKUs in other columns are untouched, unselected sheets round-trip byte-identical, and a string shared between an in-scope and an out-of-scope cell is split so the out-of-scope copy keeps its original text.
• Rich text and inline gate. When the model reorders the formatting markers inside a cell, the bold or italic run has to rebuild onto the right word. When the model drops a marker instead, the cell falls back to its original text, so the output is never worse than the source.
• Scope validation. Malformed scope input is rejected rather than silently interpreted as "translate everything".

Known limitations

• Every gate is offline, driven by fake and identity translators. It proves the engine cannot corrupt a workbook; it says nothing about how well Gemini translates one.
• Chart titles and cell comments are not translated, and so are not evaluated.

Use the Excel (.xlsx) translator → Harness: eval/run_xlsx_eval.py

What the harness checks

• No-op gate. Translate nothing. Every zip entry must be byte-identical and the mimetype entry must still be first and stored uncompressed, which is what strict readers check before anything else.
• Identity gate. Echo every unit back unchanged. The chapter documents genuinely get re-parsed and re-written here, so this is the round-trip test: text, tag structure and attributes have to survive it, the output must still be well-formed XML, and every stylesheet, image and font must still be byte-identical.
• Structure gate. Translate the whole book and check what the tool page promises: chapter text, the book title and both tables of contents are translated, the spine and manifest are unchanged so reading order survives, the language tag is rewritten to the target, the author and identifier are left alone, and links, image sources and stylesheet references are untouched.
• Marker gate. Inline emphasis is tracked with markers, and models mangle them. Reordered markers must rebuild as real tags on the right word. Dropped markers must keep the text and be counted rather than hidden. Even deliberately corrupted markers must still produce a valid document.

Known limitations

• Every gate is offline. It proves the book survives the engine, not that the prose reads well.
• DRM-protected books are not supported and are not tested.

Use the EPUB translator → Harness: eval/run_epub_eval.py

Strings seen

56

Selected for translation

38

Deliberately skipped

18

Placeholders protected

32

What the harness checks

• No-op gate. An identity translation must re-serialize byte-for-byte identical, including indent width, tabs, unicode escaping, key order and the trailing newline.
• Placeholder gate. A translator that rewrites the words but keeps the markers must return every original placeholder verbatim, once, in the right string.
• Hostile gate. A translator that drops the markers is the realistic model failure. Those strings must fall back to the original value rather than ship a locale file containing a dead variable.
• Structure gate. Keys, key order, nesting and non-string types are unchanged throughout all of the above.

Known limitations

• Every gate is offline, so these numbers are about the engine, not the model.
• ICU plural categories are not restructured. Each plural string is translated on its own, but a language whose plural rules differ from the source may need categories added or removed by hand.

Use the JSON / i18n translator → Harness: eval/run_json_eval.py

Strings seen

80

Selected for translation

42

Deliberately skipped

38

Placeholders protected

5

What the harness checks

• No-op gate. An identity translation must come back byte-for-byte identical, run three ways: default scope, a column subset, and with the header row skipped. The frame under test is the real one, meaning delimiters, quoting style, line endings, ragged rows and the trailing newline.
• Placeholder and hostile gates. Placeholders must return verbatim with the right multiplicity and no marker leaks. A model that drops them falls back to the original field bytes.
• Scope gate. With a single column selected and the header skipped, every field outside that scope must be byte-identical, and no unit may even be collected from outside it.
• Quoting gate. A translation that contains the delimiter or a quote character must be re-quoted correctly rather than allowed to split a row and corrupt the file.

Known limitations

• Every gate is offline.
• Comma, semicolon and pipe delimited files are covered by the fixtures. Other delimiters use the same code path but are not separately gated.

Use the CSV translator → Harness: eval/run_csv_eval.py

Strings seen

36

Selected for translation

27

Deliberately skipped

9

Placeholders protected

5

Comments preserved

9

What the harness checks

• No-op gate. An identity translation must come back byte-for-byte identical, including every comment, the indentation, the quoting style, anchors, block scalars and document markers. This is the gate that matters most for YAML, because a naive load-then-dump silently destroys all of that.
• Placeholder and hostile gates. Placeholders return verbatim with the right multiplicity, the output still parses as YAML, and a marker-dropping model falls back to the original value.
• Structure gate. Keys, key order, nesting, non-string types, block-scalar bytes and every comment are unchanged.
• Quoting gate. A translation containing a colon-space, a trailing comment marker or a comma must still parse and round-trip exactly. Unsafe plain scalars get wrapped in quotes rather than being allowed to corrupt the document.

Known limitations

• Every gate is offline.
• Block scalars are skipped by design. They are preserved byte-exact rather than translated, because re-wrapping them safely is not something we are willing to guess at.

Use the YAML translator → Harness: eval/run_yaml_eval.py

Text units translated

31

Coverage

100.0%

URLs and emails altered

0

What the harness checks

• No-op gate. An identity translation must be byte-for-byte identical, including line endings, the byte-order mark, blank lines and indentation.
• Skeleton gate. After a real translation the line count, the position of every blank line, and the leading and trailing whitespace of every line must be unchanged. A translated file has to paste back into whatever produced it.
• Coverage and do-not-translate. At least 99% of text lines must come back translated, and every URL, email address and file path must survive verbatim.

Known limitations

• The published live run targets French.
• As with Word, this measures fidelity rather than fluency.

Use the Plain text (.txt) translator → Harness: eval/run_txt_eval.py