Architecture

This page is for developers who want to understand how Libris works before changing it: the processes, the path a book takes, the rules the code never breaks, the data model, how jobs run, and the security design. For running Libris, see Docker and operations; for contributing, see development.

Components

Code map

The path of a book

1. Import. Files are uploaded to an import session (/api/imports), inspected, then committed: each becomes a volume (EPUB) or a chapter (text formats) of a series. The automation API writes the same rows directly.
2. Analysis. Chapter summaries, characters, relations and narrative state, then the Book Bible, stored as memories in SQL. By default the passages are analysed side by side, then each is reconciled with what precedes it (parallel analysis); the strict mode goes passage by passage. Translation starts once the analysis of the volume is complete. Its last call proposes, once per volume, values for the fields its style sheet leaves open (app.engines.style_proposals, prompt style_proposal): a bounded sample of the source and the Book Bible in, one allowed value per field with a confidence and a quotation out, kept in projects.config["style_proposals"] and applied by the autopilot from AUTOPILOT_STYLE_MIN_CONFIDENCE on, by a person otherwise. It never writes over a field a person or the series decided and never stops the job.
3. Translation. Each passage is translated with a context built from the book’s memory, then improved depending on the quality level: reviewed, and revised when the review found something. At maximum quality it is polished first, so that the review reads the polished text; a polished paragraph that lost more than a quarter of its words keeps its previous text, and a polish that adds an error of the automatic checks is refused whole (engines/translation/polishing.py). The passage’s stage says how far it went — translated, polished, reviewed, revised, then done — and a resumed job goes on from there; only a translated passage is polished.
4. Whole-book steps. Global consistency, final review, and under the autopilot the convergence rounds (recovery ladder, final review, AI arbitration). See autopilot.
5. Output. Exports from the interface, or the stored result of an automation request.

Every step writes to SQL first. External systems (OpenViking, providers, webhooks) receive copies and never hold anything that cannot be rebuilt.

Invariants

These rules hold everywhere in the code; changes must keep them.

• SQL is the source of truth. Files under DATA_DIR are referenced by SQL rows at paths Libris chooses; nothing is located by a name taken from an upload. External memory is a rebuildable copy, and an external failure never removes a local result.
• Human work wins. A passage corrected or validated by a person is never overwritten by a job, the autopilot, or a chapter replacement (unless the person or the API client explicitly discards it). A validated human correction is the top priority in the context of later passages.
• Writes are fenced. A job writes only while it holds its lease; every write checks the lease and the passage revision. A late answer from a job that lost its lease is refused, even if the provider finished the call.
• Versions, not overwrites. Every translation of a passage is a new version with its origin; the active version changes only when allowed.
• Nothing waits forever. Under the autopilot every job ends completed or failed; every automation request ends with a final status.
• Model output is data. Generated HTML is never accepted as DOM structure; book text in prompts cannot close a prompt section; web results are untrusted hints.

Important transactions

1. Saving a translation: check the job’s lease, compare the passage revision, insert a version, switch the active version only if allowed. Memory events for the send queue are written in the same transaction.
2. Human correction: access check, expected revision required, unit and code validation, new version, memory updated with priority when validated, commit.
3. Resuming: the previous lease holder is invalidated; its late writes are refused.
4. Synchronizing: SQL stays canonical; a lost write acknowledgement is replayed on the same stable URI.
5. Deleting a volume of a series: the series lock (lock_series, a PostgreSQL advisory lock) before the deletion, whose cascade reaches the series memory, then the deletion and refresh_series in one transaction: parallel deletions of a series’ volumes queue instead of crossing.

An HTTP answer received just before a crash may be computed again if it was not committed: Libris guarantees that committed results persist, not that a remote inference runs exactly once.

From source to passages

A unit is one translatable piece of text; a passage (a segments row) is a group of units and the unit of every model call.

• EPUB. XHTML and NCX documents are working sections; semantic subdivisions are kept in the units and in Segment.section. The original spine order is stored explicitly. Anchors are deterministic: resource, XPath and field type. A leaf block (p, li, td…) forms a unit; inside a parent that mixes text and blocks, each run of text and inline elements between two blocks forms a run unit. Ruby readings (rt, rp), code, formulas and preformatted text are immutable markers; SVG <text> and aria-label are translated; pre, MathML and kept SVG titles are listed in book_info.untranslated.
• Chapter kinds. Chapter.kind separates the story (narrative), documents outside the linear reading (auxiliary, linear="no", placed after the story), navigation (navigation: nav document, NCX) and metadata (metadata: dc:description and short dc:subject of the OPF, translated like passages).
• Text formats. A TXT, Markdown, HTML or DOCX file is one chapter (unless it is split at its headings, below), read as blocks (headings, paragraphs, list items, quotes); code, Markdown tables, <pre>, rules and scene separators are kept as they are. The layout (blank lines, indentation, separators, block markup such as ## , - , > ) is stored in Chapter.import_meta["layout"], never in the translated text. Markdown inline formatting stays in the text; HTML and DOCX inline formatting is flattened. HTML is parsed without network access (comments, scripts, styles, <nav> and forms ignored, entity declarations refused). DOCX goes through the same archive and XML checks as EPUB; headings come from paragraph styles, and tables and text boxes are included. JSON chapters from the automation API go through the same text path. The header a webnovel grabber writes above a chapter — the file name as a slug, the address the chapter was taken from, the site’s handle, the credits of the team that translated it — is kept in the layout as it is (text.py: header_lines): never translated, never sent to a model. Only the leading lines that say what they are count, and the scan stops at the first line that reads like the work, so a heading or an epigraph is never taken for a credit.
• Lines of a source site. engines/ingestion/boilerplate.py finds the lines a site writes into every chapter: the inspection keeps the short lines among the first and last EDGE_PARAGRAPHS (5) paragraphs of each chapter (meta.edges, never sent to the browser), and the session’s proposal (boilerplate) names a line standing in at least MIN_CHAPTERS (3) chapters and MIN_SHARE (30 %) of them, spaces and case aside. The lines the person keeps ticked (exclude_lines of the commit) are recorded on the volume (Project.import_meta["excluded_lines"]), and store.add_chapters — the one path of every text chapter added later, from the assistant, the automation API or a source watch — leaves them out of the same edges before the chapter is counted and stored. The chapter records what it lost (Chapter.import_meta["excluded_lines"], reapplied when an archive cuts it again) and its layout keeps them as {"fixed": …, "excluded": true}: the source can be rebuilt, the translated exports skip them.
• Split by headings. engines/ingestion/split.py finds the chapters of one TXT, Markdown or DOCX file: heading styles first (DOCX styles, Markdown #), then heading lines read by naming.chapter_heading (Chapter 12, CHAPTER XII, 第12章, Prologue…), then numbered lines that follow each other. A run of headings with no text between them (a table of contents) is skipped, headings whose number goes back stay in the previous chapter, and the text before the first heading becomes a front matter chapter. Each part is then stored and read as the file it would be if uploaded alone (a line range of the TXT or Markdown file, a minimal DOCX of its paragraphs), so numbering, deduplication, exports and project archives treat it like any other chapter file. The import assistant receives the proposal with the inspection and sends the chosen boundaries back; the server only accepts starts of lines (or paragraphs) of the stored file.
• Chapter map. engines/ingestion/mapping.py reads what a file name or a heading line says a chapter is: its role (called kind in the chapter map of the APIs, unrelated to Chapter.kind above; a chapter or a special: prologue, interlude, side story, extra, epilogue, afterword, author’s note, front matter, in the usual English, French, Spanish, German, Chinese, Japanese and Korean forms), its number, its part (Part 2, (2/2), partie 2, 12a, （下）) and a label (Ayla in Interlude – Ayla), with a confidence and a reason. The chapters of an EPUB are read the same way, one heading at a time and no more: the spine already gives the reading order, so the adapter only names each narrative chapter and leaves navigation and auxiliary pages out of the numbering. A batch is read together (map_names: 12.1/12.2 are the parts of chapter 12 when the batch has both and no chapter 12, or when the volume’s chapter 12 already has parts). A special’s number is its own (Interlude 2): the chapter numbering only orders chapters. order_batch puts a batch in reading order: numbered chapters by (number, part), each special after the chapter it follows in the file order (the JSON array, the split file, the rows of the import assistant), a leading prologue or front matter first, a trailing epilogue or afterword last. Labels (display_label) are localized for the interface and the exports.
• Reading position. Chapter.position, and the passages’ Segment.position laid out in the same order, are the only order of a volume: context, memories and no-spoiler rules (a passage sees what has a lower position), parallel analysis, quality rankings, follow-up scopes and exports never sort by number. store.insert_chapter places a new chapter with keys derived from the chapters already there (reading_keys: a numbered chapter is (number, 1, part); a special takes the number of the chapter before it, then 2, so it stays after all the parts of that chapter; before the first numbered chapter, front matter then prologue; an epilogue or afterword that no numbered chapter follows starts the tail). A numbered chapter goes after the last key at most its own, so a late part 2 lands right after its part 1 and gets it as preceding context; a special goes after the chapter it followed in its batch; an explicit after (chapter id, external id, number or start) or position wins. Inserting shifts the passages after it (shift_positions) and marks the later chapters context_stale. store.move_chapter moves a chapter (session API POST /api/projects/{id}/chapters/{cid}/move): the passages of the chapters between the old and new place are laid out again in their new order (remap_positions moves memories, relations, first appearances, summaries, narrative states and outbox events with them) and those chapters are marked context_stale. An EPUB volume keeps the order of its spine. A chapter sent again is matched by external id, then by role, number and part (a lone chapter and its part 1 are the same), then, for an unnumbered special, by role and label or title.
• Identifiers. Units of text chapters derive from a virtual resource (txt/<hash>, json/<hash>…, from the volume and the chapter number or external id, with the part and role for parts and specials) and the line index: importing the same chapter again gives the same identifiers, and a replaced chapter keeps the identifiers of unchanged lines.
• Long paragraphs can be split at language boundaries and reassembled before rebuilding. CJK paragraphs are cut on 。！？… (closing quotes included), then on clauses, then at the limit.
• Passage size is chosen at import time: PASSAGE_MAX_CHARS (3500 by default), a volume’s config.passage_max_chars for chapters added later, or an import’s own setting. It is recorded with the source (book_info.passage_max_chars for an EPUB, import_meta.passage_max_chars for a text chapter), so a project archive is always cut again with the size of its import. Changing the setting never re-cuts an existing book.
• Segmentation version. book_info.segmentation records the EPUB segmentation algorithm (currently 3: a navigation document outside the spine, EPUB 3 nav or NCX, is no longer a section). Books imported with an earlier version keep their units, and a project archive without the field is restored with version 1.
• Translation memory key. Segment.source_key is the SHA-256 of the units normalized with NFKC, collapsed spaces and markers included. A unit read as verse (engines/translation/verse.py) keeps its line breaks, so the same words in one line of prose have another key.

On export, translated documents receive the target language and direction (dir="rtl" for Arabic, Hebrew, Persian, Urdu…, and the spine’s page-progression-direction in EPUB 3). Elements that declared the source language switch to the target language; elements in a third language keep theirs.

Data model

SQL tables, grouped by purpose. Column types for documents are SQLAlchemy JSON (see JSON rather than JSONB).

Library

Series.paused_at is independent from archiving. It prevents new paid launches and admissions, including source-watch autostarts, while leaving reading, editing and export available. Series-wide control persists suspension before visiting volumes; refusals are reported individually, and excess resumes become durable queue intentions. Shared access to a volume never grants ownership of its series.

The integrated reader reads one chapter through /api/projects/{pid}/read/{chapter_id} with bounded pagination. It writes only through the ordinary /api/segments/{sid} revision check: there is no parallel reader-specific history, lock or correction implementation.

Book memory

Series memory

Jobs and runs

Accounts, automation and settings

JSON rather than JSONB

Document columns (projects.bible, jobs.checkpoint, llm_requests.messages, segments.units…) use SQLAlchemy JSON, which is json in PostgreSQL. This is deliberate:

• most document content is read by row and used in Python; isolated SQL predicates do not require converting all stored documents to another type;
• jsonb reorders object keys. Objects read back from the database are serialized into prompts, so a new key order would change prompt bytes, invalidate the response cache and the providers’ prefix cache, and make every book in progress pay its calls again;
• converting would rewrite every table under an exclusive lock for no benefit.

For a hand-written query that needs a JSONB operator, cast locally: column::jsonb ? 'key'. If a feature needs indexed document searches, evaluate that column separately, including its serialization/cache contract; a dedicated migration and an appropriate index must justify the change.

Context and series rules

Priority. Instructions > validated human decisions > locked glossary > locked series terms > validated structured data > external retrieval > automatic summaries > neighbouring passages > inferences.

Selection. The glossary, the character sheets, their relations, the series terms, decisions and identities of a passage are those whose names the passage or its neighbours say, looked for as the source language inflects them (engines/context/selection.py): the whole name and the short endings of the language’s cases or suffixes (Slavic, Baltic, Greek, Finnic, Hungarian, Turkic, German, English plurals and possessives, Dutch and Scandinavian, Romance plurals, Arabic and Hebrew prefixes), never a truncated start. A capitalised name is matched in its case, or all in capitals. The prompt, its overlap filter and the output check use the same source-language pattern (selection.said_pattern), so a term the check enforces is always in the prompt; scripts without spaces keep builder.name_pattern. Without a language, builder.mentioned keeps the exact rule (timeline, anchors, autopilot counts). A passage told in pronouns (“She laughed softly.”, or a Japanese sentence without a subject) also gets the sheets of the characters of its scene (selection.scene_characters): those the analysis of the passage lists (not for an analysis), those the events of the passages just before say know something, then those named last in the 12 earlier passages of the chapter, most recent first; at most 4, only names that point to one character, with their relations between characters of the scene only. They go through the same dated sheets as the others (chronology.Sheets): what the story established by the passage. Each carries its reason (why) in the context inspector and gives way before the sheets of the characters named.

Internal retrieval. The internal backend (and the database part of hybrid) scores each memory the passage may read by the features it shares with the passage, its neighbours and the names it mentions (engines/context/lexical.py): words of three letters or more less the frequent function words of the main languages, and in scripts without spaces the pairs of adjacent characters less pairs of particles or grammar characters; a memory’s field names and enumerations (summary, kind: event) never count. The share of the features of the smaller side decides, as before.

Budget. The provider’s window minus the reserved output, the operation’s response schema and a safety margin checked by llm.complete. Sizes are measured on the serialized sections (escapes and tags included); input_estimate is exactly what llm.complete compares with the window. If the passage and its mandatory rules do not fit, the error gives the numbers (window, output, schema, system prompt, passage, rules) and the window that would suffice. A first translation is then cut at sentence boundaries into parts sized for the window (half of what remains after the rules; the other half goes to the neighbourhood), translated with their context and reassembled; a revision only cuts between whole units. A passage that fits but leaves no room for an excerpt of its neighbours is treated the same way: a first translation is cut in parts, the arbitration falls back to its reduced context.

The mandatory parts are the system prompt, the user rules, the locked glossary (each term with the first 160 characters of its description), the passage, the confirmed identities, SERIES_CONVENTIONS with its locked terms, its human decisions and identities, and the operation’s material (extra: current translation, critiques, web evidence…). The optional context takes three quarters of what the window leaves after them, never less than 12 000 when the room allows it, up to the ceiling Context budget (context_budget, 32 000 by default, sized for a 64k window; a stored 12 000, the former default the settings form saved, reads as 32 000). It applies to every backend. estimate_tokens stays the conservative byte count that keeps the window safe. The optional items are kept by rank, then authority and relevance (engines/context/budget.py): validated human decisions of the memory (up to a quarter of the budget; beyond, they wait for the chapter state’s rank), the nearest neighbour on each side, the series memory of the earlier volumes (SERIES_MEMORY, within builder.series_room), a person’s unlocked series decisions (up to a quarter; beyond, with the terms), the Book Bible validated or not and the story so far (STORY_SO_FAR; if they would take more than half, at the chapter state’s rank), the sheets of the characters named (this book’s and the earlier volumes’ SERIES_CHARACTERS, validated or decided first) and the identities of the parallel analysis, the unlocked terms of the book and of the series, the state of the chapter, the chapter so far (CHAPTER_SO_FAR, its editorial context for an analysis) and any other section, the sheets of the characters of the scene, the relations, the farther neighbours, and last the retrieved memory (within Retrieval budget for OpenViking). The unlocked terms of SERIES_CONVENTIONS are therefore bounded: the kept ones stay in that section, the others are listed as SERIES_CONVENTIONS.terms. Everything left out, the farther neighbours included (by segment_ids), is listed in the inspector’s discarded with its reason, and budget sums it up (room, cap, optional, used, dropped per section).

Reduced context (recovery.reduced_messages: rung 4 of the recovery ladder, an arbitration too large for its window). No neighbours, book context, relations or retrieval: the user rules of a full context (the style sheet and the job’s instruction included), the locked terms the passage says, the passage and the operation’s material, then a compact memory of what the passage itself names, chosen by the full context’s rules (builder.reduced_memory): the confirmed identities and series conventions a full context declares mandatory, a minimal sheet of each character (aliases, gender, pronouns, register, decided_by_reader), the same of each character an earlier volume of the series knew (SERIES_CHARACTERS) and the unlocked glossary. It is bounded (8 sheets, 12 identities, 30 terms of each kind, 4 series decisions, trimmed fields) and kept section by section, in that order, as long as the provider’s window has room: on a short window the memory gives way before the rules.

Neighbourhood. The nearest passage on each side is served first so that a passage is never isolated; the farther ones only once the memory is placed. They get at most 60% of the optional budget (two thirds of it for what precedes) as soon as other material is a candidate (character sheets, glossary, chapter state, memory). A neighbour that is too long is cut to an excerpt (end of the previous passage, start of the next one) rather than dropped. Instructions and the mandatory glossary are never dropped to hide an overflow. Dropped elements and the reason are recorded in the context inspector.

Narrative state versus editorial knowledge. Character sheets and the Book Bible are built from a whole reading; a passage’s prompt only receives what the story established up to that passage (engines/memory/chronology.py, engines/memory/bible.py), whatever the operation:

• Dated sheets. Nothing on a sheet carries a date: the stored analyses (memories, kind analysis) do. Replayed in book order at their positions, the characters they name say when each name joined a sheet and when each field (gender, role, description, register, free-text relationships, translation notes) was first stated. Names given together for one character join from that passage on; before it each group is a persona with its own names and facts, shown under the name the story gives by then (a sheet renamed “Mira Voss” at the reveal is “Graymask” before it). EDITORIAL_CHARACTERS, the analysis’ CHARACTER_REGISTRY (the 100 personas named most recently), relation endpoints and the reduced context all read this view. A translation, review or arbitration sees what the analyses up to its passage established (its own included: its text says it); an analysis, what the passages strictly before it did. The analyses are cached per book and read again only for what changed, and positions follow the passages when chapters are inserted.
• Human decisions are not dated: the fields of decided, the names of an identity_validated sheet and the whole of a validated sheet apply at every passage.
• revealed_later, never the content: a gender the story only settles later (not when the next two passages, which the prompt shows, state it) is left out with the instruction to keep the source’s ambiguity; names a person joined that the story only ties later are named as such, to use without disclosing.
• Book Bible. EDITORIAL_BOOK_CONTEXT holds its fields that do not tell the story (title, author, genre, tone, narration, tense, audience, guidelines, wordplay, honorifics, conventions), the same for every passage; never the summary, which knows the ending, even in a Bible a person validated (its decisions are all in those fields, sent with human authority). A translation receives instead STORY_SO_FAR, the summaries of the three chapters before its own (600 characters each), and CHAPTER_SO_FAR, its chapter’s rolling summary as the passage before it left it (900 characters). An analysis continues its chapter’s summary as the passage before it left it (EDITORIAL_CHAPTER_CONTEXT).
• Facts no analysis states (a sheet older than its analyses, analyses deleted with a replaced chapter) count from the character’s first appearance, as before dating. A forced analysis (analysis.forget) marks the sheets a person did not validate reanalysed: what they hold from the old analysis waits until the new one states it again.

Series. A volume inherits from earlier volumes only: same series, same owner, lower volume number, and the same language pair compared on the primary subtag (en-US ≈ en). A continuous webnovel flow or an unnumbered volume has no earlier volume: it relies on its own chapters, read in order. SERIES_CONVENTIONS carries the terms, the human decisions, and known_identities (characters met in earlier volumes, under the names those volumes’ own sheets used, the most recent volume’s name first: never the series identity’s name, which a later volume analysed first may have given). Term priority: explicit instruction > validated human decision > locked volume term (or an audited series_override) > locked series term > accepted series term > automatic proposal; for one term, a locked choice beats any unlocked one, then the most recent volume wins. A person’s series decision, even unlocked, beats a book term that is neither locked nor marked series_override; such a book term is treated as an automatic proposal, whoever added it. A volume’s series_override is its own: later volumes keep the series convention (earlier volumes’ terms marked so are left out, as in the series glossary). The prompt, the output check and GET /api/projects/{id}/glossary/effective apply the same precedence (builder.outranks_book). Locked series terms are checked in the output like the book’s locked glossary, unless the book locks the same term differently. A person’s series decisions are written in the series’ languages: a volume of another pair (primary subtags compared) neither receives nor is checked on them, and a propagation through the series (engines/memory/propagation.py) only rewrites the volumes of the entry volume’s pair; the open locked_term alerts of the passages it rewrites are computed again on their new text (closed when the term is now there, restated when another locked term is still missing).

Series memory in the prompt (engines/series/memory.py). The same earlier volumes, and nothing else — never the Series Bible derived from all the volumes — give every prompt of a volume (translation, reviews, revision, final review, arbitration, polishing, quality judge, analysis, questions) two optional sections:

• SERIES_MEMORY, the same for every passage of the volume: the Series Bible a person validated, else the conventions of the earlier volumes’ bibles (honorifics, formatting, translation guidelines, wordplay), and previously, the earlier volumes’ summaries, the most recent first (at most four; a person’s chronology entry replaces a volume’s own summary), the most recent with ending, the running summary of its last analysed chapter;
• SERIES_CHARACTERS, one per character of the passage an earlier volume knew (matched on those volumes’ names), as the most recent earlier volume left them: names and aliases, role, description, gender and pronouns (named in decided_by_reader when a person decided them in any earlier volume, whose value then wins), register and voice, the latest relation with each other character of the passage (the most recent earlier volume stating the pair, then its latest passage; at most three) and the forms of address an earlier volume’s style sheet decided between them.

A validated Series Bible is a person’s text over the whole series. What it dates reaches the earlier volumes only: a chronology entry by its volume, a character by its first volume, any entry naming a volume (project_id, volume, first_volume) read at or after this volume is left out. What it does not date (universe, conventions, a note) is sent — it states the series’ own rules and a person answers for it — except a text a later volume wrote word for word (its bible, its characters’ names and sheets), which is what validating the derived bible without rewriting it leaves there; a text a person wrote themselves cannot be dated and is sent: what must not reach an earlier volume belongs in a dated entry. Its characters’ notes (role, description) join their SERIES_CHARACTERS entry, under the names of the earlier volumes; place names without a note are left out.

Budget: SERIES_MEMORY is served right after the nearest neighbouring passages (engines/context/budget.py), before every other memory, and filled by priority (the latest summary, the validated bible, the conventions, the ending, the older summaries) within a fifth of the optional context _assemble will have (builder.series_room, by the same budget.optional_budget, at most 2,400 estimated tokens): once any other memory competes, the neighbourhood takes at most three fifths of it, so the series memory always fits in what the nearest neighbours leave and never cuts into their share; on a short window it shrinks, and it is left out when not even its core fits. A character’s entry (at most 600, eight characters) is served with the sheets of the characters named (relevance 0.97 against this book’s 1), with the authority of a validated sheet (4) when a person decided part of it. Measured (tests/test_series_context.py) with a 64k window, 16k of output reserved, a passage of the default size and a deliberately heavy volume 1 (a long summary, a cast of six with long sheets, relations and decisions): the prompt of volume 2 grows by about 4,800 estimated tokens (about 1,200 real ones; SERIES_MEMORY 2,156, six characters of 374 to 516); the whole request (about 36,200 in, 16,384 out) stays under 53,000 of the 65,536.

Shared glossary. A series can follow one shared glossary (the terminology of a universe common to several series). Its accepted terms are the broadest level: book > series (a person’s series decisions, then earlier volumes) > shared glossary. A shared term only applies where the series does not decide the term, except that a locked shared term beats an unlocked term an earlier volume proposed; a person’s series decision and a book term that is locked or marked series_override always win. Shared terms travel in SERIES_CONVENTIONS.terms with origin: shared_glossary; the locked ones are enforced and checked in the output like locked series terms, and the autopilot withdraws a proposed book term that contradicts one of them. The autopilot never decides a book term a person added, corrected or imported, accepted or not, and the analysis never proposes again a term a person removed from the book, nor a term the book already holds in another case; both read a person’s acts back from the audit journal (glossary.term_decided, glossary.term_removed, engines/memory/store.py), so the glossary table needs no origin column. GET /api/projects/{id}/glossary/effective shows, for each term, the level that wins and what it replaces.

Consistency check. A sample is a subject (an accepted term or a character) and up to four passages that mention it, its first occurrence among them. It is sent with the user rules (the book’s instructions, the job’s instruction, the style sheet as rules, and a passage’s own rules and its chapter’s), the active relations between the characters its passages mention that were set before the last of them, each with its position (the validated and latest first, at most 12), and the sheets of the other characters there whose gender or pronouns a person decided (at most 8); a character subject’s sheet names those fields decided_by_reader, as a translation’s context does. A remark is stored on its passage as <unit> : description → suggestion, like an automatic check: the arbitration holds an accepted remark to a correction of that paragraph and of no other.

Glossary files. JSON, CSV and TBX files are read by engines/memory/glossary_files.py into the book, the series or a shared glossary. CSV: encoding from the byte order mark, then UTF-8, then Windows-1252; separator (;, , or tab) detected outside quoted cells; columns mapped from French or English headers, a file without header uses its first two columns, and an explicit column mapping can replace the detection. Whatever the format (and in the glossary forms of the API), a source and a translation are kept with single spaces: no spaces at the edges, no-break and repeated spaces made one; a blank one is an invalid row. Every import can be previewed first: new terms, unchanged terms, conflicts with the terms in place (matched case-insensitively), repeated sources (the first row counts) and invalid rows, with the strategy applied to conflicts: skip (keep, the default), replace (unlocked terms only) or replace_all. Applying refuses a file with an invalid row unless skip_invalid is set. CSV exports can use ; and a byte order mark for spreadsheets; a cell starting with =, +, - or @ is quoted so that it never becomes a formula. A validated human correction of a machine translation records its short replacements and the names in the passage; a later volume that mentions those names receives them in SERIES_CONVENTIONS.human_decisions, each replacement once. A correction saved, then validated, is compared with the last machine version it corrected. Names are runs of capitalised words in any script with capitals (É, Ł, Д), katakana runs, and, for scripts without capitals, the names and aliases of the book’s sheets that the passage says. A substitution a person keeps making (engines/memory/patterns.py) is target text, which no glossary term (chosen by the source text of a passage) carries to the model: made a rule with POST /api/projects/{id}/style/wording, it joins the wording of the series’ style sheet (the book’s, outside a series or when the series is not the person’s) and reaches every prompt in USER_RULES.style; a volume of another language pair than its series’ does not take the series’ wording.

Translation memory. Before calling the model for a first translation, Libris reuses a finished passage with the same source_key, same owner, same source language (primary subtag) and same target language; a version validated by a person comes first. In a series, only the same book and earlier volumes qualify. A passage of another book qualifies only if both books hold the same style sheet (series included): same value for every field, same form of address for the couples the passage names, and none of the wordings the target sheet replaces. A passage sent back to review by a change of style (style_changed still open) feeds no one. Units are copied only if the marker structure is valid and the locked glossary (series included) is respected. The version’s origin is translation_memory; review, revision and final review then apply normally. A forced rerun always calls the model. With several passages in flight, passages with the same source_key in one job run one after the other, so the second reuses the first instead of racing it.

Prompts. Section contents escape < and >, so book text cannot close a section. load_prompt adds to every prompt, overrides included, an “untrusted data” clause and, for operations that write or review, a register rule (formal or informal address) and the target language’s typography, whose dialogue marks give way to a style sheet that decides them; languages are named (“French (fr)”). The reviewers (review, fused review, final review, arbitration, consistency check, quality judge) of a language whose typography Libris sets itself (app.engines.typography.RULES, French) are told never to report or correct its spacing (the spaces around em dashes included), apostrophes and ellipses or the comma it moves out of a closing guillemet, and to reject a proposal that only concerns them. Every built-in prompt that writes, reviews or decides a passage’s text, and the consistency check, says that a sheet’s decided_by_reader fields are human decisions, and names the sections it is sent (USER_RULES, LOCKED_GLOSSARY, SERIES_CONVENTIONS, SERIES_MEMORY, TECHNICAL_CHECKS, PREVIOUS_CRITIQUES…); tests/test_review_context.py checks both.

Pitfalls of a language pair. app.languages.pitfalls(source, target) lists the mistakes a model makes without seeing them: those of the target language whatever the source (TARGET_PITFALLS, French: dangling participles, double negation, agreement of colours, a colloquial narrator turning formal, the tenses of a passé simple narration and of the speech inside it, the grammar the eye skips, the article before parts of the body), then those of the pair (PAIR_PITFALLS, English → French: calques, false friends, anglicisms, the progressive aspect, the object pronoun English states, the serial comma, sounds named instead of written). The other targets with a list are English (articles and number from a language without them, one narrative tense, pronouns a source leaves out, spoken dialogue), Spanish, German, Italian, Portuguese, Japanese, Chinese and Korean (the grammar and the translationese of each); the other pairs are Japanese, Chinese and Korean → English (honorifics and address terms, particles and onomatopoeia, set phrases, cultivation terms, speech levels) and English → Spanish, German, Italian, Portuguese, Japanese, Chinese and Korean (false friends, calques, idioms). Keys are primary subtags; a pair nobody wrote them for gets nothing. The list is appended to the prompts of the operations that write or judge a passage — translation (recovery and comparison included), revision, fused review, polishing, review, final review, arbitration, quality judge — each with its own instruction: the writer avoids them, the polisher corrects them, a reviewer looks for each one and proposes no correction that makes one, the arbiter accepts a proposal that removes one and never a correction that makes one. The consistency check, the analysis and the questions do not receive it. USER_RULES, the style sheet included, outrank it: a register the sheet sets is kept. The rules depend on the operation and the language pair only, never on the book, so an operation’s system prompt stays the same bytes for every passage — the prefix a provider’s cache serves. For English → French they add about 1,000 tokens (≈ 4 characters per token) to each writing call and 1,050 to each judging call; Japanese → French about 400; Japanese, Chinese or Korean → English about 300 to 350; English → another listed language 100 to 200; a pair without a list, nothing.

The prompt version is file-v13 or db-vN, suffixed with +rules-v6; tests/test_prompt_versions.py pins the bytes each version stands for, so a prompt file or rule cannot change under an old version. The JSON schema is sent once: in response_format in structured mode, otherwise in a system message.

Quality judge. When a volume or the installation names a judge (app.engines.quality.judge.provider_of: the book’s config["judge_provider_id"], else Settings › Autopilot or QUALITY_JUDGE_PROVIDER), the pipeline has that provider read each passage just before _complete_passage (pipeline._judge): operation quality_judge (prompts/quality_judge.txt, a ReviewResult, the rules and pitfalls of a reviewer). On errors, one translation_revision on the judge’s provider with its points as REVIEW, through the polish’s screen, judged again and kept only with fewer errors (_rework), within QUALITY_REWORK_SHARE of the job (segment_state.QUALITY_REWORK), never in fast quality. The verdict on the text kept replaces the passage’s critique (points marked by: judge), which the status, the score, the arbitration and the final review read. A judge that fails is logged and skipped. python -m app.bench uses the same operation to compare systems on a file or two volumes.

Typography. A model follows the typography rule only in part, so what the machine writes is also set by app.engines.typography in save_version: for French, a no-break space (U+00A0) before ; : ! ?, inside « » and after the dash opening a line of dialogue, the apostrophe ’ and the ellipsis …; the English habits a model carries over are undone: a comma inside the closing guillemet before an incise moves after it, a dash glued to a word is spaced, « ? ! » becomes « ?! », a space before the ellipsis of a hanging word goes. Only an existing space is made no-break, except inside guillemets and around a dash; letters, words and inline markers are never changed, and applying it twice changes nothing. unquoted is the one rule that reads the source, whatever the target language: a passage the source does not quote (no quotation mark in any script, no dialogue dash) loses the one pair of quotation marks the model wrapped around all of it — a character’s thought, in italics in the book, made speech. A person’s text (human, restore) and a retained original are stored as they are. Term propagation matches a straight and a curly apostrophe alike. A finished book is never written again, so a book translated before 0.18 is typeset when it is read out instead: delivered_units and delivered_text apply the same rule to the passages whose text is the machine’s (human and retained_source false) in every output — the text, EPUB, bilingual and Word exports (both sides of a Word revision alike), publication, delivery, automation results, the reader, proofreading links and the preview. Nothing is written back; the editor, the reader’s correction fields and the project and series archives carry the stored text.

Answers that are not JSON. A 200 whose body is prose — typically a model asking for clarification instead of answering — is not a schema violation: it is asked again once, told that the reply must be the JSON object only, and the provider’s JSON mode gives way to the next one (json_schema → json_object → plain text, whose schema travels in the system prompt). A capability error (400/422 naming response_format) drops a mode the same way. Both share the budget of three full-price attempts per call (MAX_INVALID_ATTEMPTS), so a valid first answer never costs a second call, and the fingerprint of the request — hence the response cache — does not change.

Answers cut before their end. A JSON object that stops in the middle — the provider’s finish_reason: length, or an object that simply never closes — is the opposite failure: the format was understood, the output budget was not enough. The call is repaired once, keeping the provider’s JSON mode and asking for the same object complete but much shorter; the failed request’s log names the cause and the provider’s max_output_tokens, so an operator can raise it. A provider that keeps running out of room costs that one repair, not the whole invalid budget, and the answer is still never accepted half-parsed.

Keys Libris did not ask for. The schema sent to the provider is closed (json_schema() sets additionalProperties: false), but a provider that does not enforce it may wrap its answer or invent a field. The models a provider answers with (ModelAnswer in app/schemas.py) ignore unknown keys rather than lose the answer, while every field Libris reads keeps its validation: a missing required field is still an invalid answer. The request models of the API (StrictModel) still refuse unknown keys.

Section order (context/prefix.py). Providers with a prefix cache (OpenAI and compatible servers, vLLM, llama.cpp, DeepSeek) only recompute or bill what follows the first byte that differs from an earlier request. The user message is therefore written from the most stable to the most variable: the series memory and the book context (SERIES_MEMORY, EDITORIAL_BOOK_CONTEXT, character register), then what holds for the chapter or job (USER_RULES, chapter context, STORY_SO_FAR), then what the names in the passage select (glossaries, identities, sheets, relations, series conventions and characters), then retrieved memory, chapter state and CHAPTER_SO_FAR, the neighbours, the operation’s material (CURRENT_TRANSLATION, REVIEW…) and finally TARGET_TEXT, always last. scripts/measure_prompt_cost.py measures the effect on a given configuration.

Jobs, leases and the worker

Claiming and leases

A job is claimed in a transaction: pending jobs, waiting jobs whose retry time has come, and running jobs whose lease expired. A provider’s max_concurrency bounds how many jobs use it at once. The lease lasts 60 seconds and is renewed by a heartbeat every WORKER_HEARTBEAT_SECONDS (2) on a dedicated thread pool. Leases are written and compared with the database clock, read in the same statement (jobs/clock.py: clock_timestamp() in PostgreSQL, the host clock with SQLite), so workers with drifting or jumping clocks never steal a live job or believe they lost theirs. Durations inside one process (heartbeat grace, call timeouts) use time.monotonic().

A renewal that finds the job no longer running stops the task: someone paused, cancelled or reclaimed it. A run that took its own job out of running — an unusable analysis, a volume waiting for an earlier one of the series, a budget reached — is the exception: it is already ending on its own terms, and the heartbeat leaves it alone rather than turning a clean stop into a cancellation.

A job with no provider (deleted, or never chosen) is marked blocked with the reason instead of waiting forever.

Fair queue

claim does not take the oldest job. It reads every job it may start and sorts them (jobs/fairness.py):

1. running jobs whose lease expired (their worker stopped), since they already held a slot;
2. the highest effective priority: the job’s priority, plus one level for every QUEUE_PRIORITY_AGING_MINUTES waited since it was queued (launch or resume), up to high;
3. the account (the book’s owner) with the fewest jobs running now, then the API token with the fewest;
4. the account whose last job was taken the longest ago (max(claimed_at)), so accounts of equal load take turns;
5. the time the job was queued, then its creation.

Series take turns (#153). Within one series, the places this order gives the series’ jobs are then handed out in reading order (in_reading_order, engines/series/order.py): volume 1 first, whatever order the volumes were launched in. And a series runs one volume at a time: claim skips a volume while another volume of its series has a live job, or while an earlier volume’s job waits to go on (a retry, its own series wait); the series row it already locks keeps two workers from starting two volumes together. The first volume of a series met in the walk holds the series’ turn even if it cannot start (provider full, quota), so a later volume never goes before it. Syncing the series memory (sync_memory) is not volume work and does not wait. The provider’s capacity thus goes to one volume’s passages side by side (book_parallelism), eight analyses of one volume rather than one on each of eight volumes.

It then walks that order with the existing checks: the provider’s max_concurrency (provider row locked with SKIP LOCKED), then the account’s running quota and the token’s own one, each counted under a SKIP LOCKED lock of the users or api_tokens row so that two workers cannot both take the last place. A job over a quota is skipped and stays pending; nothing about leases, checkpoints or book_parallelism changes. The waiting quota is checked where work enters the queue (a launch, a resume, an automation request, an import that starts its volumes), not by the worker: an accepted request always starts.

GET /api/queue runs the same order without locks to tell each waiting job its place in its provider’s line and what holds it (provider_busy, account_limit, token_limit, retry_scheduled, earlier_volume (a volume waiting for its series’ turn, without next_attempt, or a parallel analysis set aside until an earlier volume of its series finishes, with it), provider_missing, or starting), and whether the caller may change its priority (editable: an administrator, the book’s owner or an editor it is shared with, the rule of PUT /api/queue/{job_id}/priority). A job of a book the caller cannot open is listed as {restricted: true, title, owner, status: "running" | "waiting", mine: false, editable: false}, without job or book identifier, provider, reason or priority: the interface greys it, and the book itself still answers 404. Quotas and aging are runtime settings (app_settings["queue"], see configuration).

Checkpoint and per-passage state

jobs.checkpoint holds only a cursor and counters: step, current, total, segment_id, consecutive_failures, recovery flags, review_targets, the autopilot round and phase. Its size does not depend on the book (always under 4 KB); it is rewritten at each passage and returned by GET /api/projects/{id}/jobs.

What a job settled passage by passage lives in job_segment_state (primary key job_id, step, segment_id, key; idempotent writes):

Progress (project.progress, stats) reads these rows. Once a job has been finished for RETENTION_JOB_STATE_DAYS, only its reviewed rows are kept. Project archives carry these rows with their jobs.

The event loop and threads

All jobs of a worker share one asyncio loop, so a synchronous query or a long CPU loop there would delay the other jobs’ heartbeats and cost them their lease. SQL sessions and work proportional to the book (context preparation, memory scoring, consistency sampling, model call admission and logging, result writes) run in threads, one session per call: eight threads for this work and two reserved for lease renewals, below the connection pool size. A per-job lock serializes the checkpoint’s read-modify-write inside a worker (PostgreSQL already does it with FOR UPDATE; SQLite reads before taking its write lock).

Several passages of one book at once

The parallel analysis, translation, final review and consistency checks process several passages of a book at once, in a sliding window: passages start in book order, and at most N are in flight. N is read again before each start (jobs/concurrency.py: job_parallelism):

1. the provider’s max_concurrency, shared equally (rounded up) among the books using it at that moment, capped by WORKER_BOOK_PARALLELISM (1 makes processing strictly sequential);
2. lowered by the job’s threads (the launch’s, else the volume’s config.threads), never raised: a big book cannot take another book’s share;
3. after a provider outage (HTTP 429, overload, timeout), the resumed job runs at half the width it had (checkpoint.throttle) and regains one passage per minute without a new outage;
4. under a cost budget, the calls in flight are reserved before they start at the price of a reference call (engines/budget.py: parallel_width): the spend plus the reservations stays below the switch threshold, so near the cap the job narrows to one call and the per-call check decides as for a sequential job.

In each process, calls to a provider go through a queue sized to its capacity, served in arrival order, before the database-level admission that bounds all processes together.

The trade-off: a passage’s context only contains what is already saved. A previous neighbour still in flight shows its source only, and its narrative state is missing from CHAPTER_STATE; with N passages in flight, at most the N − 1 previous ones are affected. With WORKER_BOOK_PARALLELISM=1, each passage sees the translation of all the passages before it.

The strict analysis is sequential: each analysis reads the chapter summary, characters and relations left by the previous passages and rewrites the summary “up to this point”. The parallel analysis rebuilds that chronology after the fact instead (below).

Resumption and safety:

• A passage is marked finished only once all its steps are saved. Every write checks the lease and the passage revision; a version already applied is not applied again.
• A pause, a cancellation, a lost lease or a worker shutdown cancels every call in flight (its request becomes interrupted); the first error of one passage (provider outage, authentication) stops the others too. Interrupted passages are not marked, so a resume restarts them from what they saved, without redoing finished ones or emitting events for them.
• The ten-consecutive-failures counter follows the order in which passages finish. It never stops an autopilot job or a pipeline with automatic recovery.
• Lock order. The worker locks its job row (fence) before any passage row. API actions that touch a passage and the book’s live jobs (human correction, kept original, queuing an accepted proposal) first lock those jobs (lock_live_jobs, by increasing id), then the passage, so a passage/job deadlock with the worker cannot happen in PostgreSQL.

Parallel analysis

ANALYSIS_MODE=parallel (the default; a volume’s config.analysis_mode, a launch or an API request may choose strict) analyses a volume in five stages (engines/translation/parallel_analysis.py). The goal is speed without losing anything the strict, chronological analysis gives a passage.

1. Extraction (chapter_extraction, N passages at once). Each passage is analysed on its own: its raw neighbours (the source of the two passages before and after, as in the strict mode), what was known before the job (earlier volumes of the series, confirmed identities, the locked glossary, the memory of passages analysed by an earlier job), never the analysis of another passage of the same run. The prompt asks for every name form as written and for an unresolved event for a pronoun or description it cannot tie to someone. Stored per passage in the job state (extraction).
2. Consolidation (in memory, deterministic). The extractions are applied in book order to a timeline (engines/memory/timeline.py): names tied into identities (with the passage of their first and last naming), relations, proposed terms, the summaries of the passages. Passages already analysed (a follow-up of a webnovel) enter it with their stored analysis. A character joins a known identity only by its own name or by that identity’s canonical name, never by an alias in common: a name two identities carry (“Potter”, “Your Majesty”) resolves to neither, and identities with two known genders are never merged. The written memory (identities.upsert_profiles) follows the same rule, and sends such an alias to the sheet’s proposals instead.
3. Reconciliation (chapter_reconciliation, N passages at once). Each extraction is reviewed with the timeline as it was before that passage: KNOWN_IDENTITIES (with aliases), RECENT_CHARACTERS (the last characters named, for pronouns), KNOWN_RELATIONSHIPS, KNOWN_TERMS, EARLIER_PASSAGES (the summaries of the six passages before, and first of the passages further back where the recent characters were last named), plus everything the strict analysis reads. The model resolves aliases and references, merges entries that denote one person, keeps apart what the story has not tied yet, and writes the rolling chapter summary. Since the timeline is known for every position, these calls run in parallel. ANALYSIS_RECONCILIATION=all (default) reconciles every passage; flagged only those whose extraction left something to resolve.
4. Memory. The reconciled analyses are written in book order by the strict mode’s own code (_store_analysis: identities through upsert_profiles, relations, glossary proposals, chapter summaries, memories and their OpenViking events): the resulting memory has the same shape.
5. Book Bible as a tree: one synthesis per chapter (its passages four by four, as in the strict mode), then merged four by four, level by level, each level in parallel. A volume that already had a bible merges the new chapters’ synthesis into it last. Every call receives the same payload (bible_payload): covers, the one chapter or the first-to-last range of chapters the answer must cover — a node carries its first and last leaf, never a chain of labels — the overview to extend, the character registry, and evidence with evidence_kind saying whether its entries are passage analyses or partial syntheses of the same book, each with the range it covers.

Guarantees:

• No spoiler. The derived sections a passage receives are taken before its own extraction joins the timeline; an extraction reads no other analysis of the run. The raw neighbours are those of the strict mode. The sheets and registry an extraction or reconciliation reads from earlier jobs are dated as well (Narrative state versus editorial knowledge): a passage asked again after the whole book was stored (a catch-up pass) reads only what precedes it, and its analysis moves its chapter’s summary forward only. tests/test_parallel_analysis.py checks every prompt of a run, tests/test_dated_memory.py the catch-up, the forced analysis and the translation prompts.
• Series. Before its reconciliation, a numbered volume waits while an earlier volume of its series has a live analysis job that has not finished (the job goes back to waiting, stop_reason = earlier_volume, and is claimed again every 15 seconds). A paused, blocked or failed earlier volume does not hold it: nothing waits for a person.
• Barrier. Translation starts only when the analysis is complete: every passage holds its analysis and every section is consolidated (#161). analyze checks it after the five stages and runs them again over what is missing (passages given up, Book Bible syntheses given up, passages added meanwhile) — once on the job’s provider, then once per autopilot fallback provider; what is still missing stops the job (blocked, analysis_incomplete) unless ANALYSIS_MIN_COVERAGE was lowered. A translate job checks the same before its plan and runs analyze first when a passage it covers has no analysis. The Book Bible tree counts a chapter as consolidated only when its synthesis and every merge above it succeeded, and its stored syntheses are keyed by the tree they belong to (the chapters and analyses it consolidates), so a catch-up pass never reuses a merge made for other chapters.
• Same result whatever the threads. The views depend only on the stored extractions, applied in book order, never on the order in which calls finish.
• Resumable at every stage. Extractions, reconciliations and Book Bible syntheses are stored as they arrive; the memory stage skips passages already written; a resumed job asks the model only what it has not got, and an identical request is served from the request cache.
• Autopilot. A refused or invalid extraction skips the passage’s analysis (analysis_skipped, as in the strict mode); a failed reconciliation keeps the extraction; both are written to the decision log.

Progress: the checkpoint’s step is extraction, consolidation, reconciliation, memory, then book_bible with level/levels; GET /api/projects/{id} and the /api/v1 status document give it as analysis_phase / progress.analysis.

Evaluation

Without a real model, the design was evaluated on a synthetic serial with known ground truth (backend/tests/analysis_world.py): seven characters, a masked figure revealed half-way through, a nickname introduced after a quarter of the text and used alone afterwards, a name change in the second volume, short forms of full names, pronoun-only passages (some several passages or a chapter break away from their referent), relations and glossary terms. A deterministic simulated analyst answers from its prompt only: it recognises names but links two names, or a pronoun to a person, only when the passage or a context section it received says so. The scores therefore measure what each mode puts in front of each call. scripts/evaluate_analysis_modes.py runs every mode on the same serial:

Three seeds, 60 chapters × 2 volumes each (907 passages in all); recall and precision against the ground truth, spoilers counted in the stored memory and in every prompt sent:

The parallel mode keeps every score of the strict mode and finds far more pronoun referents (the strict rolling summary is per chapter, and its neighbours reach two passages back); reconciling only the flagged passages loses part of the alias resolution, and skipping the reconciliation loses half of it: every passage is reconciled by default. These figures measure the information each call receives, not a real model’s use of it; confirm on a real book with the protocol of development.

scripts/benchmark_analysis.py measures the analysis of a 400-chapter serial (800 passages), every model call answered after 0.2 s, on a provider allowing 16 calls at once:

With 16 calls at once the analysis ends 4.8 times sooner than the strict one, for 1.8 times more calls and 1.07 times more prompt tokens (an extraction carries no memory). Beyond 8 threads the in-process overhead of this benchmark (SQLite, context building) dominates the 0.2 s calls; with a real model taking seconds per call the gain follows the threads more closely (about calls ÷ threads × latency, plus the few levels of the Book Bible tree). The translation stage, which makes most of a book’s calls, is unchanged.

Outages and failures

A provider outage suspends the job as waiting with an exponential delay (the saved Automatic recovery delay, else PROVIDER_RECOVERY_BASE_SECONDS, capped by PROVIDER_RECOVERY_MAX_SECONDS); refused credentials make it blocked. Under the autopilot, the fallback chain takes over after a bounded wait (autopilot). A database outage suspends the job as waiting; a worker shutdown puts it back to pending.

Before each model call of a job, engines/budget.guard compares the spend of the book (usage aggregates plus the requests not rolled up yet) and of the API token that started it with their caps. Near a cap it changes the job’s provider and puts it back to pending, or pauses it (budget_exceeded) with the same fenced transition as a user pause, then stops the running coroutine (JobStopped); calls already in flight finish and are recorded.

Automation requests

A request of the automation API is saved (row, payload file, chapters when the volume is free) before the API answers. The worker’s request dispatcher checks live requests every 2 seconds: it imports the chapters of queued requests once no job is active on the volume, starts their pipeline once no job holds it, and settles running requests from their job’s state (building and storing the result, writing the report, failing stalled or overdue requests). The API also settles a request before answering, so a client never waits for the next pass. Webhooks are sent by a separate worker loop, never by the API.

Settling a running request whose client asked for chapters.translated first looks for the chapters of the request that now have every passage translated and records them as one batch in webhook_events; the webhook loop sends batches before final webhooks, with the same signature, allow-list and backoff.

Following up a volume

New chapters sent to a volume that is already translated (an automation request, or an interface import into an existing volume or the continuous feed) are inserted at their reading position (see From source to passages), and the job started for them carries follow_up_chapters: the new or replaced chapters plus any chapter still missing a translation. Analysis and translation already skip finished passages; with this option the final review targets only passages of those chapters, and the consistency check samples only their occurrences, each against the first occurrence in the volume. Earlier chapters get no model call and keep their text; they only feed the context. When the scope would be the whole volume, the option is left out and the job behaves as a first translation.

Passage quality scores

Every translated passage, and every passage kept in its original, has a score from 0 to 100 in passage_quality. It is computed from what Libris already records, without any model call (app/engines/quality/score.py): the passage starts at 100 and loses points for each signal below, each signal capped so that one kind of problem cannot hide the others.

Signals worth 0 points stay in the stored list (the interface hides them). A passage validated by a person scores 100 (signal validated): someone read it. Bands: good from 85, fair from 70, weak from 50, poor below. Passages under 70 that nobody validated are the “review these first” list.

Kept in step. Session events in app/models/quality.py collect, at each flush, the passages whose scored columns changed, whose alerts were added, changed or deleted, which received an autopilot decision or a failed model call; bulk UPDATE/DELETE statements on passages and alerts are resolved to their passages before they run. The scores of the collected passages are recomputed in before_commit, in the same transaction, with an upsert (INSERT … ON CONFLICT), so concurrent writers never conflict on a row. A passage that loses its translation loses its score. Every reader first repairs what is missing or was computed on an older passage revision (books translated before the scores existed), so no migration of existing data is needed.

Where it is read. GET /api/projects/{id}/quality and GET /api/series/{id}/quality (the series’ readable, non-archived volumes) return the summary (scored, average, minimum, bands, a ten-bucket histogram, to_review), the chapters ranked weakest first (lowest average, then lowest passage; 100 at most, chapters_total counts them all), the 20 passages to review first with their signals and, for a series, each volume’s figures. GET /api/projects/{id}/quality/passages?chapter_id= gives the editor each passage’s score. The summary is also part of GET /api/projects/{id}/completion, of the autopilot report (quality) and of the completion report of the automation API.

Exports and project archives

GET /api/projects/{id}/export/{format}:

allow_source=true exports an unfinished translation with the originals in place of missing passages; without it, an incomplete export answers 409. from_chapter/to_chapter (text formats, epub and docx) keep the chapters between those numbers in reading order, with their parts and the specials between them (the leading specials when the range starts at the first chapter, the trailing ones when it ends at the last). A chapter whose title only names it by its number or kind (generic_title) gets its localized label as heading, in the target language. merge_parts=true (text formats and epub-bilingual; also a field of POST /api/exports/text) joins the consecutive parts of one chapter under one heading; storage stays one chapter per part, which keeps deduplication by external id, follow-up scopes and the per-chapter results of /api/v1 exact, so it is an export option and not the default. The bilingual EPUB marks prologues, epilogues and afterwords with their EPUB structural semantics (epub:type, doc-prologue…). POST /api/exports/text exports several volumes (a series) as one ZIP with a NN - Title/ folder per volume, and POST /api/exports/epub several translated EPUB files. Text rendering follows the stored layout for text sources and gives one paragraph per unit for an EPUB; no internal ⟦…⟧ marker is ever written.

The bilingual EPUB (engines/exports/bilingual.py) is written from the passages, not from the original file, so it exists for every source format. Source and translation are paired per original unit (the fragments of a long paragraph joined again); text chapters follow their stored layout (scene breaks written once, headings, list markers and quotations kept), EPUB chapters give one pair per unit, the chapter title becoming the page heading with its source below it. Images, links and inline styles are left out. Each text carries its lang/xml:lang and, for right-to-left languages, dir="rtl"; the stylesheet uses table display for the two columns and stacks them under 30 em. The same builder serves GET /api/projects/{id}/export/epub-bilingual and the epub-bilingual result format of /api/v1.

Project archive (schema version 3)

translation-project.zip contains project.json and the source files under names Libris sets: sources/<n>.epub|txt|json (every source file of the volume) and, for JSON chapters, texts/<n>.txt. project.json holds the volume, its chapters, passages, versions, memory, glossary, jobs and job state, plus the series (name, kind, authors, author), the source files (file, format, original_name, media_type, sha256, meta), and for each text chapter the file to cut again and the options that give back the same units.

The archive keeps everything that makes up the work on the book: settings (title, series and volume, languages, quality, memory backend, instructions), chapter and passage instructions, translations with their status, stage, critiques and uncertainties, the full version history, glossary, Book Bible and its revisions, characters, merges and links, memories, quality issues, jobs with their checkpoints and per-passage state, and the figures of the model calls (operation, model, tokens, duration, cost, status). It serves as a backup of one volume or to move it to another instance. Export refuses an archive that the import could not read back (MAX_UPLOAD_MB, MAX_UNPACKED_MB), and the message names the setting to raise on both servers.

Restoring (POST /api/projects/import) validates everything before writing anything: only project.json, original.epub (archives of schema 1 and 2) and the sources/… and texts/… names are accepted, none is used as a path; entry count, declared sizes, compression ratio, reads bounded by the declared size, SHA-256 checksums and the consistency between sources and chapters are all checked. An EPUB is imported again from the archive; a text volume is recreated with its recorded resources, so the unit identifiers match, and every passage must have the same source text as in the archive. It all happens in one transaction, and files are removed on failure. The restoring user becomes the owner; the series is found or created by normalized name among theirs; a second continuous flow or an already used volume external_id in the series is refused (409). Owners, members, permissions and providers are never restored, nor are the full prompts and answers of model calls, progress events and the OpenViking send queue. Interrupted jobs come back paused, without provider, so nothing restarts or is billed without an action; an archived book comes back active. Archives of schema versions 1 and 2 remain readable. NOT_ARCHIVED lists the columns deliberately left out; a test fails if a new column is neither archived nor listed.

Series archive (schema version 1)

GET /api/series/{id}/export (series owner only) writes <name>.libris-series.zip, stored rather than deflated: volumes/<n>.zip is the project archive above of each volume of the series in reading order (archived ones included), built by the same code, and series.json holds schema_version, libris_version, exported_at, the series settings (name, kind, authors, author, languages, quality, context_backend, instructions, style, bible, bible_validated, archived_at, created_at), volumes (id on the exporting server, file, sha256, title, archived_at), entities (series identities with merged_into_id), links (series_entity_id, the volume project_id and the entity_id inside that volume’s archive, status, confidence, reason, human), relations and glossary. The archive is written to a temporary file volume after volume and refused (413, naming MAX_UPLOAD_MB, MAX_ENTRIES or the volume) when the import could not read it back.

POST /api/series/import reads the spooled upload from disk entry by entry, in a worker thread. Before anything is written it checks the entry names (series.json, volumes/<n>.zip, nothing else, none used as a path), sizes, compression ratio, every field of series.json, the SHA-256 of each volume archive, each volume archive in full (the checks of POST /api/projects/import), and every reference: identities, merges (no loop), relations, links to a volume and to an entity of that volume, first volumes, and the uniqueness constraints (identity name, glossary source, relation, link, volume external_id, a single continuous flow, no EPUB twice nor already in the owner’s library). A refusal names the field or the volume. The restore is one transaction: a new series of the restoring user — never merged; a taken name becomes Name (2), Name (3)… (renamed_from in the answer) — then each volume through the volume restore, then the series rows, every identifier remapped (volumes, identities, the volume entities of each link, and the identifiers the Series Bible names). Stored files are removed if it fails. NOT_ARCHIVED lists the series columns left out (providers, paused_at), NOT_TRANSPORTED the tables tied to a series that never travel (audit log, attached shared glossary, automation requests); tests fail when a new column or a new table referencing a series is in neither.

Security design

Untrusted files

• ZIP archives (EPUB, DOCX, project archives) are never extracted to paths the file chooses. Paths, duplicate entries, symbolic links, sizes, entry count and the whole-archive compression ratio are checked before anything is unpacked (MAX_UNPACKED_MB, MAX_ENTRIES, MAX_COMPRESSION_RATIO).
• XML parsers refuse entity declarations and never load DTDs or network resources; glossary TBX files use the same reader.
• Request bodies are bounded before they are buffered: 1 MiB without a session or Bearer token, MAX_UPLOAD_MB or API_MAX_PAYLOAD_MB otherwise. EPUBCheck runs are bounded in number and memory.

Error messages

• Messages are written in French where they are raised; when a request prefers English (Accept-Language: en, sent by the interface in English), the exception handlers translate the detail from the catalog in backend/app/i18n.py. Translation only rewrites the message text and never adds internal information. A test fails if a message raised in the code has no English entry.
• Unexpected errors answer with a diagnostic reference that points to the server log, never with a stack trace. Validation errors of the automation API never echo submitted values.

Model output and previews

• Model-generated HTML is never accepted as DOM structure; translations are text reinjected into the original markup.
• Chapter previews are sanitized and shown in a sandboxed iframe under a restrictive Content Security Policy. The interface sends a strict CSP, X-Frame-Options, nosniff and a same-origin referrer policy.

Secrets

• Provider keys, the OpenViking key, token webhook secrets and the saved webhook secret are encrypted at rest with SECRET_KEY (at least 32 characters). They never appear in API answers, traces or exports; the automation API names a provider and model only.
• API tokens are stored as SHA-256 hashes and compared in constant time.

Access

• Libraries are private, with per-book access control, including logs, event streams, versions and exports. Non-administrators see providers without their address. Changing the address or type of a provider that holds a key requires entering the key again, so a stored key is never sent to a new host.
• A shared editor cannot attach a book to a series of the owner that contains books the editor cannot read.
• Session cookies are HTTP-only, SameSite=Strict, and Secure with COOKIE_SECURE=true. State- changing requests are refused from origins outside ALLOWED_ORIGINS and from cross-site browser contexts (Sec-Fetch-Site).
• Failed sign-ins are throttled per client and account (20 failures in five minutes) and per client (200). The throttle and the automation API rate limit live in each process’s memory: they are not a complete internet-facing abuse control.
• /openapi.json (every route) requires a session and can be disabled with OPENAPI_ENABLED=false. The public description of the automation API alone is the file docs/openapi/libris-v1.json, not a route. /metrics exists only when METRICS_TOKEN is set, and requires it.
• Live event streams are bounded per account and per process.

Outbound connections

• Only services an administrator configured are called. Outbound clients ignore proxy environment variables, do not follow redirects, and bound the size of answers (SearXNG: 2 MiB).
• Webhooks go only to allowed hosts, to public addresses (unless an allowed private network), with the address checked just before each call (details).
• Web search queries chosen by the model remain a possible exfiltration channel under prompt injection: enable SearXNG only towards an instance you control.

Containers and supply chain

• Every container runs with no-new-privileges and cap_drop: [ALL]; the database keeps only the five capabilities its entry point needs to own its data and switch to the postgres user. Root file systems are read-only, /tmp is in memory, and large temporary files go to the data volume (TMPDIR=/data/tmp). Application processes run as non-root users.
• PostgreSQL and the Codex bridge are not published on host ports by the supplied Compose file.
• Python dependencies are installed with --require-hashes from hashed lock files (backend/requirements.lock, codex_bridge/requirements.lock, generated by scripts/hash_lock.py): a modified or substituted package is refused at image build time.

The operator’s side (HTTPS, firewalling, backups, provider privacy terms, retention of model traces) is covered in operations and in the repository’s security policy.

Extending

The ContextProvider boundary lets other memory engines be added without touching the translation engine. Generation models are configurable, and no assumption about a specific model family or context window is encoded in the prompts.