Operations

This page is for administrators who keep Libris running: updating it without losing work, sizing its capacity, monitoring it, keeping its database small, controlling model costs and solving common problems. Installation is covered in the Docker guide, every setting in the configuration reference, and backups in backup and restore.

All commands run from the Libris directory unless stated otherwise.

Update Libris

The simple customer path is the one in the Docker guide: back up, then run the registry bootstrap again. It needs neither Git nor a source archive. It runs the migrations, then restarts the web application and the worker; running books resume from their last checkpoint, so no finished work is lost, but the passages being translated at that moment are sent again. The migrations run just before the worker is replaced, while the previous one may still be working: pause the running books first, as the Docker guide says, or use scripts/deploy.sh.

For a development checkout, scripts/deploy.sh gives finer control over when the worker restarts. Unlike the registry bootstrap, it does not change the image named in .env: set LIBRIS_IMAGE there to the new release first.

sed -i 's|^LIBRIS_IMAGE=.*|LIBRIS_IMAGE=registry.libris-translate.com/libris/libris:<new version>|' .env
LIBRIS_DEPLOY_SOURCE=pull ./scripts/deploy.sh --worker-when-idle

A wrong mode or LIBRIS_DEPLOY_SOURCE exits with status 2.

If the new version fails to start, the script puts the previous image back (tagged libris:rollback) and restarts it. It cannot undo a migration: for that, see roll back a failed update.

With deploy.sh, the migrations never run while a worker of the previous version is running. Use --api-only for a quick fix of the web side, then finish with --worker-when-idle when books are done, so that the worker does not keep running an older version for long.

Production deployment script

deploy/libris-production-deploy deploys a tagged release, already loaded as images, onto a dedicated host. The project’s own pipeline no longer deploys anything (a tag only publishes images and a release, see development): the script stays for an operator who drives a host from a pipeline of their own, and is of no use to an installation made with the installer. Install it as /usr/local/sbin/libris-production-deploy.

Host-specific settings can be kept in /etc/libris-production.conf (root-owned, mode 0600), using deploy/libris-production.conf.example. This trusted shell file is sourced before the script resolves its settings, including for rollback and pruning; its assignments override the environment. It is not the application’s .env and must never be writable by an application account.

libris-production-deploy COMMIT VERSION APP_IMAGE_ID CODEX_IMAGE_ID   # deploy images already loaded
libris-production-deploy --rollback                                  # back to the images it replaced
libris-production-deploy --check-compose                             # is the installed Compose file the deployed version's?
libris-production-deploy --prune-images [--dry-run]                  # remove old Libris images

A deployment checks the images’ version and revision labels, refuses to go back to an older version (use --rollback), dumps the database into $LIBRIS_PRODUCTION_BASE/backups/ (last five kept), stops the application, installs the Compose file shipped inside the new image, runs the migrations, starts the API and the Codex bridge, checks /health, and only then starts the worker. If anything fails before the schema changed, it restarts the previous images; after a schema change it leaves the services stopped and names the dump to restore.

Change docker-compose.yml in the repository, never on the host: host-specific values belong in the .env (BIND_ADDRESS, PORT…) or in the override file, and the replaced file is kept as docker-compose.yml.before-<version>. When disk space is short, delete old files inside backups/, never the directory: without its Compose file the next deployment stops with Production configuration is not provisioned (exit 65) before touching anything. Reinstall /usr/local/sbin/libris-production-deploy whenever deploy/libris-production-deploy changes, and validate a deployment and a rollback on a disposable stack before trusting a new host.

Each deployment also refreshes /usr/local/sbin/librisctl from the image it deploys (the image carries deploy/librisctl in /app/deploy/), so the operator command line is always the one of the running version.

Operator command line

librisctl is the everyday command line of such a host: one command per task, with the Compose project, the configuration file and the Codex profile resolved exactly as the deployment script resolves them. It is installed as /usr/local/sbin/librisctl by each deployment, or by hand with install -m 0755 deploy/librisctl /usr/local/sbin/librisctl. Run it with no argument for the list, or librisctl <command> --help for one command. Its output is plain text without colour or pager, so it reads the same over SSH, in a script and in a log.

Exit codes: 0 success, 1 a check failed, 64 wrong usage, 65 production is not provisioned on this host or a precondition is not met, 77 refused (a destructive command without --confirm). librisctl down and friends are refused outright: the data volumes are the only copy of the books and of the database.

The common tasks:

librisctl status                     # what is running, and is it healthy?
librisctl doctor                     # everything that should be true, checked in one pass
librisctl restart worker             # the worker stopped taking jobs; books resume from their checkpoints
librisctl logs -f worker             # follow it
librisctl logs --since 30m api worker
librisctl journal -f                 # every service in one chronology, as it happens
librisctl journal export --since 2h --with-containers --output /root/incident.log
librisctl backup                     # a database dump before touching anything
librisctl rollback --confirm         # back to the version the last deployment replaced

Nothing prints the configuration or a key: status shows where the .env is, never what it holds.

Rename the Compose project

An installation made with the installer is renamed by the installer itself (Docker). On a production host driven by libris-production-deploy, the historical project name is epub-translator, and its volumes hold the books and the database, so it cannot simply be renamed. librisctl migrate-project [NAME] --confirm (libris by default) does it in one go: it refuses while a job is active, stops the installation, lets Compose create the new project’s volumes, copies each volume into its twin, compares their sizes, starts the installation under the new name and records it in $LIBRIS_PRODUCTION_BASE/project, which every script then follows. The old volumes are left exactly as they were; stateless service containers are recreated under the stable libris-* names. Removing $LIBRIS_PRODUCTION_BASE/project and running librisctl up returns to the old volumes, and docker volume rm epub-translator_database epub-translator_books epub-translator_codex-state removes them once the new name has proved itself. Add LIBRIS_PROJECT=<name> to the .env as well if you also run plain docker compose commands: the Compose file takes its project name from that variable.

The Compose options for a manual command on such a host, when librisctl is not enough:

docker compose --project-name epub-translator --env-file /opt/libris-production/.env \
  --file /opt/libris-production/docker-compose.yml --profile codex <command>

Capacity and concurrency

Two limits decide how much work runs at once:

• Concurrent books, set on each provider in Settings › LLM providers: how many books may use that provider at the same time, all operations included (analysis, translation, review). Providers are independent: one set to 3 and another set to 1 allow four active books.
• WORKER_BOOK_PARALLELISM: how many passages of one book are in flight at once, for the analysis (in the default parallel mode, ANALYSIS_MODE=parallel) as for the translation and the reviews. The default, 0, uses the provider’s capacity, shared between the books using it. 1 processes one passage at a time. A volume (Passages worked on at once in its settings), a launch or an API request (threads) can only ask for fewer. After a provider answers 429 or is overloaded, the job resumes at half its width and widens again by one passage per minute.

The strict analysis (ANALYSIS_MODE=strict) reads one passage after the other whatever these limits; the parallel mode makes about 1.8 times more analysis calls but ends several times sooner on a long book (see architecture).

When books wait, they do not start oldest first: the fair queue takes the highest priority, then the account and the API token with the fewest jobs running, in turn between accounts, so one person’s backlog does not hold everyone else back. On a shared installation, Settings › Queue (or QUEUE_*, see configuration) can also cap each account’s running and waiting jobs. The Queue page shows each waiting job’s place and what holds it.

A book stays queued while its provider has no free slot. To change a book’s provider mid-way, pause it, choose the new provider in its configuration and resume: the rest of the book uses the new one.

One worker is enough for most installations. It renews a 60-second lease on each job; if the worker stops abruptly, another start picks the job up after the lease expires.

Worker processes

A worker process runs on one CPU core, whatever the number of books it carries. On a machine with spare cores, WORKER_PROCESSES=N (2 to 16; default 1, the single process described above) makes the worker container run the books in N job processes:

• The worker’s main process keeps everything that must run once: licence renewal, watched sources, retention, mail, webhooks, library publication, the memory catalog and OpenViking. It starts the N job processes, starts one again when it dies, and on docker compose stop passes the signal on and waits for them (25 seconds, inside the service’s 30-second stop_grace_period).
• Each job process takes books from the same queue. Leases, the fair queue, a provider’s Concurrent books and “one volume of a series at a time” are kept in the database and hold across processes. A process leaves the next book to the others once it holds more than its share (books under way divided by N, rounded up), so the books spread over the processes.
• If a job process dies, its books are picked up by another one when their 60-second lease expires; the log says process=died … role=jobs then process=started, and status=reclaimed for each book taken over.

Limits to know before raising it:

• One book alone stays on one core: its passages are threads of one process. The setting helps when several books run at once, not a single long one.
• One worker container only. Do not scale the worker service or start a second one: the loops kept in the main process (licence, watched sources, retention) are not protected against running twice.
• Every process has its own database pool: raise max_connections with the number of processes (see the pool settings in .env.example).
• The worker service’s pids_limit: 512 covers up to 8 job processes (a process uses at most about 45 threads); raise it in a Compose override beyond that.
• A demo instance (DEMO_MODE) starts no job process.

The worker also sends memory updates to OpenViking and, when the cleanup is on, removes the OpenViking documents of deleted volumes and series. A removal holds a 5-minute lease too; a waiting one is visible, with its last error, in Settings › Memory · OpenViking › Cleanup log.

Database connections

Every process holds a connection pool of its own, DB_POOL_SIZE + DB_POOL_MAX_OVERFLOW connections (80 by default). PostgreSQL must accept all of them at once:

max_connections >= (API process + worker processes) x (DB_POOL_SIZE + DB_POOL_MAX_OVERFLOW) + 20

With WORKER_PROCESSES=1 there is one worker process. Above 1 there are N children and the parent that supervises them, so N + 1. The 20 are left for librisctl, a psql and the database’s own maintenance. The supplied Compose file reads max_connections from POSTGRES_MAX_CONNECTIONS in .env (200 by default, enough for the defaults with one worker: (1 + 1) x 80 + 20 = 180).

Both processes check it at start-up. When max_connections is below the budget they log db_pool=over_budget max_connections=… needed=… hint=set POSTGRES_MAX_CONNECTIONS=… and start anyway: the failure would then show as pool_exhausted or refused connections under load. Fix it by raising POSTGRES_MAX_CONNECTIONS (and restarting the database service), or by lowering DB_POOL_SIZE and DB_POOL_MAX_OVERFLOW — the pool only needs the sum of the providers’ Concurrent books plus 16.

Monitoring

Health

GET /health answers {"status":"ok","version":"<version>","configuration":"ok"} when the API and its database connection work. It does not test model providers; test those in Settings.

configuration compares the containers with what the image’s own Compose file (/app/deploy/docker-compose.yml) expects: every mount it gives the application (/data, /etc/machine-id, /logs) and a journal that is written. ok; outdated when a mount is missing (the Compose file predates the version); degraded when a feature is off because of it — a journal that cannot be written. The API and the worker say it at start-up (configuration=… check=… in their output). The list of checks, each with the command that fixes it, is in the answer for a signed-in administrator, or for everyone with PUBLIC_HEALTH_DETAILS=true; administrators also see it in Settings › Installation health, and librisctl doctor prints it.

With the same details, openviking is the state the worker last saw (reachable, unreachable, unknown or not_configured), since when, and the send queue (pending, error, next_attempt); never its address. OpenViking is optional: an outage there never changes status nor configuration. See OpenViking.

curl --fail http://127.0.0.1:8088/health

Prometheus metrics

GET /metrics exposes the state of the installation in Prometheus text format. It is disabled until you set METRICS_TOKEN:

sed -i "s|^METRICS_TOKEN=.*|METRICS_TOKEN=$(openssl rand -hex 32)|" .env   # the line exists in a generated .env
grep '^METRICS_TOKEN=' .env                                                  # the value to give Prometheus
docker compose up -d --no-build --wait

Each scrape must send the token as Authorization: Bearer <token>; a browser session is not enough. Without the variable the endpoint answers 404, with a wrong token 401.

scrape_configs:
  - job_name: libris
    scrape_interval: 60s
    metrics_path: /metrics
    scheme: https
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/libris-metrics-token
    static_configs:
      - targets: ["books.example.com"]

Counters are computed from the database: they count everything since installation, are the same in every process and survive restarts. Deleting a book deletes its requests, which Prometheus sees as a counter reset. Use rate() or increase() for a time window. No label contains a book title, text, id or provider address. The output is cached for 10 seconds, so scraping every 30 to 60 seconds is enough.

Useful alerts:

Behind a reverse proxy, expose /metrics only to your Prometheus network if you can.

Statistics in the interface

Statistics shows token usage by model, and each book shows what it has spent. Costs use the price recorded with each request, so a later price change on a provider does not rewrite past costs.

Logs

docker compose logs --since=30m api worker
docker compose logs -f worker

Each container keeps at most three log files of 10 MB. Review logs before sharing them: remove account names, addresses and anything that looks like a key.

Journal

To reconstruct an incident, read the journal instead: one chronology of the API, the worker, the migrations and the deployments, in files that survive a container’s recreation.

Where. Each process writes its own file in the journal directory: api.log, worker.log, migrate.log, and deploy.log for the production deployment script. A second process of the same service (several API workers, a docker compose run) writes api-<pid>-<random>.log instead, so no two processes ever append to, or rotate, the same file. On a host deployed by libris-production-deploy the directory is /opt/libris-production/logs ($LIBRIS_PRODUCTION_BASE/logs, or LIBRIS_PRODUCTION_LOG_DIR); elsewhere it is the logs volume of the Compose project (docker volume inspect libris_logs), or the host directory named by LIBRIS_LOG_DIR in the .env — create it first, owned by uid 10001 and mode 0700, or Docker creates it for root and the journal stays off.

What a line says. One line per event:

2026-09-24T08:15:03.123+00:00 ERROR    worker   pid=7 req=- job=6f0c1e2a-… epub.worker | job=6f0c1e2a-… status=failed …
    Traceback (most recent call last):
      File "/app/backend/app/jobs/worker.py", line 290, in _execute
    …

The time with its UTC offset (LOG_TIMEZONE), the level, the service, the process, the request (req=, the X-Request-ID every API response carries, taken from the client when it sends a well-formed one), the job (job=; deploy-<time>-<pid> for the steps of one deployment), the logger and the message. Starts, readiness and stops of each process, each API request (method, path, status, duration; /health and static files excepted), each job’s start (resumed=yes when it picks up a checkpoint), completion, pause and failure, the migrations run, and the steps of a deployment are all there, next to what the code already reported (status=database_unavailable, db_pool=…, licence=…).

A traceback follows its event on indented lines. It keeps every frame and the chain of causes, and the message of the exceptions whose message is a technical cause (a refused connection, a missing file, a TLS or HTTP error); for the others — which may quote a passage, a prompt or an account — and for database drivers, which repeat rejected values, only the type is kept (the SQLSTATE code for the latter).

What is never written. Passwords, cookies, Authorization headers, API tokens (their public lbr_… prefix is kept), licence keys, registry and other access tokens, private keys, credentials in URLs, and every value of the configuration whose name says it is a secret (SECRET_KEY, POSTGRES_PASSWORD, *_TOKEN…) are replaced by ***. Line breaks and control characters in a message are escaped (\n), so a file name or a header cannot forge an event, and a message is cut after 4,000 characters, so no chapter is ever copied whole. The same protections apply to the container output. An export still names accounts, books and addresses: review it before sharing it.

Reading it. On a production host:

librisctl journal                                  # the last 200 events of every service
librisctl journal -f --service api --service worker
librisctl journal --level WARNING --since 2h
librisctl journal --request 3f9c0a1b2c3d4e5f      # everything one request did
librisctl journal --job 6f0c1e2a-…                  # everything one job did
librisctl journal export --since 2026-09-24T07:00 --until 2026-09-24T09:00 --with-containers \
    --output /root/incident.log                     # a period, with the database and Codex output merged in

librisctl journal reads the files with the deployed image, in a container without network that mounts them read-only: it works while the installation is stopped. Without the production scripts, run the same tool in a container of the installation (python -m app.journal --help lists the options):

docker compose exec api python -m app.journal -n 100
docker compose exec api python -m app.journal follow
docker compose exec api python -m app.journal export --since 1d > incident.log

Size and permissions. Each file is rotated at LOG_MAX_MB (10 MB) and LOG_BACKUPS (5) rotated files are kept per process; files older than LOG_RETENTION_DAYS (14) go, but never the one a running process writes. The directory is mode 0700 and the files 0600, owned by the application’s user (uid 10001); root reads them on the host. See configuration.

When it cannot be written. A service never stops for its journal. If the directory cannot be opened at start-up (journal: disabled for api, … is not writable on the container output), or a write fails later (a full disk: journal: cannot write … events still go to this output, at most once every five minutes, with the count of the events missing from the file), the events keep going to the container output and the file is tried again at the next event. A deployment that cannot write deploy.log says so and goes on.

Data retention

Every model call leaves a row with its prompt and answer, which the request inspector shows. Without limits, this history grows quickly. The worker cleans it up once at start-up and then every hour, in small batches, following the RETENTION_* settings (configuration):

• prompts, raw answers and context traces of requests older than 30 days are emptied (the inspector no longer shows them), while tokens, cost, duration, status and the cached answer are kept;
• progress events older than 7 days are deleted, always keeping the last 500 of each book;
• OpenViking updates already delivered are deleted after 7 days;
• only the 20 most recent automatic Book Bible revisions of each book are kept (human revisions are all kept);
• the per-passage resume state of jobs that ended more than 30 days ago is deleted. If such a job is resumed later, its finished passages are still not translated again (unless you force a new translation);
• result files of automation requests are deleted after 30 days; asking for the result again rebuilds it;
• files of expired imports are deleted from DATA_DIR/staging.

Before deleting anything, the same pass adds requests older than two hours to daily usage totals, which the statistics and /metrics read. With RETENTION_REQUEST_ROWS_DAYS set (for example 180), whole request rows older than that are then deleted; statistics stay correct, but the response cache and the inspector lose them.

See what a pass would remove, or run one now:

docker compose exec api python -m app.maintenance.retention --dry-run
docker compose exec api python -m app.maintenance.retention

PostgreSQL reuses the freed space, but gives it back to the system only after a full vacuum. It locks the table, so stop the worker first and make sure the disk has as much free space as the table’s useful size:

docker compose stop worker
docker compose exec database sh -c 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "VACUUM (FULL, ANALYZE) llm_requests;"'
docker compose start worker

Maintenance commands

Cost control

Each model call (translation, review, revision, polishing, final review) carries the same fixed context (instructions, glossary, Book Bible, character notes, neighbouring passages), whatever the length of the passage. Three things reduce what you pay:

• Longer passages. PASSAGE_MAX_CHARS (3500 characters by default, 500 to 20000) sets the passage size of books imported afterwards. A volume can have its own (passage_max_chars in its configuration, applied to chapters added later), and so can an import. Longer passages share the fixed context between more text. Beyond 8000 to 10000 characters, the expected answer approaches many providers’ maximum output and a truncated answer costs more than it saves. Books already imported keep their cut.
• Fused review. REVIEW_MODE=fused, or a volume’s review mode, reviews and corrects a passage in one call at high and maximum quality, instead of a review call followed by a revision call.
• Prompt caching. Automatic: prompt sections go from the most stable to the most variable, so that providers with prefix caching can reuse the common start of successive calls.

Other levers: a lower quality level on books that do not need it, FINAL_REVIEW_ENABLED=false, and the per-book estimate shown before each launch.

To cap what a book or an integration may spend, give it a budget: a book’s own cap, the installation default (BUDGET_DEFAULT_BOOK or Settings › Budgets) or an API token’s monthly or total cap. Near the cap, a job moves to a cheaper fallback provider, or pauses until the cap is raised; see the user guide. Budgets only see calls made with a price: give every paid provider its input and output prices.

Measure a configuration

scripts/measure_prompt_cost.py runs the translation pipeline on a synthetic book against a simulated provider (no network, nothing billed) and reports calls and tokens per passage, the share a prefix cache could reuse, and for each operation where successive prompts start to differ. Run it from the repository with the backend’s development environment (development):

python scripts/measure_prompt_cost.py --quality high --passage-chars 3500 --review-issues 0.5
python scripts/measure_prompt_cost.py --review-mode fused --json

The figures compare settings of the same code; they do not predict the bill of a real book.

Compare providers

app.maintenance.compare_providers has several providers translate the same sample of a book’s passages (spread over its narrative chapters), with the prompt and context the pipeline would build, and without the response cache. Nothing is written to the book.

docker compose exec api python -m app.maintenance.compare_providers \
  --project <book id> --providers <provider id>,<provider id> --sample 5 --output comparison.json

The table lists, per provider, passages translated and failed, seconds per passage, tokens, cost and the findings of the automatic checks (locked glossary, markers, untranslated text…). The JSON report adds failure reasons, length ratios and the translations side by side. A relative --output path is written under DATA_DIR/tmp (/data/tmp in the container, since the application directory is read-only), and the command prints the final path; copy the file out with docker compose cp api:/data/tmp/comparison.json .. These calls are real and billed; they appear in the statistics under the operation provider_comparison.

Troubleshooting

For installation, network and sign-in problems, see the Docker guide.