Project Doctor

DATS keeps a project in four data stores: Postgres (documents, annotations, codes, tags, ...), the filesystem (the uploaded files and their thumbnails), Elasticsearch (the full-text index) and Weaviate (the embeddings). A crash, a failed job or a manual intervention can leave them out of sync, for example a document whose file is gone. Such a document breaks the project export:

SourceDocumentNotFoundInFilesystemError: The original file of SourceDocument 6471 (031 Der Koenig von Narnia.mp3) cannot be found in the DATS Filesystem at /dats/docker/backend_repo/projects/27/docs/031 Der Koenig von Narnia.mp3

The project doctor (tools/project_doctor) finds these inconsistencies and heals them, in two steps:

check reads all stores, prints the issues and writes them to a JSON report. It changes nothing.
heal takes the report and applies the fix of each issue. Without --apply it only shows what it would do.

Usage

Run the doctor on the host of the deployment, from the checkout whose docker/ folder runs DATS. The wrapper script starts a one-off container of the backend image with the same environment and volumes as the API (like the scripts in bin/prod), so it reaches all stores with the instance's own configuration.

The doctor imports no DATS code, only the libraries of the backend image (SQLAlchemy, Elasticsearch, Weaviate). It therefore works with the DATS version the deployment runs, also when the checkout is newer than the deployment: copy tools/project_doctor into the checkout of the deployment if it is missing there. Before it touches anything, the doctor verifies that the database has every table and column it uses; on an incompatible version it stops with an error that names the missing columns.

# 1) check one project (or omit --project to check all projects and the leftovers of deleted ones)
tools/project_doctor/dats-doctor.sh check --project 27 -o report.json

# 2) review report.json: delete the issues that heal must not touch

# 3) see what heal would do, then do it
tools/project_doctor/dats-doctor.sh heal report.json
tools/project_doctor/dats-doctor.sh heal report.json --apply

# 4) verify
tools/project_doctor/dats-doctor.sh check --project 27

heal --only <check> limits heal to the issues of one check (repeatable). check --all prints every issue instead of 20 per check, -v prints the tracebacks of failing fixes.

In development, just doctor check|heal ... runs the doctor against the dev stack of the checkout (backend/.env).

Warning

Stop uploads and processing in the affected projects while you heal them. check skips files younger than --min-file-age minutes (default 60), as they may still be processed.

Checks

Check	Finds	Heal
`missing_original_file`	Documents whose uploaded file is missing (breaks the export)	Deletes the document from all stores
`missing_thumbnail`	Images whose `.webp` or thumbnail is missing	None: recompute in the document health view
`orphaned_file`	Files in a project's `docs/` folder that belong to no document (see below)	Moves the file to the quarantine folder
`orphaned_project_dir`	Project folders of deleted projects	Moves the folder to the quarantine folder
`missing_sdoc_data`	Documents without processed data (content, sentences) after a failed or finished processing (breaks the export)	Deletes the document (retry its processing first to keep it)
`failed_processing`	Documents with failed processing steps	None: retry in the document health view
`cross_project_reference`	Annotations, folders, tags or metadata that reference another project	Deletes the annotations, tag links and metadata; folders need a manual decision
`annotation_out_of_bounds`	Span and sentence annotations beyond the text of their document	Deletes the annotations
`missing_es_document`	Indexed documents missing in the project's Elasticsearch index (breaks the search)	Indexes them again (if the index exists)
`orphaned_es_document`	Elasticsearch documents of deleted documents	Deletes them
`orphaned_es_index`	Elasticsearch indices of deleted projects	Deletes them
`missing_embeddings`	Documents without sentence or image embeddings, although the embedding step finished	None: recompute in the document health view
`orphaned_embeddings`	Weaviate embeddings of deleted documents	Deletes them
`orphaned_weaviate_tenant`	Weaviate tenants of deleted projects	Deletes them

DATS keeps some files that no document references, so orphaned_file skips them: the original of an upload that the chunking split into several documents (named like the folder of its chunks) and uploaded .zip archives, whose content became documents.

If the file of a document is missing, but a file exists whose name differs only in case or unicode normalization, the doctor proposes to rename that file instead of deleting the document.

Export and import

Duplicating a project means a full project export and a full project import. Both make assumptions about the stored data that DATS itself does not always keep. The export validates every entity and aborts on the first violation. The import reads the exported CSV files back with pandas, which turns empty cells and values like NA, N/A, None or null into missing values, and matches codes, tags, folders, users and metadata by name. These checks find the data that breaks either one (checked against DATS v1.12.1):

Check	Finds	Heal
`blank_span_text`	Span annotations with a blank text, or one the import reads as missing (`NA`)	Takes the text from the document; deletes the annotation if it is blank there
`blank_value`	Blank memo, whiteboard and analysis titles, code, tag and folder names, metadata keys and descriptions; code and tag descriptions the import reads as missing	Sets a placeholder (`Untitled memo 12`, `N/A (12)`, a single space for descriptions)
`empty_sdoc_html`	Documents whose processing did not finish and whose HTML is empty	Sets an empty HTML document
`duplicate_folder_name`	Normal folders with the same name, or the name of a document folder	Appends the folder id to the name
`broken_hierarchy`	Codes, tags and folders whose parent is in another project, of the wrong type, or in a cycle	Makes them root entries
`system_code_conflict`	Own codes named like a system code; system codes moved away from their configured parent	Renames the codes (`PERSON (custom)`); moves the system codes back
`metadata_key_conflict`	Metadata keys DATS configures with another type (e.g. `language` as a number)	Renames the key (`language (custom)`)
`invalid_sdoc_metadata`	Document metadata without a value for their type, or of a key of another document type	Writes the default of the type; deletes the latter
`invalid_bbox`	Bounding boxes with negative or inverted coordinates	Clamps them to 0 and orders min and max
`non_member_author`	Users who annotated or wrote memos, but were removed from the project	Adds them to the project again
`unsupported_memo_attachment`	Memos attached to a span group, user or memo	Attaches them to the project
`invalid_memo_content`	Memos whose editor content is no valid JSON	Clears the editor content (the memo text stays)
`dangling_whiteboard_node`	Whiteboard nodes of deleted documents, codes, tags or annotations; code nodes with a stale parent	Removes the nodes and their edges; corrects the parent
`invalid_timeline_filter`	Timeline analysis filters on deleted tags, codes, folders or users, or on unsupported columns	Removes these filter expressions

Some data the import cannot carry over, whatever the source project looks like: documents lose their placement in normal folders, imported codes are always enabled and never system codes, timeline analyses lose their date metadata, and whiteboard memo nodes keep pointing at the memos of the source project.

How heal stays safe

The report is the approval. Heal only touches issues that are in the report. Delete an issue from the report to keep it as it is.
Fresh state. Heal checks again before it applies anything and takes the fixes from the fresh check. Issues that were resolved in the meantime are skipped, and the ids it acts on are current.
Documents first. Heal deletes documents first, then checks again and applies the remaining fixes, so they do not act on deleted documents.
Same deletes as DATS. Heal deletes documents and annotations like DATS does: the database cascades remove the dependent rows (data, annotations, tags, metadata, memos, ...), and heal removes the files, the Elasticsearch documents, the Weaviate objects and the document folders that are empty afterwards.
Quarantine instead of delete. Files and folders, also those of deleted documents, are moved to <filesystem root>/quarantine/<timestamp>/, from where you restore them or delete them by hand.
One transaction per fix. A failing fix does not roll back the others; heal reports it and continues.