# `.sci` v1 — the science file format of record This is version 1.0 of this specification. It describes the format at minor version 1.4. Published records are written at 1.2; versions 1.0, 1.1 and 1.3 are earlier and still valid. Owner: **Corpus Engine**. Status: format of record, published. This document says what the format is, who owns it, what it is not, and how a consumer obtains, validates and reads it. It is the answer to a question that was asked repeatedly and answered badly: there are two formats in this platform, SEP v1 and `.sci` v1, and the confusion came from the Corpus Engine's own README calling the wrong one normative. This document settles it. ## 1. The decision **`.sci` v1 is the format of record.** One document describes one research record: the paper and the experiment behind it. It is owned by the Corpus Engine, which publishes it, validates it, and serves it over HTTP. **SEP v1 is a deprecated input dialect.** It is read by the Corpus Engine and converted mechanically. No document is published in it. The specification, validator and builder for SEP v1 remain owned by the Automated Assays lane, which is free to keep them; that ownership does not make SEP an interchange contract. The mechanism of the demotion is in the code, not only in this document: | Artefact | Where | What it says | |---|---|---| | `sci/spec.py` | Corpus Engine | `SCHEMA` is the canonical identifier of the format of record. `DEPRECATED_INPUT_FORMATS` names SEP v1 with `status: deprecated-input` and the converter that reads it. | | `sci/convert.py` | Corpus Engine | `SEP_SCHEMA_URI = "urn:bntrg:sep:1.0.0"` — the deprecated dialect is named by URN, and the conversion is the whole of what remains. | | `sci/spec.py` | Corpus Engine | `SCHEMA_DOCUMENT["$id"]` is the canonical URI, so the schema and the documents agree. | ### Why the split in practice is not a split in the format A paper and the experiment behind it are one research record. They are written at different times — the paper is published, the experiment is inferred or reproduced afterwards — and that timing difference is why practice splits them. The format does not. `.sci` has one document type with four independently-stated layers, each carrying its own `state`, so a paper whose experiment is not yet known is a valid document that says so rather than a missing file. ## 2. The identifier ``` schema_uri https://corpus.bntrg.co.uk/sci/1.0/sci.schema.json urn urn:bntrg:sci:1.0 version 1.0 ``` The schema URI resolves. It is served by the Corpus Engine application, so the schema a consumer validates against is the schema the owner publishes, and it cannot drift from the implementation. The previous identifier, `https://bntrg.sh/sci/1.0/sci.schema.json`, named a host with no DNS record. Every document produced before 1 November 2026 carried it, so every document pointed at nothing. It is replaced, not deprecated-with-a-grace-period: a URI that never resolved cannot be resolved to. **The file extension is cosmetic.** `.sci` and `.science` are both fine, and so is `.json`. The extension does not identify the format; the schema URI and the version do. This matters because an extension can be renamed by any tool in the chain while the schema URI cannot. If you are choosing a convention, use `.sci` — it is short and it is what the publisher writes — but do not key anything on it. ## 3. What one document contains `.sci` is JSON. One document, one record, twenty top-level fields, of which fifteen are required, and one optional binding block: ```json { "sci_version": "1.1", "schema": "https://corpus.bntrg.co.uk/sci/1.0/sci.schema.json", "id": "pmid:12962150", "kind": "science-record", "revision": 1, "content_hash": "sha256:...", "created_at": "2026-11-01T...", "updated_at": "2026-11-01T...", "generator": { "name": "corpus-assay-engine", "converter": "sep_to_sci_v1", "source_package_schema": "urn:bntrg:sep:1.0.0" }, "paper": { "state": "complete", "title": "...", "doi": "...", "pmid": "...", "authors": [], "journal": "...", "year": 2024, "abstract": "...", "claims": [], "gaps": [] }, "experiment": { "state": "partial", "objective": "...", "steps": [], "materials": [], "instruments": [], "reagents": [], "conditions": [], "gaps": [] }, "data": { "state": "partial", "assets": [], "group_sizes": { "Normal": 5, "Sclerotic": 5 }, "gaps": [] }, "analysis": { "state": "partial", "methods": [], "targets": [], "gaps": [] }, "capability": { "state": "partial", "requirements": [], "gaps": [] }, "readiness": { "state": "blocked", "tier": "R4", "blocking_axis": "capability", "blocked_on": "capability:animal_model", "axes": {}, "gaps": [] }, "review": { "state": "absent", "...": "..." }, "execution": { "state": "absent", "...": "..." }, "provenance": { "entities": [], "activities": [], "agents": [], "...": "..." }, "lineage": { "parent_id": "...", "parent_hash": "...", "fork_of": null, "generation": 1 }, "gaps": [], "assays": [ { "role": "...", "order_basis": "...", "definition_id": "...", "definition_version": "...", "definition_digest": null, "required_operations": [], "support_state": "unknown", "source_hashes": {} } ] } ``` Fifteen of the twenty fields are required. `assays` is optional and carries no meaning of its own: it binds the record to the assay definitions somebody else owns, one for every process the experiment ran. ### The two fields version 1.1 added `sci_version` names the minor version a document was written to. The schema at the format's URI accepts `1.0` and `1.1`, so a 1.0 document is still valid and unchanged, and a 1.1 document is valid at the same URI. The frozen 1.0 schema is kept at `sci/schema/sci-1.0.schema.json` and the current one is `sci/schema/sci-1.1.schema.json`. **`data.group_sizes`** states the number of samples in each comparison group of the source data, as the repository metadata states: ```json { "Normal": 5, "Sclerotic": 5, "Calcified": 5 } ``` It is the field that separates a reproduction from a re-analysis. A consumer holding the samples can check them against the paper's design instead of running the paper's method over a different set of samples. The value is `null` when the groups are not known, and the repository metadata that states it is recorded beside the asset as `group_sizes_basis`. **`assay`** binds the record to an assay definition: ```json { "definition_id": "assay:one-way-anova", "definition_version": "1.0.0", "definition_digest": "sha256:...", "required_operations": ["group_comparison_anova"], "support_state": "supported", "source_hashes": { "assay_definition": "sha256:..." }, "confidence": 0.8, "evidence": ["one-way ANOVA with Tukey post hoc test"] } ``` The definition itself, the provider, the price and the catalogue state are not in the record. They belong to the platform that catalogues assays and they change; the record names what it needs and leaves the rest to its owner. `confidence` and `evidence` say why the binding is there. The confidence is an evidence score in `[0,1]`, not a probability, and the evidence is the passage of the paper that supports it, so a binding can be checked against the paper's own text rather than taken on trust. Both are optional, and both are absent when the platform is binding a record to a definition it was given rather than one it read out of the paper. Version 1.2 replaced this one binding with the ordered array `assays`. The flat binding above is kept, because a 1.1 document carries it and stays valid. ### What version 1.2 added Version 1.2 replaces the single binding with `assays`, the ordered array of every assay the experiment ran, and adds three optional blocks inside each binding, so a record can say what the assay definition was, what was asked of it, and what came back. **An experiment is a set of processes, and each process is an assay.** The measurement that produced the data, the analysis that carries the claim, the validation beside it, the normalisation before it: an experiment that left one of them out did not run as described, so every one of them is required. There is no primary assay and no optional one. The array is never empty, and its order is the order of the experiment. ```json { "assays": [ { "role": "data_generation", "order_basis": "paper_text_order:4821", "definition_id": "assay:microarray-hybridisation", "definition_version": "1.0.0", "definition_digest": "sha256:...", "required_operations": [], "support_state": "supported", "source_hashes": { "assay_lexicon": "sha256:..." }, "confidence": 0.9, "evidence": ["Total RNA was labelled and hybridised to the array."], "definition": { "...": "..." }, "request": { "...": "..." }, "result": { "...": "..." } }, { "role": "analysis", "order_basis": "consumes_earlier_output", "definition_id": "assay:one-way-anova", "...": "..." } ] } ``` `role` says what the process is, in free text, because what a process is for depends on the experiment. `order_basis` says how the binding's place in the order was established: `paper_text_order:` when the paper's own text describes the process there, `consumes_earlier_output` when it consumes what the earlier ones produced. A consumer that wants to run the experiment walks the list in order. The three blocks `definition`, `request` and `result` say what the assay definition was, what was asked of it, and what came back: ```json { "definition": { "definition_id": "assay:one-way-anova", "definition_version": "1.0.0", "definition_digest": "sha256:...", "required_operations": ["group_comparison_anova"], "support_state": "supported", "input_contract": { "...": "..." }, "parameter_contract": { "...": "..." }, "output_contract": { "...": "..." } }, "request": { "source_record_id": "pmid:26981379", "source_content_hash": "sha256:...", "operation": "group_comparison_anova", "operation_version": "1.0.0", "inputs": [], "parameters": { "...": "..." }, "request_id": "...", "provider": "..." }, "result": { "provider": "...", "execution_id": "...", "input_hash": "sha256:...", "result_hash": "sha256:...", "evidence_hash": "sha256:...", "status": "succeeded", "finding": "...", "result": { "...": "..." } } } ``` Each block is optional on its own — a record may be bound to a definition and never run, or run and never judged — but a binding that names none of them is not a binding and the schema refuses it. The three hashes in `result` are what make a run a reproduction rather than an assertion. `input_hash` says which input produced the outcome, `result_hash` says what came out, and `evidence_hash` says what shows the work. A result without them is a claim; a result with them can be checked. The request binds to the version it was made against. `source_record_id` names the record and `source_content_hash` names the exact body of it at the time, so an execution is tied to one version of the record rather than to whatever the record says now. A 1.1 document keeps its flat binding and stays valid: the 1.1 schema is frozen at `sci/schema/sci-1.1.schema.json`. A 1.2 document may not carry the 1.1 binding, and the validator refuses it and says why. ### What version 1.3 added Version 1.3 adds one optional field, `assays[].depends_on`, so a record can say which assay needs the output of which other assay. The array says the order of the experiment. The dependency says the dependency. They are different facts, and a record that carries one does not carry the other: an entry with no `depends_on` states no dependency, and an ordered array does not say that each entry needs the one before it. A consumer that draws the experiment as a chain of processes needs the edges, and a consumer that walks the experiment in order does not. ```json { "assays": [ { "id": "assay:1", "role": "data_generation", "definition_id": "assay:qpcr", "depends_on": [] }, { "id": "assay:2", "role": "analysis", "definition_id": "assay:one-way-anova", "depends_on": ["assay:1"] }, { "id": "assay:3", "role": "validation", "definition_id": "assay:immunohistochemistry", "depends_on": ["assay:1"] } ] } ``` The rules: - **Optional.** A record that states no dependency carries no `depends_on`, or carries an empty one. Both say the same thing. - **Written, never inferred.** A dependency is a statement about the experiment as it ran, so it comes from whoever ran it. Nothing in the format derives a dependency from the order of the array, and a reader must not do it either. - **Within one document.** A dependency names an entry of the same document, by that entry's identifier. It does not cross to another record: a dependency on an assay in a different record is a fact about two records, and the record that states it is the one that owns it. The corpus states such a fact as a claim or a gap, not as a link. - **Forward references are allowed.** An entry may name an entry that comes later in the array. The array is the order of the experiment, and the order is not the dependency graph: an assay may consume an output that the record lists further down. - **No cycles.** The dependencies of one document may not form a loop. The validator refuses a cycle and names it. - **The target must exist.** A dependency on an identifier the document does not carry is refused, and so is a dependency on the entry itself. The corpus publishes no dependency edges. The corpus reads a paper's claims, methods and data routes; it does not read the flow between the steps of an experiment, and a paper says which operation followed which only in prose. A corpus record therefore carries `order_basis` and no `depends_on`, and a corpus record stays at version 1.2. A record whose author knows the flow — a document written from a run, by the platform that ran it — carries the edges, and is at version 1.3. A 1.2 document is unchanged and still valid: the 1.2 schema is frozen at `sci/schema/sci-1.2.schema.json`, and a 1.2 document that carries `depends_on` is refused. ### What version 1.4 added Version 1.4 adds one optional field, `assays[].operations`, and one optional field inside the result block, `assays[].result.operation`. The reason is that an assay definition can implement more than one operation, and the operations do not take the same inputs, refuse the same things or return the same outputs. A group comparison may be run as an analysis of variance, which returns an F statistic and a post hoc comparison; as a rank-sum test, which returns a U statistic and a p-value; or as a t-test, which returns a t statistic and an interval. One contract cannot say all three, and a consumer that reads the wrong one reads a promise the operation does not keep. The array of assays stays the array of assays. One assay is one process in the experiment, and it appears once. The operations it may be run as are listed inside its own entry, one item each, and each item carries that operation's own contract. ```json { "assays": [ { "id": "assay:1", "role": "analysis", "definition_id": "assay:group-comparison", "operations": [ { "operation": "group_comparison_anova", "operation_version": "0.1.0", "support_state": "supported" }, { "operation": "group_comparison_rank_sum", "operation_version": "0.1.0", "support_state": "supported" } ] } ] } ``` The rules: - **Optional.** An entry that states no operations says nothing about how it may be run. That is not the same as saying it may be run as nothing, so the field is never required and never required to be empty. - **One operation once.** An entry may not list the same operation twice. An operation named twice is named once, and a request cannot say which of the two it means. - **A request or a result names a listed operation.** Where an entry lists its operations, a request or a result inside it that names an operation the entry does not list is refused: the request is being made against something else. A version that disagrees with the listed version of the same operation is a warning, not a refusal. - **An entry is identified by its definition.** Two entries of one array may not carry the same definition id, version and digest. The identity of an assay is the definition it is bound to. An assay that may be run as several operations lists them in its own `operations` and is not repeated in the array to carry one operation each. - **A result may name its operation.** `assays[].result.operation` says which operation produced the result. Where the entry lists several operations, a result that does not say which one it is says less than the reader needs. A 1.3 document is unchanged and still valid: the 1.3 schema is frozen at `sci/schema/sci-1.3.schema.json`, and a 1.3 document that carries `operations` is refused. The corpus's own records stay at 1.2: they name no operations, and a record that names no operation is honest about the fact that the corpus read a paper and did not run it. ### The four layers `paper`, `experiment`, `data` and `analysis` are the record. Each carries a `state` of `absent`, `partial`, `complete` or `blocked`, and each carries its own `gaps`. The four states are what make a partial document honest: a paper that has been read but whose experiment has not been determined is `paper.state = "complete"`, `experiment.state = "partial"`, and the gaps name what is missing. A document is never silently incomplete. `capability` and `readiness` are the answer to "what would it take to run this". `readiness` gives the tier (`R0`–`R5`), the blocking axis (`method`, `capability`, `data` or `none`) and the per-axis detail, so a consumer can tell a paper that needs a method determined from one that needs a bench built. `review`, `execution` and `provenance` record who has checked the record, whether it has been run, and where every part of it came from. ### The method layer, and what its two operation lists mean `analysis.methods[]` carries one row per method the paper used, each with the evidence that named it. Two lists on a row say two different things, and the difference matters: - `dsl_operations` — DSL operations that **exist in the catalogue** and express this method. Every name in this list is an operation the Journal's DSL can run today. - `missing_operations` — operations the catalogue **does not have**. These are what a build list is made from. They are named by the corpus, and a name here is not a catalogue operation. `expressible` is true when `missing_operations` is empty. The catalogue decides what the DSL can express, and the catalogue is the Journal's own published one: it publishes the operations that express each method family, and it publishes the methods it cannot do at all. A method the catalogue names as unsupported stays unsupported. A method the catalogue covers through its family gets that family's operations. The corpus does not get to invent an operation name and then report it as missing, which is missing by construction and says nothing about the DSL. Where a method is named but the catalogue cannot express it, the row still publishes the paper's own words for it. A method named but not expressible is information; a method invented to fit the catalogue would not be. ### The data layer and the file formats External files are **referenced, never embedded**. Every file a record rests on is an entry in `data.assets` carrying a URI, a SHA-256, a declared media type, an access state and a reader state. A table already published inside the source article is the one deliberate exception: it is an inline `jats-table` asset whose rows travel in the document, because the article's own table is otherwise a machine-readable data route that no consumer can read. Every published record's full text is fetched once, sectioned, and its tables are indexed this way: 839 tables across 392 of the 806 records, each with its own checksum and its own locator. The registry of thirty-two formats is in `sci/spec.py` (`FORMAT_REGISTRY`) and in the schema, and it covers the formats the platform was asked to carry: | Format | Media type | Reader state | Note | |---|---|---|---| | `csv` | `text/csv` | `available` | working reader | | `jats-table` | `application/vnd.jats.table+json` | `available` | rows carried inline in the `.sci` asset | | `geo-series` | `application/vnd.ncbi.geo-series+json` | `reader_required` | `geo_series_read`; resolution does not download data | | `tsv` | `text/tab-separated-values` | `converter_required` | `delimited_to_csv` | | `xlsx` | spreadsheetml | `converter_required` | `xlsx_to_csv` | | `xls` | `application/vnd.ms-excel` | `converter_required` | `xls_to_csv` | | `pdf` | `application/pdf` | `reader_required` | `pdf_text_extract` | | `vcf` | `text/x-vcf` | `reader_required` | `vcf_parse`, reference-dependent | | `bcf` | `application/x-bcf` | `reader_required` | `vcf_parse`, binary | | `gvcf` | `text/x-gvcf` | `reader_required` | `gvf_parse`, legacy — never parsed as VCF | | `fasta` | `text/x-fasta` | `reader_required` | `fasta_parse` | | `fastq` | `text/x-fastq` | `reader_required` | `fastq_parse` | | `h5` | `application/x-hdf5` | `reader_required` | `hdf5_read` | | `zarr`, `ome_zarr` | zarr / OME-Zarr | `reader_required` | `zarr_read` | | `parquet` | parquet | `reader_required` | `parquet_read` | | `npz` | `application/x-npz` | `reader_required` | `npz_read` | | `txt`, `md`, `xml`, `json`, `yaml` | text | `reader_required` | text-bearing | | `png`, `jpg`, `tif` | images | `reader_required` | `image_read` | | `zip`, `gz`, `tar_gz` | archives | `reader_required` | archive handling | | `py`, `r`, `sql`, `ipynb` | code | `reader_required` | `code_read`, `notebook_read` | Two rules the registry exists to enforce: 1. **The engine never guesses a reader from an extension.** A file declared `format: "pdf"` with `reader_status: "reader_required"` is a PDF the engine cannot read yet, and the record says so. 2. **GVCF is never parsed as VCF.** They are different formats with different reference semantics, and the registry carries both. `reader_status` is a capability statement, not a set of working readers. CSV and inline JATS tables have working readers today. That is the honest state of the bench, and the format is what makes it visible instead of hiding it. An inline table is carried in `data.assets[].table`: ```json { "id": "asset:table:pmid:28529692:1", "role": "table", "format": "jats-table", "sha256": "sha256:...", "bytes": 1234, "identifier": "section[1]/table-wrap[1]#T1", "reader": "read_table", "reader_status": "available", "table": { "label": "Table 1", "caption": null, "rows": [["Gene", "Lum A"], ["ATP1A1", "1.53 (0.009016)"]], "row_count": 2, "column_count": 2, "numeric_cells": 1, "header_rows": 1, "locator": "section[1]/table-wrap[1]#T1" } } ``` The asset `sha256` covers the canonical JSON encoding of the complete `table` object: sorted keys, compact separators, UTF-8. `bytes` is the length of those encoded bytes. The source locator identifies the enclosing section, the `table-wrap` ordinal and the JATS element id when one exists. This extension is additive and does not change the schema URI or any top-level field. ### The PDF route: a paper whose open text is a PDF Most papers are in PMC and their open text is JATS. Some are not. Their only open text is the publisher's PDF, or the supplementary file the publisher deposited beside the article. Such a paper can still be fetched, sectioned, read and published, and the record says exactly what it rests on. The manuscript asset of such a record **is** the PDF: ```json { "id": "source:pmid:26203686", "role": "manuscript", "format": "pdf", "media_type": "application/pdf", "uri": "https://ndownloader.figshare.com/files/2350161", "sha256": "sha256:6dbd53d906348f21636323a018356b51e1f8cf2bb58982eb3f0f6aff74315f2c", "bytes": 722529, "reader": "read_pdf", "reader_status": "available", "structure": { "source": "publisher_pdf", "coverage": "whole_supplementary_document", "document_kind": "supplementary_material", "licence": "CC BY 4.0", "text_hash": "32ac39d2...", "text_hash_scope": "rendered_pdf_text", "cut": [{"what": "article-body", "reason": "..."}] } } ``` Four rules hold for this route. 1. **The checksum covers the PDF bytes.** It is the checksum of the file the URI points at, so a consumer can fetch it and check it. The hash of the *text read out of* the PDF is published beside it as `structure.text_hash`, because that is the text every quotation is checked against. 2. **The file is published as an asset in its own right**, with its licence and its `document_kind`, under the `supplementary` role. 3. **The cut is recorded.** Where the article body is not open, the record carries a `cut` entry naming what is missing and why. A record never pretends to be the whole paper. 4. **The coverage value is honest.** A PDF paper whose whole open document was read carries `coverage: "whole_supplementary_document"`, which says the supplementary document was read whole and says nothing about the article body. The reader is the corpus's own PDF text extractor (`pdftotext -layout`, with a `pdfminer` fallback). It extracts text; it does not parse a PDF's tables into row assets the way a JATS table is parsed. A number that a paper is judged on and that lives only in a PDF table is published in the record's analysis targets with its table locator, not as an inline table asset. ### The content hash `content_hash` is a SHA-256 over the canonical JSON of the document body with the `content_hash` field itself removed (`sci.spec.content_hash`). It is reproducible: the same document always hashes to the same value, and `revision` is the counter that changes when the body changes. The canonical encoding is stated here in full, because a consumer has to reproduce it byte for byte to check the hash: | Part | Rule | |---|---| | Keys | Sorted, compared as strings. | | Separators | Tight. No space after a colon or a comma. | | Text | UTF-8, with slashes and non-ASCII characters left as they are. | | Objects and arrays | Written as JSON objects and arrays. An empty object is not an empty array. | | The hash field | Excluded from the body it is computed over. | **Numbers** are written with the fewest decimal digits that read back to the same value. A value of at least 0.0001 and less than 10^16 is written in plain decimal, and a whole value keeps its fractional part there (`2.0`). Anything else is written in scientific notation, with an exponent that carries a sign and at least two digits and a mantissa that carries no trailing zero: `1.85e-07`, `8.5e-09`, `1e+16`. This is the part of the rule where languages disagree, and the difference is silent. PHP writes `1.85e-7` and `1.0e-5` where the corpus writes `1.85e-07` and `1e-05`; JavaScript writes `0.00001` where the corpus writes `1e-05`. A consumer that decodes a document and re-encodes it has to write the number this way, or the recomputed hash will not match the declared one. The corpus's own writer is `sci.spec.canonical_number`. ### Revisions and lineage A published document is immutable. When the record behind it changes, the document is published again as the next revision of the same record, and it names the revision it replaces: ```json { "revision": 2, "lineage": { "parent_id": "pmid:28529692", "parent_hash": "sha256:5059f49a...", "fork_of": null, "generation": 1 } } ``` `parent_hash` is the `content_hash` of the revision this one replaces, so a consumer holding the earlier revision can prove what it holds and can fetch the one that supersedes it. `generation` counts the revisions before this one: the first revision of a record has `generation` 0 and no parent. `fork_of` names the record a review fork was made from, and is `null` for anything else. The earlier revision is kept beside the current one rather than overwritten, so the history of a record is readable. Every version a record has had is published and served, not only the current one. A version is addressed by the hash of its own body, which is the address that survives the record being revised: the record id names the record and the revision number names its position in it, but the hash names the bytes. This matters because a document names the version it was made from — `lineage.parent_hash` is the `content_hash` of the revision that preceded it — and a hash a consumer cannot fetch is a claim it cannot check. A consumer that read a version, read it again later, and found different bytes under the same hash has been given a broken promise. What does not make a new revision is a rebuild. The revision number, the lineage, the timestamps and the hash are bookkeeping, so a rebuild that changes nothing else leaves the published document exactly as it was — same bytes, same hash, same revision number. ## 4. How to obtain and validate a document ### Over HTTP The Corpus Engine publishes the format and its documents: | Endpoint | What it returns | |---|---| | `GET https://corpus.bntrg.co.uk/sci/1.0/sci.schema.json` | The JSON Schema, raw. This is the URI every document carries. | | `GET /api/v1/sci` | The format contract: what the format is, the schema URI, the input dialects and their status. | | `GET /api/v1/sci/schema` | The same schema inside the published envelope. | | `GET /api/v1/sci/fields` | The field reference: every key of a document, its path, its type, whether it is required and what it means. | | `GET /api/v1/sci/documents` | The published documents, with `tier`, `blocking_axis`, `review_state` and `execution_state` per document. Filterable and pageable. | | `GET /api/v1/sci/documents/{id}` | One whole `.sci` document. | | `GET /api/v1/sci/documents/{id}/versions` | Every published version of that record: its `content_hash`, its revision number, and whether it is the current one. | | `GET /api/v1/sci/versions/{content_hash}` | One exact version, by the hash of its own body. The bytes served are the bytes published, so recomputing the canonical hash over them gives the hash that was asked for. | | `GET /api/v1/papers/{pmid}/sections` | The section index for a paper: every section, its role and its size, plus the body hash and whether the body was cut. | The `/api/v1` paths are relative to the Corpus Engine's public host. The schema URI is absolute, so a document carries the address of its own schema. ### Locally ```bash cd corpus-assay-engine python3 tools/sci_build.py # build and validate the documents python3 -c "import json,sys; sys.path.insert(0,'.'); \ from sci.validate import validate_document; \ d=json.load(open('out/sci/pmid_12962150.sci.json')); \ print(validate_document(d).as_dict())" ``` `tools/sci_build.py` reads SEP packages and writes validated `.sci` documents, the JSON Schema and an index. It makes no provider calls: the conversion is model-free and deterministic. ### Validating against the schema The schema is JSON Schema 2020-12. A validator that understands that draft will do: ```bash curl -s https://corpus.bntrg.co.uk/sci/1.0/sci.schema.json | \ python3 -c "import json,sys; json.dump(json.load(sys.stdin), open('sci.schema.json','w'))" python3 -c "import json; from jsonschema import Draft202012Validator as V; \ s=json.load(open('sci.schema.json')); \ d=json.load(open('out/sci/pmid_12962150.sci.json')); \ V(s).validate(d); print('valid')" ``` The engine's own validator (`sci/validate.py`) does the schema check and three more that a schema cannot express: the content hash must match the body, a step may only cite an asset that exists, and a method may only claim operations the DSL catalogue knows. ## 5. Converting from the deprecated dialect SEP v1 packages are read and converted: ```bash python3 tools/sci_build.py --packages out/sep/packages --out out/sci ``` The conversion is total and mechanical: `sci/convert.py` maps each SEP layer onto the `.sci` layer that carries it, records the source package in `provenance` and `generator.source_package_schema`, and states what the package did not carry in the layer's `gaps`. A converted document names its input dialect by URN (`urn:bntrg:sep:1.0.0`), not by URL. The URL form of that identifier names an internal host, and a published document must not carry an internal host: the release gate scans every published body for build-machine paths and local addresses and refuses the release if it finds one. ## 6. What this format is not - It is not a claim that any paper has been replicated. `execution.state` says whether a record has been run, and no record has been. - It is not a container for datasets. Files are referenced by URI and checksum, never embedded. A table already present in the source article is the exception: its rows remain in the article and travel in the record as a checksummed inline asset. A 40-million-record corpus is not a repository of PDFs. - It is not a journal submission format. The Journal consumes `.sci` documents and produces review forks from them; a fork is a `.sci` document with a `lineage.fork_of` and a higher `generation`. - It is not the SEP format, and it does not replace SEP in the Automated Assays lane. It replaces SEP as the thing that crosses between services. ## 7. The state of the format on 3 November 2026 | Fact | Number | |---|---| | Documents built and validated | 9,745, all valid, zero errors, zero warnings | | Current minor version | 1.4, at the same schema URI as 1.0, 1.1, 1.2 and 1.3 | | Published versions of records | 42,412, of which 32,667 are superseded and kept | | Historical versions the release gate refused | 7, listed in the coverage index | | Sectioned papers with a published section index | 9,995 | | Sections in that index | 272,711 | | Tables published as first-class assets | 839, across 392 records | | Fields described in the field reference | 250 | | Formats in the registry | 32, of which 2 have a working reader, and PDF text extraction is now a working corpus-side reader | | Papers published through the PDF route | 1, `pmid:26203686` | | DSL operations the published records need | 60, of which 10 the catalogue cannot yet express | | Method families the published records name | 19, over 60 named specific methods | | Papers replicated | 0 | | Datasets downloaded or licence-checked | 0 | The seven adapter disagreements between the corpus-side `.sci` hand-over and the Automated Assays SEP hand-over are recorded in `out/sci/journal/disagreements.json`. They are disagreements about a deprecated dialect's fields, not about the format of record. ## 8. Change control The format changes by version, and the version is in the schema URI. A change that alters the meaning of an existing field, removes a field, or changes the content-hash rule is a major version and a new URI. A change that only adds an optional field is a minor version and the same URI with a new `sci_version`. The owner is the Corpus Engine. A change proposed by another lane is supplied as a patch with its reason and its evidence, and the Corpus Engine applies it. No lane forks the format. ### How a change is made The schema, the field reference and the converter are one format and they move together. The field reference is generated from the schema, so a field that the schema does not declare has no address, and a field that the schema declares and the reference does not describe is a hole in the format rather than a private note. 1. Edit the schema in `sci/schema/sci-1.4.schema.json` and the description of the field in `sci/field_descriptions.json`. 2. Regenerate the reference and the documents, and run the tests: `python3 tools/sci_spec.py`, `python3 tools/sci_build.py`, `python3 -m unittest discover -s sci/tests -t .`. 3. Write the change down in this section, with the version it belongs to and why the format needs it. 4. Publish the schema, the reference and this document together, so the three that a consumer reads are the three that were tested. A change that only adds a field, widens a description or fixes a wording is a documentation change and does not move the version. A change that accepts or rejects a document a different version would reject is a minor version, published at the same schema URI. A change that removes or renames a field is a major version and a new URI. ### 1.1 Version 1.1 is the first minor version. It adds two optional fields, `data.group_sizes` and the `assay` binding, and changes nothing else: the schema URI is unchanged, a 1.0 document is unchanged and still valid, and the content-hash rule is unchanged. A client that hardcodes `https://corpus.bntrg.co.uk/sci/1.0/sci.schema.json` reads the same address and validates both versions. ### 1.2 Version 1.2 replaces the 1.1 `assay` binding with `assays`, the ordered array of every assay the experiment ran. Each binding may carry three optional blocks, `definition`, `request` and `result`, and four optional properties: `role`, saying what the process is; `order_basis`, saying how its place in the order was established; `confidence`, an evidence score in `[0,1]`; and `evidence`, the passage of the paper that supports the binding. The schema URI is unchanged, the 1.0 and the 1.1 schemas are kept frozen at `sci/schema/sci-1.0.schema.json` and `sci/schema/sci-1.1.schema.json`, a 1.1 document is unchanged and still valid, and the content-hash rule is unchanged. The reason is that a record could not say what it asked an assay definition to do, nor bind the outcome to the input that produced it, and had exactly one slot for the assays of an experiment that usually has several. An experiment is a set of processes and each process is an assay, so the binding had to become the experiment's ordered list, with every entry required. The 1.1 binding is not reused. A consumer that reads `assay` keeps working, because a 1.1 document still validates against the frozen 1.1 schema; a 1.2 document that carries `assay` is refused. The operations an assay needs are the array, `assays[].definition.required_operations`. #### The amendment to 1.2 on 1 November 2026 1.2 was first published on 1 November 2026 with the singular `assay` binding still in place, and was amended the same day to the ordered `assays` array. The amendment is recorded here rather than hidden, because a version that changes shape inside a day is only safe if it is said out loud. What changed: `assays` became an array, `assay` was removed from the 1.2 schema, and the `definition`, `request` and `result` blocks moved inside each array entry. What did not change: the schema URI, the accepted `sci_version` values, the content-hash rule, the frozen 1.0 and 1.1 schemas, or any published document. No consumer had adopted 1.2 before the amendment: Automated Assays was still emitting 1.1, and the corpus published no document at 1.2 before the array was in place. The first published 1.2 documents are the ones that carry the array. Two additions were made to 1.2 after the array, both in the same day and both additive: - `assays[].id`, **required**. Every entry names itself, so a request or a result can point at one entry and not at the array as a whole. The identifier is the position: `assay:1` is the first assay the experiment ran. - `assays[].request.assay_id` and `assays[].result.assay_id`, **required** in their blocks. A request names the entry it runs and a result names the entry it belongs to. With the record's `source_content_hash` beside it, that says exactly which assay of which version of which record a run concerns. `assays[].definition.limitations` and `assays[].definition.interpretation` were added at the same time. `analysis.methods[]` is `additionalProperties: false`, so it cannot hold an operation's limits, and an operation's limits are a property of the operation rather than of the record that used it. The catalogue owns the operation; the definition carries its limits. ### 1.3 Version 1.3 adds one optional field, `assays[].depends_on`: the entries of the same document whose output this entry needs, named by their identifiers within that document. The schema URI is unchanged, the 1.0, 1.1 and 1.2 schemas are kept frozen at `sci/schema/sci-1.0.schema.json`, `sci/schema/sci-1.1.schema.json` and `sci/schema/sci-1.2.schema.json`, a 1.0, a 1.1 and a 1.2 document are unchanged and still valid, and the content-hash rule is unchanged. A 1.2 document that carries `depends_on` is refused. The reason is that the ordered array says the order of the experiment and the format had no way to say the dependency. A consumer that draws the experiment as a chain had to infer the edges from the order, and an inferred edge is a claim nobody made. The rules are in the section above: the field is optional; a dependency is written and never inferred from the order; it names an entry of the same document and does not cross to another record; a forward reference is allowed; a cycle is refused and named; and a dependency on an identifier the document does not carry, or on the entry itself, is refused. The corpus publishes no dependency edges and its records stay at 1.2. The corpus reads a paper's claims, methods and data routes, not the flow between the steps of an experiment; a corpus record carries `order_basis`, which says how the place in the order was established, and no `depends_on`. A document written from a run, by the platform that ran it, carries the edges and is at 1.3. ### 1.4 Version 1.4 adds one optional field, `assays[].operations`: the operations an assay may be run as, one item each, each carrying that operation's own input, parameter and output contract. It also adds one optional field, `assays[].result.operation`, so a result says which operation produced it. The schema URI is unchanged, the 1.0, 1.1, 1.2 and 1.3 schemas are kept frozen at `sci/schema/sci-1.0.schema.json`, `sci/schema/sci-1.1.schema.json`, `sci/schema/sci-1.2.schema.json` and `sci/schema/sci-1.3.schema.json`, a 1.0, a 1.1, a 1.2 and a 1.3 document are unchanged and still valid, and the content-hash rule is unchanged. A 1.3 document that carries `operations` is refused. The reason is that an assay definition may implement several operations and one contract cannot describe them all. A group comparison can be run as an analysis of variance, a rank-sum test or a t-test, and those take different inputs, refuse different things and return different outputs. The alternative — repeating the assay in the array once per operation — was considered and refused: the array is the list of processes in the experiment, and repeating a process to carry one contract each makes the array describe something the experiment did not do. The operations therefore sit inside the one entry. Two rules came with it. An entry may not list the same operation twice, so a request naming it is unambiguous. And two entries of one array may not carry the same definition identity — definition id, version and digest — because the identity of an assay is the definition it is bound to, and two entries that are the same assay are one assay. The corpus publishes no operations, and its records stay at 1.2. The corpus reads what a paper did; it does not decide which operations an assay definition may be run as, and it does not invent an operation list. A document written from a run, by the platform that ran it, carries the operations and is at 1.4. ## 9. Field reference Every key of a `.sci` document, in alphabetical order. The path is the key's place in the document: `paper.claims[].target.value` is the value of the target of a claim of the paper layer. Each field has its own anchor, so any key can be linked to directly, and each folds away until it is wanted. `250` fields, `250` of them described here. The schema is the authority on what is valid; this section is the authority on what each key means. ### The top level | Field | Type | Required | What it carries | |---|---|---|---| | [`analysis`](#field-analysis) | object | yes | The analysis: the methods named, the operations they need and the published target a run would check. | | [`assays`](#field-assays) | array | optional | Every assay the experiment ran, in the order it ran them. An experiment is a set of processes and each process is an assay: the measurement, the analysis, the validation, the normalisation. Every binding here is required for the experiment as described; there is no primary assay and no optional one. A consumer that wants to run the experiment walks the list in order. | | [`capability`](#field-capability) | object | yes | The capabilities the experiment needs to run. | | [`content_hash`](#field-content-hash) | string or null | yes | SHA-256 over the canonical JSON of the record body, as `sha256:`. A consumer recomputes it to check the record. | | [`created_at`](#field-created-at) | string or null | optional | When the record was first published, in UTC. | | [`data`](#field-data) | object | yes | The data the work rests on: every file, table or repository route, with its checksum and access state. | | [`execution`](#field-execution) | object | yes | Whether the experiment has been run, and the evidence if it has. | | [`experiment`](#field-experiment) | object | yes | The experiment behind the paper: what was done, with what, in what order. | | [`gaps`](#field-gaps) | array | yes | Every gap in the record, gathered from the layers. | | [`generator`](#field-generator) | object or null | optional | The tool and converter that wrote the record, and the input dialect it was read from. | | [`id`](#field-id) | string | yes | The record identifier. A corpus record is `pmid:`; a user-authored fork is `fork::`. | | [`kind`](#field-kind) | any — one of `science-record` | yes | What kind of record this is. Always `science-record` in version 1. | | [`lineage`](#field-lineage) | object or null | optional | The version this record supersedes, and the generation it belongs to. | | [`paper`](#field-paper) | object | yes | The source: what the paper says, and the claims it makes. | | [`provenance`](#field-provenance) | object | yes | Where every part of the record came from. | | [`readiness`](#field-readiness) | object | yes | How ready the record is to run, and what has to be resolved first. | | [`review`](#field-review) | object | yes | What the readers made of the paper, and where that critique is published. | | [`revision`](#field-revision) | integer | yes | The counter that increases each time the body of the record changes. The first publication is revision 1. | | [`schema`](#field-schema) | any — one of `https://corpus.bntrg.co.uk/sci/1.0/sci.schema.json` | optional | The absolute URI of the JSON Schema the document validates against. It resolves to the schema and to this specification. | | [`sci_version`](#field-sci-version) | enum — one of `1.0`, `1.1`, `1.2`, `1.3`, `1.4` | yes | The minor version of the format this document is written in. 1.0, 1.1 and 1.2 are accepted at the same schema URI. | | [`updated_at`](#field-updated-at) | string or null | optional | When the record body last changed, in UTC. | ### Every field
analysis — object - **Required**: yes, in every document - **What it carries**: The analysis: the methods named, the operations they need and the published target a run would check.
analysis.gaps — array - **Required**: yes, in every document - **What it carries**: What is missing from the analysis layer.
analysis.methods — array - **Required**: yes, in every document - **What it carries**: The methods the paper used.
analysis.methods[].data_shapes — array - **Required**: yes, in every document - **What it carries**: The shapes of data the method consumes.
analysis.methods[].dsl_operations — array - **Required**: yes, in every document - **What it carries**: The engine operations the method needs.
analysis.methods[].evidence — object - **Required**: yes, in every document - **What it carries**: How the method was identified, and how sure the engine is.
analysis.methods[].evidence.band — string or null - **Required**: yes, in every document - **What it carries**: The interval the agreement score is reported with.
analysis.methods[].evidence.quote — string or null - **Required**: yes, in every document - **What it carries**: The text the method was identified from.
analysis.methods[].evidence.score — number or null - **Required**: yes, in every document - **What it carries**: The measured agreement score behind the identification.
analysis.methods[].expressible — boolean or null - **Required**: yes, in every document - **What it carries**: Whether the method can be expressed with the operations the engine has.
analysis.methods[].family — string or null - **Required**: yes, in every document - **What it carries**: The family the method belongs to, for example `statistical_test`.
analysis.methods[].id — string - **Required**: yes, in every document - **What it carries**: The identifier of the method within the record.
analysis.methods[].label — string or null - **Required**: yes, in every document - **What it carries**: A short human-readable name for the method.
analysis.methods[].missing_operations — array - **Required**: yes, in every document - **What it carries**: The operations the engine does not have yet.
analysis.methods[].parameters — object - **Required**: yes, in every document - **What it carries**: The parameters of the method, keyed by name.
analysis.methods[].specific — string or null - **Required**: yes, in every document - **What it carries**: The specific method, as the paper names it.
analysis.state — enum — one of `absent`, `partial`, `complete`, `blocked` - **Required**: yes, in every document - **What it carries**: How complete the analysis layer is.
analysis.targets — array - **Required**: yes, in every document - **What it carries**: The published values a run would check itself against.
analysis.targets[].claim_id — string or null - **Required**: yes, in every document - **What it carries**: The claim the target belongs to.
analysis.targets[].expected_value — number or null - **Required**: yes, in every document - **What it carries**: The published value.
analysis.targets[].id — string - **Required**: yes, in every document - **What it carries**: The identifier of the target within the record.
analysis.targets[].note — string or null - **Required**: optional - **What it carries**: A note about the target, in the words of the source.
analysis.targets[].operation — string or null - **Required**: yes, in every document - **What it carries**: The engine operation that produces the value.
analysis.targets[].state — enum — one of `absent`, `partial`, `complete`, `blocked` - **Required**: yes, in every document - **What it carries**: How complete the target is.
analysis.targets[].tolerance — number or null - **Required**: yes, in every document - **What it carries**: The tolerance a recomputed value must fall inside.
analysis.targets[].unit — string or null - **Required**: yes, in every document - **What it carries**: The unit of the published value.
assays — array - **Required**: optional - **What it carries**: Every assay the experiment ran, in the order it ran them. An experiment is a set of processes and each process is an assay: the measurement, the analysis, the validation, the normalisation. Every binding here is required for the experiment as described; there is no primary assay and no optional one. A consumer that wants to run the experiment walks the list in order.
assays[].confidence — number or null - **Required**: optional - **What it carries**: How strongly the evidence supports this binding, in [0,1]. An evidence score, not a probability.
assays[].definition — object or null - **Required**: optional - **What it carries**: The assay definition this record belongs to, with the contract the record relies on. The definition itself is owned by the catalogue; this is the binding.
assays[].definition.definition_digest — string or null - **Required**: optional - **What it carries**: The checksum of the assay definition, so the binding names the exact definition it was made against.
assays[].definition.definition_id — string - **Required**: yes, in every document - **What it carries**: The identifier of the assay definition.
assays[].definition.definition_version — string - **Required**: yes, in every document - **What it carries**: The version of the assay definition this record is bound to.
assays[].definition.input_contract — object or null - **Required**: optional - **What it carries**: What the assay definition takes as input, as the catalogue states it.
assays[].definition.interpretation — string or null - **Required**: optional - **What it carries**: How a result of this assay should be read: what a reproduction of it would and would not show. Free text, written by the assay's owner.
assays[].definition.limitations — array or null - **Required**: optional - **What it carries**: The standing limitations of the operation: what it cannot establish and the assumptions it rests on. They live here and not in `analysis.methods[]`, which forbids extra keys: an operation's limits are a property of the operation, and the catalogue owns the operation.
assays[].definition.output_contract — object or null - **Required**: optional - **What it carries**: What the assay definition returns, as the catalogue states it.
assays[].definition.parameter_contract — object or null - **Required**: optional - **What it carries**: The parameters the assay definition accepts, and their types and defaults, as the catalogue states them.
assays[].definition.required_operations — array - **Required**: optional - **What it carries**: The operations the record needs from that assay definition. The list is an array of operation names.
assays[].definition.support_state — enum — one of `supported`, `partial`, `unsupported`, `unknown` - **Required**: optional - **What it carries**: Whether the platform supports the record's operations: supported, partial, unsupported or unknown.
assays[].definition_digest — string or null - **Required**: optional - **What it carries**: The checksum of the assay definition.
assays[].definition_id — string - **Required**: optional - **What it carries**: The identifier of the assay definition.
assays[].definition_version — string - **Required**: optional - **What it carries**: The version of the assay definition.
assays[].depends_on — array or null - **Required**: optional - **What it carries**: The assay entries of this document whose output this entry needs, named by their identifiers within this document. It is a statement about the experiment as it ran, and it is written, never inferred from the order of the array: an entry that states no dependency states no dependency, and an ordered array does not say that each entry needs the one before it. A dependency names an entry in this document; it does not cross to another record, and the dependencies of a document do not form a cycle.
assays[].evidence — array - **Required**: optional - **What it carries**: The passages of the paper that support this binding, so the binding can be checked against the paper's own text.
assays[].id — string - **Required**: yes, in every document - **What it carries**: This entry's identifier within the record. It is required, because a request or a result names it and a reference that may be missing is not a reference. The order of the list is the order of the experiment, so the identifier is the position: `assay:1` is the first assay the experiment ran.
assays[].operations — array or null - **Required**: optional - **What it carries**: The operations this assay may be run as, each with the contract of that operation. An assay definition can implement several operations and the definition is still one assay, so the operations are stated inside the entry rather than by repeating the assay in the array. Optional: an entry that states no operations says nothing about how it may be run.
assays[].operations[].input_contract — object or null - **Required**: optional - **What it carries**: What this operation takes as input, as the catalogue states it.
assays[].operations[].interpretation — string or null - **Required**: optional - **What it carries**: How a result of this operation should be read: what a reproduction of it would and would not show. Free text, written by the owner of the operation.
assays[].operations[].limitations — array or null - **Required**: optional - **What it carries**: The standing limitations of this operation: what it cannot establish and the assumptions it rests on.
assays[].operations[].operation — string - **Required**: yes, in every document - **What it carries**: The name of the operation, as the owner of the catalogue publishes it, so a consumer can look it up.
assays[].operations[].operation_version — string or null - **Required**: optional - **What it carries**: The version of the operation, when the owner publishes one. The name alone does not say which version of an operation was run.
assays[].operations[].output_contract — object or null - **Required**: optional - **What it carries**: What this operation returns, as the catalogue states it.
assays[].operations[].parameter_contract — object or null - **Required**: optional - **What it carries**: The parameters this operation accepts, and their types and defaults, as the catalogue states them.
assays[].operations[].support_state — enum — one of `supported`, `partial`, `unsupported`, `unknown` - **Required**: optional - **What it carries**: How well this operation can be run today: supported, partial, unsupported or unknown. It is a property of this operation and can differ from the support state of the assay that contains it.
assays[].order_basis — string or null - **Required**: optional - **What it carries**: How this assay's place in the order was established: `paper_text_order:` when the paper's own text describes this process at that offset, `consumes_earlier_output` when it consumes what the earlier ones produced. Free text. The order of the list is the order of the experiment.
assays[].request — object or null - **Required**: optional - **What it carries**: A request to run one operation against one source record. The source record id and the hash of its body bind the request to the version that was asked for.
assays[].request.assay_id — string - **Required**: yes, in every document - **What it carries**: The identifier of the assay entry in the source document's `assays[]` that this request runs. It is required. Together with `source_content_hash` it says exactly which entry, in which version of the record, the request concerns.
assays[].request.inputs — array - **Required**: optional - **What it carries**: The input values the request carries, one object per input.
assays[].request.operation — string - **Required**: yes, in every document - **What it carries**: The DSL operation the request asks for.
assays[].request.operation_version — string or null - **Required**: optional - **What it carries**: The version of that operation the request asks for.
assays[].request.parameters — object or null - **Required**: optional - **What it carries**: The parameters the request sets, as the operation takes them.
assays[].request.provider — string or null - **Required**: optional - **What it carries**: The provider the request asks to run it.
assays[].request.request_id — string or null - **Required**: optional - **What it carries**: The identifier of the request itself, so the same request can be made twice and told apart.
assays[].request.source_content_hash — string - **Required**: yes, in every document - **What it carries**: The hash of that record's body when the request was made, so the request is tied to one version of it.
assays[].request.source_record_id — string - **Required**: yes, in every document - **What it carries**: The identifier of the record the request runs against.
assays[].required_operations — array - **Required**: optional - **What it carries**: The operations the record needs from that assay definition.
assays[].result — object or null - **Required**: optional - **What it carries**: The outcome of one request. The input, result and evidence hashes bind the outcome to the input it came from and to the evidence for it.
assays[].result.assay_id — string - **Required**: yes, in every document - **What it carries**: The identifier of the assay entry in the source document's `assays[]` that this result belongs to. It is required, so an outcome always names the assay it came from.
assays[].result.evidence_hash — string or null - **Required**: optional - **What it carries**: The hash of the evidence the run produced with the result.
assays[].result.execution_id — string - **Required**: yes, in every document - **What it carries**: The identifier of the execution that produced the outcome.
assays[].result.finding — string or null - **Required**: optional - **What it carries**: What the run found, in one sentence.
assays[].result.input_hash — string or null - **Required**: optional - **What it carries**: The hash of the input the run was given.
assays[].result.operation — string or null - **Required**: optional - **What it carries**: The operation this result is the result of. An assay entry may be run as more than one operation, so a result that does not say which operation produced it says less than the reader needs.
assays[].result.provider — string or null - **Required**: optional - **What it carries**: The provider that ran it.
assays[].result.result — object or null - **Required**: optional - **What it carries**: The operation's own result body, as the operation returns it.
assays[].result.result_hash — string or null - **Required**: optional - **What it carries**: The hash of the result the run produced.
assays[].result.status — string - **Required**: yes, in every document - **What it carries**: What the run did: succeeded, failed, refused, or whatever vocabulary the provider uses.
assays[].role — string or null - **Required**: optional - **What it carries**: What this process is: the measurement, the analysis, the validation, the normalisation. Free text. It names the process; it does not make the process optional, because every assay in the list is required.
assays[].source_hashes — object - **Required**: optional - **What it carries**: The checksums of the artefacts the binding was made against.
assays[].support_state — enum — one of `supported`, `partial`, `unsupported`, `unknown` - **Required**: optional - **What it carries**: Whether the platform supports the record's operations: supported, partial, unsupported or unknown.
capability — object - **Required**: yes, in every document - **What it carries**: The capabilities the experiment needs to run.
capability.gaps — array - **Required**: yes, in every document - **What it carries**: What is missing from the capability layer.
capability.requirements — array - **Required**: yes, in every document - **What it carries**: Each capability the record needs.
capability.requirements[].assay_families — array - **Required**: yes, in every document - **What it carries**: The assay families the capability belongs to.
capability.requirements[].availability_state — string or null - **Required**: yes, in every document - **What it carries**: Whether the capability is owned, candidate or missing.
capability.requirements[].candidate — string or null - **Required**: optional - **What it carries**: Whether the capability is a candidate for building.
capability.requirements[].capability_id — string - **Required**: yes, in every document - **What it carries**: The identifier of the capability in the engine's catalogue.
capability.requirements[].label — string or null - **Required**: yes, in every document - **What it carries**: The human-readable name of the capability.
capability.requirements[].tier_unlocked — string or null - **Required**: yes, in every document - **What it carries**: The readiness tier the capability unlocks when it exists.
capability.state — enum — one of `absent`, `partial`, `complete`, `blocked` - **Required**: yes, in every document - **What it carries**: How complete the capability layer is.
content_hash — string or null - **Required**: yes, in every document - **What it carries**: SHA-256 over the canonical JSON of the record body, as `sha256:`. A consumer recomputes it to check the record.
created_at — string or null - **Required**: optional - **What it carries**: When the record was first published, in UTC.
data — object - **Required**: yes, in every document - **What it carries**: The data the work rests on: every file, table or repository route, with its checksum and access state.
data.assets — array - **Required**: yes, in every document - **What it carries**: Every file, table or repository route the record rests on.
data.assets[].access — enum — one of `open`, `restricted`, `embargoed`, `unknown` - **Required**: yes, in every document - **What it carries**: Whether the asset is open, restricted, embargoed or of unknown access.
data.assets[].bytes — integer or null - **Required**: yes, in every document - **What it carries**: The size of the asset in bytes, or null when it is not known.
data.assets[].format — string - **Required**: yes, in every document - **What it carries**: The format of the asset, as the engine names formats: `jats-table`, `geo-series`, `vcf`, `csv`, `xlsx` and so on.
data.assets[].id — string - **Required**: yes, in every document - **What it carries**: The identifier of the asset within the record.
data.assets[].identifier — string or null - **Required**: optional - **What it carries**: The accession, DOI or locator that names the asset elsewhere.
data.assets[].media_type — string - **Required**: yes, in every document - **What it carries**: The declared media type of the asset.
data.assets[].note — string or null - **Required**: optional - **What it carries**: A note about the asset, in the words of the source.
data.assets[].reader — string or null - **Required**: yes, in every document - **What it carries**: The engine component that reads the asset.
data.assets[].reader_status — enum — one of `available`, `converter_required`, `reader_required`, `unknown` - **Required**: yes, in every document - **What it carries**: Whether that reader exists today: available, converter required, reader required or unknown.
data.assets[].role — enum — one of `manuscript`, `protocol`, `raw`, `processed`, `derived`, `code`, `supplementary`, `figure`, `table`, `reference` - **Required**: yes, in every document - **What it carries**: What the asset is: the manuscript, a protocol, raw or processed data, derived data, code, supplementary material, a figure, a table or a reference.
data.assets[].sha256 — string or null - **Required**: yes, in every document - **What it carries**: The checksum of the asset's own bytes, as `sha256:`, or null when the asset was not read.
data.assets[].state — enum — one of `declared`, `resolved`, `verified`, `unavailable` - **Required**: yes, in every document - **What it carries**: How far the asset was resolved: declared, resolved, verified or unavailable.
data.assets[].structure — object or null - **Required**: yes, in every document - **What it carries**: What the asset contains: its shape, its parts and where they sit.
data.assets[].table — object or null - **Required**: optional - **What it carries**: The table itself, for an asset whose role is `table`.
data.assets[].table.caption — string or null - **Required**: yes, in every document - **What it carries**: The caption of the table.
data.assets[].table.column_count — integer - **Required**: yes, in every document - **What it carries**: How many columns the table has.
data.assets[].table.header_rows — integer - **Required**: yes, in every document - **What it carries**: How many rows at the top of the table are headings.
data.assets[].table.label — string or null - **Required**: yes, in every document - **What it carries**: The label of the table in the article, for example `Table 1`.
data.assets[].table.locator — string - **Required**: yes, in every document - **What it carries**: Where the table sits in the article, for example `section[1]/table-wrap[1]#T1`.
data.assets[].table.numeric_cells — integer - **Required**: yes, in every document - **What it carries**: How many cells hold a number. A table with none cannot carry a published result.
data.assets[].table.row_count — integer - **Required**: yes, in every document - **What it carries**: How many rows the table has.
data.assets[].table.rows — array - **Required**: yes, in every document - **What it carries**: The rows of the table, each a list of cells.
data.assets[].uri — string or null - **Required**: yes, in every document - **What it carries**: Where the asset can be fetched, or null when it lives inside the record.
data.assets[].version — string or null - **Required**: optional - **What it carries**: The version of the asset, where the source states one.
data.gaps — array - **Required**: yes, in every document - **What it carries**: What is missing from the data layer: every asset that was not read, downloaded or licensed.
data.group_sizes — object or null - **Required**: optional - **What it carries**: The number of samples in each comparison group, as the repository metadata states it. It is what separates a reproduction from a re-analysis.
data.state — enum — one of `absent`, `partial`, `complete`, `blocked` - **Required**: yes, in every document - **What it carries**: How complete the data layer is.
execution — object - **Required**: yes, in every document - **What it carries**: Whether the experiment has been run, and the evidence if it has.
execution.gaps — array - **Required**: yes, in every document - **What it carries**: What is missing from the execution layer.
execution.mode — string or null - **Required**: yes, in every document - **What it carries**: Whether the run was real or synthetic.
execution.runs — array - **Required**: yes, in every document - **What it carries**: The runs of the experiment, each with its evidence.
execution.state — enum — one of `absent`, `partial`, `complete`, `blocked` - **Required**: yes, in every document - **What it carries**: How complete the execution layer is.
execution.status — string or null - **Required**: yes, in every document - **What it carries**: The execution status, as the run reports it.
experiment — object - **Required**: yes, in every document - **What it carries**: The experiment behind the paper: what was done, with what, in what order.
experiment.conditions — object - **Required**: yes, in every document - **What it carries**: The conditions the experiment runs under, keyed by name.
experiment.gaps — array - **Required**: yes, in every document - **What it carries**: What is missing from the experiment layer.
experiment.instruments — array - **Required**: yes, in every document - **What it carries**: The instruments the experiment uses.
experiment.materials — array - **Required**: yes, in every document - **What it carries**: The materials the experiment consumes.
experiment.objective — string or null - **Required**: yes, in every document - **What it carries**: What the experiment set out to do.
experiment.reagents — array - **Required**: yes, in every document - **What it carries**: The reagents the experiment consumes.
experiment.state — enum — one of `absent`, `partial`, `complete`, `blocked` - **Required**: yes, in every document - **What it carries**: How complete the experiment layer is.
experiment.steps — array - **Required**: yes, in every document - **What it carries**: The steps of the experiment, in order.
experiment.steps[].blocked_on — string or null - **Required**: optional - **What it carries**: What stops the step running: a missing datum, a missing method, a missing capability, or null.
experiment.steps[].description — string or null - **Required**: optional - **What it carries**: What the step does.
experiment.steps[].duration_hours — number or null - **Required**: yes, in every document - **What it carries**: How long the step takes, in hours.
experiment.steps[].id — string - **Required**: yes, in every document - **What it carries**: The identifier of the step within the record.
experiment.steps[].inputs — array - **Required**: yes, in every document - **What it carries**: The assets the step consumes.
experiment.steps[].instruments — array - **Required**: optional - **What it carries**: The instruments the step uses.
experiment.steps[].kind — enum — one of `physical`, `computational`, `data-preparation`, `analysis`, `other` - **Required**: yes, in every document - **What it carries**: Whether the step is physical, computational, data preparation, analysis or other.
experiment.steps[].name — string or null - **Required**: yes, in every document - **What it carries**: The name of the step.
experiment.steps[].order — integer - **Required**: yes, in every document - **What it carries**: The position of the step in the sequence.
experiment.steps[].outputs — array - **Required**: yes, in every document - **What it carries**: The assets the step produces.
experiment.steps[].parameters — object - **Required**: yes, in every document - **What it carries**: The parameters of the step, keyed by name.
experiment.steps[].reagents — array - **Required**: optional - **What it carries**: The reagents the step consumes.
experiment.steps[].reagents[].catalogue — string or null - **Required**: optional - **What it carries**: The catalogue the reagent comes from, where it is named.
experiment.steps[].reagents[].name — string or null - **Required**: yes, in every document - **What it carries**: The name of the reagent.
experiment.steps[].reagents[].quantity — number or null - **Required**: yes, in every document - **What it carries**: How much of the reagent is used.
experiment.steps[].reagents[].unit — string or null - **Required**: yes, in every document - **What it carries**: The unit of that quantity.
experiment.steps[].state — enum — one of `absent`, `partial`, `complete`, `blocked` - **Required**: yes, in every document - **What it carries**: How complete the step is.
gaps — array - **Required**: yes, in every document - **What it carries**: Every gap in the record, gathered from the layers.
generator — object or null - **Required**: optional - **What it carries**: The tool and converter that wrote the record, and the input dialect it was read from.
generator.converter — free - **Required**: optional - **What it carries**: The converter that turned the input dialect into this record.
generator.name — free - **Required**: optional - **What it carries**: The name of the tool that wrote the record, for example `corpus-assay-engine`.
generator.source_package_schema — free - **Required**: optional - **What it carries**: The URN of the input dialect the record was converted from.
id — string - **Required**: yes, in every document - **What it carries**: The record identifier. A corpus record is `pmid:`; a user-authored fork is `fork::`.
kind — any — one of `science-record` - **Required**: yes, in every document - **What it carries**: What kind of record this is. Always `science-record` in version 1.
lineage — object or null - **Required**: optional - **What it carries**: The version this record supersedes, and the generation it belongs to.
lineage.fork_of — string or null - **Required**: optional - **What it carries**: The record this one is a fork of, for a user-authored fork.
lineage.generation — integer or null - **Required**: optional - **What it carries**: How many revisions deep the record is.
lineage.parent_hash — string or null - **Required**: optional - **What it carries**: The exact content hash of the revision this one replaces.
lineage.parent_id — string or null - **Required**: optional - **What it carries**: The identifier of the record this one supersedes.
paper — object - **Required**: yes, in every document - **What it carries**: The source: what the paper says, and the claims it makes.
paper.abstract — string or null - **Required**: yes, in every document - **What it carries**: The abstract of the paper.
paper.authors — array or null - **Required**: yes, in every document - **What it carries**: The authors of the paper, as a list of names.
paper.claims — array - **Required**: yes, in every document - **What it carries**: The claims the paper makes, each with its target and its evidence.
paper.claims[].evidence — array - **Required**: yes, in every document - **What it carries**: The evidence for the claim: where in the record it came from and what was quoted.
paper.claims[].evidence[].asset — string or null - **Required**: yes, in every document - **What it carries**: The asset the quotation was taken from.
paper.claims[].evidence[].locator — string or null - **Required**: yes, in every document - **What it carries**: Where in the asset the quotation sits.
paper.claims[].evidence[].quote — string or null - **Required**: yes, in every document - **What it carries**: The quoted text, checked against the asset it names.
paper.claims[].id — string - **Required**: yes, in every document - **What it carries**: The identifier of the claim within the record.
paper.claims[].kind — enum — one of `quantitative`, `qualitative`, `methodological` - **Required**: yes, in every document - **What it carries**: Whether the claim is quantitative, qualitative or methodological.
paper.claims[].state — enum — one of `absent`, `partial`, `complete`, `blocked` - **Required**: yes, in every document - **What it carries**: How complete the claim is.
paper.claims[].statement — string or null - **Required**: yes, in every document - **What it carries**: The claim, as the paper states it.
paper.claims[].target — object or null - **Required**: yes, in every document - **What it carries**: The published value the claim is about, where the paper states one.
paper.claims[].target.source — string or null - **Required**: yes, in every document - **What it carries**: Where the published value came from: a table, a figure or the text.
paper.claims[].target.tolerance — number or null - **Required**: yes, in every document - **What it carries**: The tolerance a recomputed value must fall inside for the claim to be reproduced.
paper.claims[].target.unit — string or null - **Required**: yes, in every document - **What it carries**: The unit of the published value.
paper.claims[].target.value — number or null - **Required**: yes, in every document - **What it carries**: The published value, as a number.
paper.doi — string or null - **Required**: yes, in every document - **What it carries**: The DOI of the paper.
paper.gaps — array - **Required**: yes, in every document - **What it carries**: What is missing from the paper layer.
paper.journal — string or null - **Required**: yes, in every document - **What it carries**: The journal the paper was published in.
paper.pmid — string or null - **Required**: yes, in every document - **What it carries**: The PubMed identifier of the paper.
paper.state — enum — one of `absent`, `partial`, `complete`, `blocked` - **Required**: yes, in every document - **What it carries**: How complete the paper layer is: absent, partial, complete or blocked.
paper.title — string or null - **Required**: yes, in every document - **What it carries**: The title of the paper.
paper.year — integer or null - **Required**: yes, in every document - **What it carries**: The year of publication.
provenance — object - **Required**: yes, in every document - **What it carries**: Where every part of the record came from.
provenance.activities — array - **Required**: yes, in every document - **What it carries**: The acts that produced the record: reading, extraction, conversion, publication.
provenance.activities[].agent — free - **Required**: optional - **What it carries**: The agent that carried out the activity.
provenance.activities[].id — free - **Required**: optional - **What it carries**: The identifier of the activity.
provenance.activities[].type — free - **Required**: optional - **What it carries**: What the activity was, for example `read` or `convert`.
provenance.activities[].used — free - **Required**: optional - **What it carries**: The entity the activity used.
provenance.agents — array - **Required**: yes, in every document - **What it carries**: Who or what carried out each activity.
provenance.agents[].id — free - **Required**: optional - **What it carries**: The identifier of the agent.
provenance.agents[].label — free - **Required**: optional - **What it carries**: The human-readable name of the agent.
provenance.agents[].type — free - **Required**: optional - **What it carries**: What kind of agent it is, for example `person` or `tool`.
provenance.capability_table_fingerprint — string or null - **Required**: optional - **What it carries**: The fingerprint of the capability table the record was built against.
provenance.dsl_catalogue_fingerprint — string or null - **Required**: optional - **What it carries**: The fingerprint of the operation catalogue the record was built against.
provenance.entities — array - **Required**: yes, in every document - **What it carries**: The things the record is about: papers, datasets, instruments, people.
provenance.entities[].id — free - **Required**: optional - **What it carries**: The identifier of the entity.
provenance.entities[].label — free - **Required**: optional - **What it carries**: The human-readable name of the entity.
provenance.entities[].type — free - **Required**: optional - **What it carries**: What kind of thing the entity is, for example `paper` or `dataset`.
provenance.snapshot_id — string or null - **Required**: optional - **What it carries**: The corpus snapshot the record was read from.
provenance.source_run_id — string or null - **Required**: yes, in every document - **What it carries**: The corpus run the record was read from.
readiness — object - **Required**: yes, in every document - **What it carries**: How ready the record is to run, and what has to be resolved first.
readiness.axes — object - **Required**: yes, in every document - **What it carries**: The state of each axis: method, capability and data.
readiness.axes.capability — free - **Required**: optional - **What it carries**: The capability axis: the tier, the blocker and the detail behind it.
readiness.axes.data — free - **Required**: optional - **What it carries**: The data axis: the tier, the blocker and the detail behind it.
readiness.axes.method — free - **Required**: optional - **What it carries**: The method axis: the tier, the blocker and the detail behind it.
readiness.blocked_on — string or null - **Required**: yes, in every document - **What it carries**: The specific blocker, in the form `axis:reason`.
readiness.blocking_axis — enum — one of `method`, `capability`, `data`, `none`, `None` - **Required**: yes, in every document - **What it carries**: Which axis blocks the record: method, capability, data or none.
readiness.can_run — object - **Required**: yes, in every document - **What it carries**: What can run today: the operations, the assets and the steps that are available.
readiness.can_run.assets — array - **Required**: yes, in every document - **What it carries**: The assets that can be read today.
readiness.can_run.dsl_operations — array - **Required**: yes, in every document - **What it carries**: The engine operations that can run today.
readiness.can_run.steps — array - **Required**: yes, in every document - **What it carries**: The protocol steps that can run today.
readiness.gaps — array - **Required**: yes, in every document - **What it carries**: What is missing from the readiness layer.
readiness.must_build — object - **Required**: yes, in every document - **What it carries**: What has to be built before the record can run.
readiness.must_build.capabilities — array - **Required**: yes, in every document - **What it carries**: The capabilities that have to be built.
readiness.must_build.converters — array - **Required**: yes, in every document - **What it carries**: The converters that have to exist, named by the format they read.
readiness.must_build.data_gaps — array - **Required**: yes, in every document - **What it carries**: The data that has to be obtained before the record can run.
readiness.must_build.dsl_operations — array - **Required**: yes, in every document - **What it carries**: The engine operations that have to exist.
readiness.must_build.dsl_operations[].operation — string - **Required**: yes, in every document - **What it carries**: The operation that has to exist.
readiness.must_build.dsl_operations[].priority — integer or null - **Required**: yes, in every document - **What it carries**: The order the operation should be built in.
readiness.must_build.dsl_operations[].unlocks_papers — integer or null - **Required**: yes, in every document - **What it carries**: How many papers that one operation would unblock.
readiness.must_build.readers — array - **Required**: yes, in every document - **What it carries**: The readers that have to exist, named by the format they read.
readiness.state — enum — one of `absent`, `partial`, `complete`, `blocked` - **Required**: yes, in every document - **What it carries**: How complete the readiness layer is.
readiness.tier — enum — one of `R0`, `R1`, `R2`, `R3`, `R4`, `R5`, `None` - **Required**: yes, in every document - **What it carries**: The readiness tier: R0 runs today, R5 needs the most.
review — object - **Required**: yes, in every document - **What it carries**: What the readers made of the paper, and where that critique is published.
review.gaps — array - **Required**: yes, in every document - **What it carries**: What is missing from the review layer.
review.human_critique — object or null - **Required**: yes, in every document - **What it carries**: The human critique of the record.
review.human_critique.disagreements — free - **Required**: optional - **What it carries**: Where the critique and the machine reading disagreed.
review.human_critique.providers — free - **Required**: optional - **What it carries**: The people who wrote the critique.
review.human_critique.status — free - **Required**: optional - **What it carries**: The state of the human critique.
review.machine_critique — object or null - **Required**: yes, in every document - **What it carries**: The machine critique of the record.
review.machine_critique.agreement — free - **Required**: optional - **What it carries**: The agreement between the readers.
review.machine_critique.agreement_basis — free - **Required**: optional - **What it carries**: What the agreement figure was measured against.
review.machine_critique.blinded — free - **Required**: optional - **What it carries**: Whether the readers could see each other's answers.
review.machine_critique.disagreements — free - **Required**: optional - **What it carries**: Where the readers disagreed, quoted.
review.machine_critique.providers — free - **Required**: optional - **What it carries**: The readers that produced the critique, each with its model and its provider.
review.machine_critique.status — free - **Required**: optional - **What it carries**: The state of the machine critique: absent, partial, complete or blocked.
review.machine_critique.target_providers — free - **Required**: optional - **What it carries**: The readers that were asked for the published target.
review.stages — array - **Required**: yes, in every document - **What it carries**: The review stages the record has been through.
review.state — enum — one of `absent`, `partial`, `complete`, `blocked` - **Required**: yes, in every document - **What it carries**: How complete the review layer is.
revision — integer - **Required**: yes, in every document - **What it carries**: The counter that increases each time the body of the record changes. The first publication is revision 1.
schema — any — one of `https://corpus.bntrg.co.uk/sci/1.0/sci.schema.json` - **Required**: optional - **What it carries**: The absolute URI of the JSON Schema the document validates against. It resolves to the schema and to this specification.
sci_version — enum — one of `1.0`, `1.1`, `1.2`, `1.3`, `1.4` - **Required**: yes, in every document - **What it carries**: The minor version of the format this document is written in. 1.0, 1.1 and 1.2 are accepted at the same schema URI.
updated_at — string or null - **Required**: optional - **What it carries**: When the record body last changed, in UTC.
### Contributing to the format The schema, this reference and the converter are one format and they move together. A change to any of them is a change to the format. 1. Edit the schema in `sci/schema/sci-1.4.schema.json` and the description of the field in `sci/field_descriptions.json`. 2. Run the converter and the validator: `python3 tools/sci_build.py` then `python3 -m unittest discover -s sci/tests -t .`. 3. Write the change down in the change-control section above, with the version it belongs to and why the format needs it. 4. Publish the schema and this reference together, so the two that a consumer reads are the two that were tested. A change that only adds a field, widens a description or fixes a wording is a documentation change and does not move the version. A change that accepts or rejects a document a different version would reject is a minor version, published at the same schema URI. A change that removes or renames a field is a major version and a new URI.