The term annealment is borrowed from metallurgy, in which repeated heat treatment brings a metal to a durable structure. Knowledge behaves in the same way. A corpus begins as text, which is words on pages. Every pass that adds verified structure, such as clause anchors, defined terms, typed tables or executable constraints, anneals the corpus into something firmer: a body of knowledge that can be measured, cited and, at the top of the ladder, executed. As with metal, the structure is proven by testing rather than assumed, and every pass is followed by an evaluation cycle that either confirms the improvement or sends it back for rework.
This page is the instrument for that process. It presents the ladder of question classes that the corpus can answer, in which each class becomes answerable only because a specific kind of structure was added. It reports the measured effect of each data level on the ladder, and it describes the capabilities that lie beyond retrieval altogether.
The data levels being compared
All markup is stripped; the corpus is words on pages.
Answers 2 of the 18 graded questions.
The AsciiDoc markup is kept, so clause titles, tables and cross-references survive as text.
Answers 3 of the 18 graded questions.
Metanorma Knowledge Objects (MKO): typed units with clause anchors, cross-references and figure assets.
Answers 6 of the 18 graded questions.
The machine-readable Recommendation model: typed values, constraints with numeric limits, tests and state sequences.
Answers 13 of the 18 graded questions.
An ablation of the executable model with the structure flattened into prose-like units.
Answers 12 of the 18 graded questions.
The typed-unit and executable-model representations composed into one index.
Answers 14 of the 18 graded questions.
Each level indexes the same corpus, which is the R 60 family of publications, with the same embedding, enrichment and serving configuration. Only the representation differs, so the ladder measures the effect of the representation.
The tiers of data
Six experimental levels, the production corpus and the members-only ISO corpus are separated by a single controlled boundary: every unit is validated on entry and committed only to the index it was declared for, so an experimental corpus can never enter the production index and public content can never reach the members-only index.
The ladder
Each question class requires strictly more structure than the class before it. Selecting a probe opens it in the chat.
// The knowledge-annealment ladder — the L0–L9 staircase of question // classes, the capability ceilings of the measured representations, // the frontier beyond L9, and the three eras. Hand-authored inline SVG // riding the site's CSS tokens (light/dark). --- ---The system finds the clause where the corpus addresses a topic and cites it. This is the baseline that every representation is expected to pass.
Expected answer: A citation anchored in R 60-1 §3 or in R 60-2 §2.11, where creep is addressed.
Expected answer: A citation of R 60-1 §6.2, which carries the marking requirements.
The system returns a normative value verbatim from the prose. The only difficulty is finding the sentence that carries it.
Expected answer: The value 0.7, quoted from the note that instructs the creep limit to use this value regardless of the manufacturer's declaration.
Expected answer: The date 2021-01-01, quoted as the validity date of the 2021 edition.
The system connects the vocabulary the user writes to the corpus's defined terms, and returns the authoritative definition with the clause that defines it.
Expected answer: The term durability, which the vocabulary register defines through VIML clause 5.15.
Expected answer: VIML 2022 clause 5.15, cited as the source of the concept.
The question asks for a value whose retrieval requires identifying both the row and the column of a table. The answer must be delivered as the typed table object, not retyped as prose.
Expected answer: The value 5000, delivered as the table block with the class B row.
Expected answer: Verification intervals (v), stated in the typed column of the table.
The answer must join two or more clauses, often together with a table, and return both citations together with the relation between them.
Expected answer: Three citations together: the requirement in R 60-1, the repeatability test in R 60-2, and the report form in R 60-3.
Expected answer: The MPE table with its tier breakpoints, together with the apportioning factor.
The answer follows a reference from one standard to another and returns the linked identity. The public data levels return the link itself; ISO/IEC content never enters a public index.
Expected answer: The cited ISO/IEC document identifier and clause, returned as the link itself.
Expected answer: ISO/IEC 17000 and ISO/IEC 17065, cited as the governing vocabularies.
The answer depends on which edition of a publication applies, or on what changed between two editions.
Expected answer: The 2021 edition, which supersedes the 2017 edition.
Expected answer: The 2017 edition, determined from its validity window.
The evidence for the answer is in the drawing itself rather than in the caption text, so the system must read the image.
Expected answer: The labels A, B and C, read from the drawing itself.
The answer is a value that the documents define but never print, so it must be computed from the model. The expected result is deterministic, so no judge is involved in grading.
Expected answer: The verdict is fail, because d_max must be at least 0.9 times E_max, which is 27000, and the standard's own violation wording accompanies the verdict.
The question concerns particular objects and ordered activities. Only representations that record instances and sequences can answer them.
Expected answer: The answer is no, because the MDLO test establishes the baseline and running creep first contaminates it.
Expected answer: R 60-3 §2.1.3, in the execution form.
Where each data level wins
The results come from the full six-level matrix, in which 18 class-tagged probes were graded by witness containment. Each class is mapped to the first data level that clears it and to the levels that never do. A richer representation is not uniformly better, because different representations unlock different capabilities.
| Class | First level that clears it | The capability that unlocks it | Levels that never do |
|---|---|---|---|
| L0 locate and cite | D Executable model (2/2) | Every unit carries a stable clause anchor, while the plain-text and AsciiDoc levels carry anchors on only about half of their chunks. | A, B |
| L1 extract values | D Executable model (2/2) | Typed attributes carry exact values together with their provenance. | none |
| L2 nomenclature | F Composed (2/2) | The everyday phrasing of the question meets the model's defined terms through the published vocabulary; the model alone has the terms but not the everyday phrasing. | A, B, C |
| L3 tabular geometry | production serving (2/2) | Typed tables require the full pipeline, including the family pin and the contract enforcement, to be delivered as objects. | no single raw-text data level |
| L4 composition | C Typed units (2/2) | Typed cross-references make the relation between a requirement and its validating test a link rather than a sentence. | A, B |
| L5 cross-standard | partial everywhere; production only (2/2) | Answering across standards requires the referenced standards to be indexed, which is an artifact of the isolation design rather than a limit of the representations. | A, B, C, E, F |
| L6 diachrony | D Executable model / F (1/2) | Edition lineage is carried as data. Complete jurisdiction answers require machine models of several editions. | C |
| L7 perception | C Typed units (1/1) | Figure objects carry captions and image references; the model levels carry no figures at all. | A, B, D, E |
| L8 computation | D Executable model (1/1) | The machine limit exists as an executable expression in the model level and as prose only in every text level. | A, B, C |
| L9 instance and process order | D Executable model (2/2) | Test sequences and the preconditions for valid runs are recorded as typed objects. | A, B |
Across the 18 probes, the data levels answer as follows: A plain text 2, B raw AsciiDoc 3, C typed units 6, D executable model 13, E executable model flattened 12, and F composed 14. The production serving stack, which composes the richest representation with retrieval, steering and verification techniques, answers 17 to 18. The serving techniques are therefore worth about three classes over the best single data level, which means that representation quality and serving quality multiply rather than add.
Beyond retrieval
The following capabilities cannot be provided by retrieval under any data level, because they require the executable semantics of the model: a closed world of requirements, machine-checkable constraints and recorded instances. Five of them are live in the service today, and the remainder are waiting on named prerequisites that lie outside this service.
Conformance questions are answered by execution: the service evaluates the machine-checkable rule against the values in the question and attaches the verdict, with the full chain of clauses, as data.
Hypothetical parameter values are evaluated against the same constraints, and table lookups run inverted. Documents state facts; the model evaluates hypotheses.
Because every requirement and test is enumerated, the question of whether R 60 requires something it does not mention is answered by an enumeration certificate rather than a refusal.
The corpus is machine-checked for consistency, and the assistant can verify its own model-grounded claims by execution.
A normative note such as the instruction that the creep limit always uses a particular value is carried by the model as a defaulted attribute, and every derived limit applies it as a rule rather than quoting it as prose.
Severity menus of the test-method packages evaluate as machine membership: the stated quantities are checked against each set's value-plus-tolerance bands, the matching set is named, and an out-of-band combination fails naming the nearest set — computed server-side, carried as data.
With both editions modelled, the differences between them become computable, and questions about which edition governed a past evaluation are answered from validity windows.
Waiting on: Needs the complete 2017-edition model packages, which are being prepared; the current packages cover 2021.
A photograph of a nameplate or a dial is grounded under the same citation contract as published figures.
Waiting on: Grounding a member's own photograph works in the prototype; shipping it needs the account-data integration completed.
The MPE table becomes a function that is evaluated at the member's own instrument, using its maximum capacity, verification interval and accuracy class.
Waiting on: Requires recorded instrument instances from the certification system, which is a programme-level integration rather than a service feature.
The assistant reports where an evaluation stands, which step gates which, and what runs next.
Waiting on: Depends on process and gateway records that live in the certification platform, not in this corpus.
A committee asks what changes when a limit changes: every requirement, test and verdict that depends on it.
Waiting on: Needs bidirectional bindings between requirements, tests, formulas and tables to be complete for whole publications, which is committee-scale work.
The model carries its own renders, so the service can check that the published table matches the model.
Waiting on: Requires the published rendering and the model to be compared automatically, which needs the publishing pipeline to expose its renderings.
Answers cite each layer separately: the base standard says one thing, and the OIML recommendation overlays another.
Waiting on: Needs the ISO/IEC base layers as models, which exist for the internal tier but not yet as a public composition.
How a class is graded
The measurement follows one discipline, which is that grading is deterministic wherever it can be, and wherever a model judges the result, the score is labelled as judged.
- Required evidence. Each probe declares the citation it must return and a witness, which is a span of text that must appear character for character inside one of the retrieved passages. A script checks this containment mechanically, no judge participates in the check, and no partial credit is awarded.
- Structural answers. Probes that ask for a table, a formula or a figure require the answer to carry the object itself. Retyping the contents of the object as prose does not pass, because the structure of the object is part of the answer.
- Model-judged scores. Where correctness is semantic rather than mechanical, a judge model that took no part in generating the answer scores its faithfulness and relevancy over the full set of retrieved passages. These scores are always reported as judged and never as mechanical.
A change is never promoted on the strength of a single successful run. The promotion gate repeats the golden set and the annealment battery against the production service, and only repeated passes clear a change for deployment. Determinism probes additionally verify that the same question yields the same answer, and a gate is never run while the index is being modified, because a settled system is a precondition for an honest measurement.
Machine verdicts over test conditions
The frontier’s newest live capability answers a severity question by execution rather than by quotation. The test-method packages (the IEC 60068 environmental family and the IEC 61000 electromagnetic-compatibility family, cited by the Recommendations for their test procedures) encode their severity menus as condition sets: each set carries typed quantity values — a value, a unit and a tolerance, inseparable by the kernel’s quantity doctrine — with clause-level provenance back to the standard’s own table.
When a question states quantities and asks about a severity, the service evaluates the stated values against every set’s bands: a combination admitted by a set’s tolerance bands passes and names its set; a combination no set admits fails and names the nearest set with the violated bands. The arithmetic is computed server-side and attached to the answer as data — the answer model narrates the computed verdict and is instructed never to recompute or soften it. The severity membership probes in the annealment battery (the in-band and out-of-band pair on IEC 60068-2-78’s Table 1) gate this capability on every promotion, and the research notes (docs/research/llm-calculation-2024-2026.md) ground the design in the 2024–2026 tool-execution and table-QA literature.
Aggregation over typed tables
The same execution posture answers questions whose answer IS a table’s content. The model plane carries each publication’s tables as typed payloads — columns named and typed, per-column units, rows of values — so the service computes count, minimum, maximum and interval lookups directly over the payload instead of asking the language model to read numbers out of retrieved text.
Four operations, all live: a count names the distinct values of a column (“how many accuracy classes does R 60-1 define?” → 4, from the MPE tier table); a minimum or maximum reads a class column across its rows (“the shortest stabilisation time for a class B load cell” → 15 s); an interval lookup selects the row a stated quantity falls in, following the standards’ own half-open convention, and returns the named cell (“after a 500 kg load change, class C” → 30 s, from the row greater than 100 kg up to and including 1 000 kg). The question’s stated unit and class are the strongest signals when choosing the table: a question that states kilograms cannot be answered from a table whose intervals are denominated in verification scale intervals, and when no table carries a matching signal the service declines to guess.
Like the severity verdicts, the computed value rides to the client as a typed block — operation, column, value, unit and the full matched row — and the answer model is instructed to present it and cite the table’s clause, never to recompute it. The live probes above (min, count and both lookup shapes against OIML R 60 and R 91) run on every deployment.
The judge sees what the answer saw
The faithfulness judge — the instrument that scores whether every claim in an answer is supported by retrieved passages — is only as honest as its grounding. Three rounds of triage against the measured weak spots reshaped it:
- The machine grounding rides along. A computed verdict or an aggregation result is stated by the answer legitimately — it came from the service’s own arithmetic, not from a passage. The answer’s typed blocks now travel to the judge as a distinct grounding family, taught in the judge’s prompt, so a machine-computed value can never read as a hallucination. The context builder is a dependency-free module with its own tests.
- Tables speak at full length. A table’s evidence lives across its whole row set; truncating a table passage at prose length hid the very rows an answer computed from. Table passages now travel with more room in both the evidence view and the judge’s grounding.
- Verify mirrors the ask. The verify endpoint prefers the grounding the answer was actually given — the caller may pass it, and the site’s verify button does — falling back to its own retrieval only when none is supplied. Judging an answer against passages it never saw was the whole failure behind the context-family’s zero scores, and the fix took that family from 0.0 to a perfect score on the live specimen.
- Superseded editions lose on purpose. The dirty corpus’s superseded editions (a 2017 R 60-1 beside the 2021 one) compete on shared vocabulary and made current-edition answers read as ungrounded against their own past. Retrieval now tilts to the publication’s active edition — the model-bound ask by its package’s edition, every publication-named question through the profile codec’s family key — and the verify grounding excludes superseded same-document candidates outright. (The full exclusion waits on corpus metadata normalization — TODO.ai-platform/12 in the smart repo — so the ask path steers by demotion until the corpora’s doc_number and edition spell one way.)
Keeping the instrument honest
The ladder is only as trustworthy as the corpus integrity beneath it. The following invariants are enforced operationally, and each one was learned from a real failure.
- The canonical set is declared once. Every chunk file that serves production is enumerated in a single place, and reconciliation, which deletes stray vectors, consults exactly that declaration. Nothing that serves can be classified as stray.
- Enrichment is replayable. Rebuilding the index overwrites contextual enrichment with raw text, and a durable record allows every enrichment to be replayed without regeneration. The one occasion on which this step was skipped produced a measurably weaker index, so the replay is now a mandatory step and the gate detects its absence.
- The wire format obeys a law. Vector metadata may contain only scalars and arrays of strings, and a single projection enforces this rule on every path from a producer to an index, so that a rich payload cannot reject a whole batch.
- Caches carry a generation stamp. After any change to the index, the answer caches stop serving the previous generation, and stale answers expire within minutes instead of describing a corpus that no longer exists.