Part 3: The full harness

1The problem: engineering work that has to build on itself

We want an AI to do engineering work the way a small team does it over weeks: research a system, model it, run studies against the model, find what is wrong, fix it, and go around again. Part 2 showed how a study evaluates one version of the model. This part is about the rest of the loop: how the model gets its numbers, how it changes, and how anyone can tell whether a change was right.

The hard part is not any single session. A session with a good prompt does good work. The hard part is that every session starts from what earlier sessions wrote down, because that is all it has. The model, the sources, the study records and the decisions are the memory of the whole effort, and an agent's own context is gone when its session ends.

AI has inherent entropy. Left to its own devices, a modeling environment goes the way an unattended codebase does: it becomes chaotic and incomprehensible, which is what people mean by slop. Fighting that entropy is the job of the harness. The aim is a closed system that is never deterministic but stays predictable over time. Some of the problems it has to solve:

• Avoid multiple, conflicting sources of truth. The first study run under the study tooling found four values the study needed that no source in the repository supplied (.project/research/20260822-120756_research-extraction-harness.md). An agent that needs a value and cannot find one will supply a plausible one from memory, and the next session reads it as a fact.
• Keep work continuous across agent sessions, so that a fresh session can pick up where the last one stopped without redoing or undoing it.
• Keep intent consistent, so that agents working on one piece do not lose the forest for the trees.
• Hold the level of performance as the system grows in volume. That takes an architecture and a set of patterns that an agent, or a person, can still read and understand.

Part 1 argued that SysML v2's strict semantics help protect the model from this kind of entropy: the language fixes what a model can say and how, so an agent cannot redefine the semantics as it goes. But the model is a small part of the record. Nothing in the language protects the research behind a number, the study that found a problem, the decision about what to do about it, or the account of what was actually done.

So what is a harness? For this project it is four kinds of thing:

• Tools that provide deterministic structure: one script registers a source, one operation mints a work item, one pipeline regenerates the program from the model.
• Distinct prompts that define process stages, each doing a bounded chunk of work with a defined input and a defined record.
• Other agent prompts and reference material that enforce patterns and behaviors, so that reviewers, critics and rules keep the system in check.
• A file system that keeps everything organized and interpretable, so that anything the process names can be found.

Ours is far from refined. It has not demonstrated full closure over a long horizon, and it grew with need and by trial and error rather than from a design. But some of its techniques have shown promise, and the rest of this part walks through them: how the work sorts into classes, where each class writes its record, and what checks that record before the next session builds on it.

2The mental model: three classes of work, and a loop above them

The main post describes exploratory modeling as a cycle: build a rough design, evaluate it, refine it, and go again. Inside that cycle an engineer does three recurring things: finds out what is known about the system, changes the model to reflect it, and evaluates the model to see what it now says. The harness treats each as a class of work with its own tools, prompts and record, and a person can run any of them on its own.

Research answers "we need to know X." Agents search and triage candidate sources, but one script does all the writing into the knowledge base. Given a URL or a PDF and three sentences about what the source is for, how to check its numbers, and what limits its authority, it captures the source, checks it against the quarantine, and commits the extraction, a manifest row and an index entry together, or writes nothing at all (docs/research_seam_operator_guide.md). A number in the model can then cite a page in the repository, and a search that found nothing is recorded so nobody runs it again.

Model updates change the SysML v2 in an organized way. Before an edit, the agent writes down what will change, which definitions and consumers it touches, and what source the change rests on. Then it makes the change, regenerates the program, and records the result so a reviewer can check the change against what was intended rather than only whether the numbers moved the right way. The stages that exist for this (spec, design, plan, implement, review) can be brief or skipped when the change is small, but the evidence obligation does not shrink with the stage count (modeling_project/MODELING_PROCESS.md).

Studies probe a generated program after codegen. A study runs against one pinned version of the generated program, sweeps inputs, and records what pushed back, what did not, and what the agent judges that to mean. It stops only on mechanical faults, never on a result that looks wrong; a surprising result is recorded and argued, not suppressed (.claude/skills/run-study/runbook.md). Part 2 covered the machinery; here the point is that a study is a bounded unit of work with a record that a later session can read cold.

3Where everything lives

Agent sessions keep no memory between them, so the work carries forward only through the repository. A new session has to find what earlier sessions wrote and be able to trust it. We divide the repository into a few areas, one for each part of the harness described in section 2, and organize every area the same way so that a session knows how to find and add to any of them.

In Figure 2, start at the top bar, then read down each column: the heading names the work, and the cards below it are the areas that work writes to.

Each class of work from section 2 writes to its own areas, the goal loop has one, and the harness itself has one:

We reuse similar patterns across these components: an index file where a session starts, a tool that adds entries so they keep their structure, and a record that keeps the history of changes.

Indexes. Each area has one file where a session starts, and that file points to the rest. A session looking for the source behind a number reads SOURCE_INDEX.md, which says what each source is for, how to check its numbers and what limits its authority. A session resuming a goal reads goal.md and then the trail. A session can therefore find what it needs by following references from CLAUDE.md instead of searching the repository.

Write tools. Where a record needs a strict structure, agents add to it through a tool instead of editing the file. In round 1 of the goal in section 4, a research request found two papers, and source_registry.py registered each one by writing its extraction, its manifest row with the hash and its index entry in one step. If any part had failed, it would have written nothing. The model and the goal records have no write tool, for different reasons. The model's structure comes from SysML itself: the language fixes what a model can say, and the validator checks each change. We built the goal records as plain files with written rules and add a tool only when a real run shows a written rule failing (.project/adr/0003-lean-first-persistence.md). So far their templates and fixed headings have been enough.

History. Each area keeps a record that later work can rely on, and entries are added to it rather than edited. The trail is only appended to, and a correction is a dated amendment that names what it corrects. A committed study record is not changed. A finding's first sighting in the discovery log stays as written, and later rows record what was done about it. A new session can therefore read an old record and know it describes what happened at the time.

Three further rules apply across all the areas.

• Records refer to each other by path instead of copying. A number in the model cites a file in knowledge/sources/, and a goal's trail cites a study's directory instead of restating its results. Each fact has one home, so a correction cannot leave two versions that disagree.
• The status of a piece of work is read from its files, not stored separately. We avoided separate status fields because they drift out of step with the records they describe (work/orchestration/GOAL_RUNBOOK.md § Opening and closing a round).
• Searches that found nothing are recorded with the queries that were tried, and a round that ends without a result still writes one. A new session can see that the ground was already covered.

Table 1 shows how each area implements the pattern.

fusion-tea/
├── CLAUDE.md                        the first file every session reads; it points to everything below
├── models/                          the SysML v2 model
│   ├── library/                     reusable definitions: components, calculations, cost accounts
│   └── designs/stellarator_09/      the Stellaris design: its values and its design choices
├── exploration/stellarator_e2e/     what is generated from the model, and what runs against it
│   ├── models/                      a copy of models/ that codegen reads; a test fails on any difference
│   ├── generated/                   the generated program (Part 2)
│   └── studies/
│       ├── manifest.json            the pin: fingerprints of the program version studies run against
│       ├── DISCOVERY_LOG.md         every finding a study sighted, and what was decided about it
│       └── 20260905-stored-energy-basis/    one study record: plan, results, synthesis
├── knowledge/                       what the model's numbers cite
│   ├── sources/                     one directory per registered source: extracted text, page images
│   ├── MANIFEST.jsonl               one row per source, with the hash that identifies it
│   ├── SOURCE_INDEX.md              what each source is for, how to check it, what limits it
│   ├── research/requests/           research requests (REQ-W-01.json); negatives/ records searches that found nothing usable
│   └── holdout/                     quarantined: never read
├── modeling_project/                the modeling rules: requirements, modeling process, study policy
├── work/                            the modeling records
│   ├── BACKLOG.md                   the work item list, changed only through the modeling PM's commands
│   ├── active/                      work items in progress
│   ├── completed/20260906_WI-042_sourced-helium-ash-profile/    one closed work item: spec, design, plan, evidence
│   └── orchestration/
│       ├── GOAL_RUNBOOK.md          how to run a goal, for a person or an agent
│       └── goals/stored-energy-basis/    goal.md, trail.md, learnings.md, evidence/
├── .claude/                         agent prompts: commands (/spec-model, /research-acquire), skills (run-goal, run-study)
├── scripts/                         deterministic tools: source_registry.py, research_seam.py, integrate.py
└── .project/                        the engineering of the harness itself, with its decision records in adr/

The next section follows one goal through these areas, from a discrepancy in a single number, through a research request and a work item, to a re-run study.

4A worked example: the stored-energy goal

To show how the loop in Figure 1 works in practice, we follow one goal from its question to its close. The goal investigated why the model's stored plasma energy was 9 percent above the value printed in the Stellaris design paper. It ran two rounds between 4 and 6 September 2026 (work/orchestration/goals/stored-energy-basis/).

5Research: from a question to a citable source

We want every number in the model to cite a page in a source that anyone can open in the repository. An agent that cannot find a value will often supply a plausible one from memory, and the next session reads that value as a fact. The research workflow therefore separates judgment from writing: agents search for sources and decide which are useful, and one script does all the writing into the knowledge base. We follow round 1's research request from section 4 through each step and the record it leaves.

**Source**: knowledge/concept_research/09-qi-stellarator-hts/iter-01/sources/stellaris-design-details.md
    …
    knowledge/sources/a_general_stellarator_version_of_the_systems_code_process/
    (Lion et al. 2021, Eqs. 8-11: the W definition) and
    knowledge/sources/systems_code_models_for_stellarator_fusion_power_plants_and/
    (Lion 2023, Eqs. 2.10-2.13), both registered at d4059ef1;

Section 6 follows the work item that used this definition to change the model.

6Model updates: the ash-profile work item

Model changes go through work items, each with four stages: a spec, a design, a plan and the implementation, checked by the six-level validation stack. The earlier post describes these stages. Here we show one work item, WI-042, which made round 2's change (work/completed/20260906_WI-042_sourced-helium-ash-profile/).

# The ash shape: the fusion-rate profile normalised to its peak, S(0) = 1
# (Eq. A.5 pointwise, tau* uniform in rho; WI-042 D1).
def S(u: float) -> float:
    return (u ** (2.0 * alpha_n)) * _sigv_dt(T_i0 * (u ** alpha_T)) / sigv_peak

# ...

# The derived electron profile and its diagnostics (D1).
def n_e(u: float) -> float:
    return 2.0 * n_D0 * (u ** alpha_n) + 2.0 * n_He0 * S(u)

Section 7 describes the checks that ran along the way and what they caught.

7Checks: code and fresh reviewers

A major challenge in running an AI system over many sessions is catching its errors early, before later work builds on them. An agent can state a number it never computed (a hallucination), misread its own results, or claim more than its evidence supports. To catch these, we use two general classes of checks: checks written as code, which pass or refuse with no judgment involved, and reviews by an agent in a fresh session that did not do the work. Below we describe the mechanisms in each class, each with an example from the stored-energy goal.

In Figure 9, the steps of round 2 run left to right along the middle, and each check sits above or below the step where it ran. Each card gives the check's question and, where there was one, what it caught on this goal.

What the checks establish. The code checks show that the model is well formed and that the program computes it correctly. The agent reviews check whether the study's design and its conclusions fit the evidence, and on this goal they caught the errors described above.

These checks show that the work is consistent with the model and its sources. They do not show that the model predicts a real plant correctly, which is what the hold-out test in Part 4b is designed to test.

8Where the harness stands

We built the harness so that an AI could carry engineering work across many sessions and have that work build on itself. Our assessment has three parts: the harness is still being refined, it runs our demo's goals almost on its own, and its analysis still needs a domain expert and has leaned on a published design to check against. That last part leaves our biggest open question.

It is still being refined. We continued to develop the harness as we used it and found issues. Many of those fixes were new rules written into the agents' instructions. As much as possible, moving these prompt-rules into deterministic scripts improves scalability.

It runs goals almost on its own. We estimate that about 95 percent of the work on our demo's goals ran without us. The stored-energy goal was an early one, and the owner stepped in at four points: approving the goal, leaving the research decision to the agent, ruling on the fix, and closing the goal. Agents did everything else, including the research request, the work item, the regenerated program and its pin, the 7,712-point study, and six reviews by fresh agents. Later goals needed even less. The four goals that built the integrated ARIES model in Part 4b ran seven rounds between them, and the owner's only input was a written brief at the start and the decision to close at the end.

Its analysis still needs a domain expert, and it has leaned on ground truth. Our colleagues found mistakes in the model's assumptions and in how we read the results. Many of the issues we caught were found by comparing the model with a published design, such as the Stellaris paper.

The open question. Our biggest open question is whether the harness's own checks are enough in a domain with no published design to compare against. The checks in section 7 test the work against the model and its sources, so they do not test whether the model's own assumptions are right. Part 4b describes the hold-out test we designed to probe this: the model was built without the ARIES-CS publications and then compared with them.