Scoring what extraction writes

Who this is for: whoever is about to change the model an instance extracts with, raise the graphiti-core cap, or edit an extraction prompt — and wants a number rather than an eyeballed ingest. You need: a running instance with its key and graph declared — an instance declaring no provider is refused by name, below; the seeded run bills that key (about a dollar at gpt-4.1-mini for the full corpus), the local check bills nothing. Not here: what a stale corpus is (failure-and-staleness.md), or the five-arm benchmark against flat markdown, which is parked.

Every read the engine serves answers from what extraction wrote, and until this command nothing measured that. The suite's LLM answers every prompt with nothing, so what a real model puts in the graph — which statements it finds, how exactly it quotes them, how many concepts it mints — was known only from looking. revetly eval is the measurement: a corpus whose right answers are known before any model reads it, sent through this instance's model, scored.

The corpus

revetly eval run generates about fifty-five artifacts from a seed — a fictional product, Harbourmaster, berth booking for marinas. Three epics, five features and fourteen stories with a declared parent hierarchy (two stories hang straight under an epic, two retire earlier ones with supersedes); fourteen conversation notes, each stating zero to three planted decisions, assumptions and constraints as exact sentences — ten of each type across the whole corpus, which note carries how many and which sentence it carries drawn from the seed, so a prompt cannot be tuned to one layout or one assignment; four decision notes; six delivery records, two of them with a departed verdict naming its reason and who decided it. Every story cites notes; every note mentions stories. Two kinds of decoy ride along: a ticket id no source in the corpus writes, and eight sentences that read like reasoning and state none — an open question, an option being weighed — placed one for every three planted statements, so precision is tested in proportion to recall.

Beside those, nineteen planted questions of the kind a product owner, an analyst or a tester asks (below, under Retrieval), and what they need: eight more decision notes — four answering a "why", two where a later decision supersedes an earlier one, and a live and a withdrawn answer to the same question — and a small code repository under code/ holding four commits, read by the commits connector. Each commit shares the words of one "why" question and answers none of them: it says what changed, and the decision note says why. None of this names a backlog id and all of it is dated after the corpus's other artifacts, so every other gold, and every read slot, is the one it was. The statement score does not move either: a decision note is never read for statements — it is the record of its decision already — so these notes can add no spurious statement and plant none. That brings the corpus to fifty-four artifacts and four commits. --size small plants none of it.

The gold is derived from the seed's structured data, never from a model reading the prose: every declared edge, every planted statement with its passage, every branch, departure, window and supersession. The same seed is the same files, byte for byte, so a report from last month and one from today scored the same corpus.

The corpus is written into a scratch instance of its own, with one git commit per artifact carrying its planted date — the file connectors read source time from git — and ingested into an isolated namespace of the graph this instance declares, on the key this instance declares. The scratch instance carries this instance's own [llm] and [embedding] declarations, copied verbatim, so the provider that scores it, the endpoint the bodies reach and the key billed are this instance's — the report header and the JSON name all three — and an instance whose declaration does not resolve is refused rather than scored on the engine default. The namespace is cleared afterwards unless you keep it. Nothing about your own corpus is read or written.

A corpus that withdraws a note is read twice. The first pass ingests all of it; then the withdrawn note's file is deleted in one dated commit, the sources are read again, and an ordinary second pass tombstones it. The second pass sends no episode, since nothing else changed, and anything it did cost is added to the report's cost line. The report names what it withdrew under pass.withdrawal.

Running it

revetly eval run                 # price it, ask, send, score, report
revetly eval run --yes           # no question
revetly eval run --size small    # a dozen artifacts, for a quick look
revetly eval run --keep          # leave the scratch instance and the namespace for inspection
revetly eval run --seeds 7,11,13,17,19   # several seeds in one run, quoted as one set
revetly eval run --seeds 7,11,13,17,19 --tiers statements   # the same seeds under another tier
revetly eval compare <before> <after>    # two saved reports, on the seeds they share
revetly eval rescore <report>    # the retrieval questions again, over the graphs a --keep run left
revetly eval list                # the reports kept here, newest first
revetly eval clear <namespace>   # drop a kept namespace (only ones this command created)

One seed by default; more on demand. --seed 7 is the default and stays the default: every seed is its own billed pass, so five seeds cost five times one, and they are opt-in rather than the price of asking. --seeds and --seed are mutually exclusive — a set is one seed or several, never both. --seeds 7,11,13,17,19 prepares every seed first, prints the price for the whole set before anything is sent, and then sends and scores each in turn. Every seed gets its own report as a single run does, and the set gets one report of its own: kind: "seed-set", named for its seeds (for example ...-seeds-7-11-13-17-19-<stamp>.json), holding each per-seed report under runs, the spread between them, and the summed cost. That set report is what latest.json holds. set_lines renders it: each metric's mean, min–max and per-seed values, then the planted sentences missed on every seed apart from those missed on some — each with the note it sat in — then, per type, how large a recall change that seed spread can resolve.

revetly eval compare <before> <after> takes each side as a report path or as a stamp revetly eval list printed. A single-seed report is treated as a set of one. It compares only the seeds both sides hold, and refuses by name when they share none or when a shared seed was scored against a different corpus: a report written before this story carries no gold digest, and is refused rather than compared. Per metric it prints the difference per seed, the mean difference, the baseline's own spread between seeds, and a sentence saying whether the mean difference is larger than that spread.

Comparing extraction tiers. --tiers scores the same seeds under another tier (revetly manual show extraction-policy). One tier — entities, statements or index — moves every seeded kind the engine's defaults extract (backlog items and documents); kind=tier pairs, comma separated, move exactly the kinds named (--tiers doc=statements,pbi=index). The scratch instance declares the override as [extraction.tiers], the report carries it under tiers, and its file name says -tiers-<kind>-<tier>, so a paired comparison is the default run and the override run on the same seeds, read with revetly eval compare. Nothing else about the run changes.

Measuring a read change without paying for extraction again. A change to the reads alone — a ranking dial, the binding sort, a trimmed shelf — needs the retrieval score and nothing else, and a fresh paid run would fold extraction's noise between seeds into it. Run once with --keep (revetly eval run --seeds 7,11,13,17,19 --keep --yes): every seed's namespace stays in the graph and the set report names them under groups. Then revetly eval rescore <report>, with the set report's path or stamp, asks the planted questions again of those same graphs with the read code running now, inside each run's kept scratch instance, because its ingest ledger is one of the search rank's signals. The corpus is regenerated from each run's seed and must match its gold digest, or the rescore is refused rather than asking the wrong questions; the questions' query vectors are the ones the kept run embedded (kept under .revetly/eval/query-vectors.json), so the rescore makes no provider call. It is refused by name when a kept namespace or scratch instance is gone. The report has kind: "rescore" ("rescore-set" for a set), is named with -rescore, holds scores.retrieval alone and leaves latest.json alone; revetly eval compare <run> <rescore> compares it with the run, or with another rescore, on the retrieval figures both carry. revetly eval clear drops each kept namespace when you are done, and the scratch folders under .revetly/eval/ are yours to delete.

revetly eval run is refused by name on a lite instance — one declaring [llm] provider = "none" and [embedding] provider = "none" — because it scores what a model extracts and a lite instance declares none. A row of zeros would read as a model that did badly. revetly eval check runs there and says, above its misses, that nothing it scored was written by a model, and the report it writes carries lite.

The corpus gate runs first, and the namespace half is what binds it. Before the pass is priced and before anything is billed, revetly eval run refuses an instance whose declared namespace has moved: the eval writes an isolated namespace of its own and reads none of your corpus, so the corpus revision is not a fact about the data it scores, but an instance whose corpus this engine has refused is not one to run a paid measurement on. The revision half is not needed because the pass prepares the database itself: revision 6's migration builds the keyword leg's fulltext index, and an ordinary pass calls durability.ensure_indices before its first provider call — naming that index among its markers — so a database one migration behind is built rather than scored half-blind. The refusal, and everything the gate covers, is in failure-and-staleness.md.

The pass is priced with the same instrument revetly ingest --dry-run uses, and nothing is sent until you say so or pass --yes. Progress is one line per artifact. The report lands under this instance's .revetly/eval/, named for the engine version, the model and the graphiti-core version it scored, with latest.json beside it; context_health and revetly status name the last one and say whether it scored the combination that is running now. Exit status is non-zero when a structural score failed (below).

What the scores mean

Two kinds of score, and the distinction is the whole point.

Structural scores are properties of the engine's own deterministic writes. No model was consulted for them, so they must be perfect on every model, and a miss is a defect in the engine rather than a habit of the model:

• link recall over every declared edge — parent, cites, supersedes, delivers, explicit mention — must be 1.0. Precision is reported beside it; an extra deterministic edge is a gold the seed missed, and is named.
• phantom ids must be none: the decoy id no source wrote, recorded on a node as if it were an artifact. Only the deterministic pass writes ids, so one of these is the engine's.
• read slots — the children beneath each epic, the departures with their reasons and deciders, what changed in a window on the source clock, the decision notes and statements that bind a story, and current truth after a supersession — must every one be exact, through the same reads a client calls. The binding slot expects the planted statements the model actually wrote: whether it found a sentence is the extraction score's business, whether the read reaches every sentence that landed is this slot's.

Extraction scores are what the model did with the prose. They are reported, not judged; the report is what lets two of them be compared:

• statements: precision, recall and F1 against the planted decisions, assumptions and constraints, with a per-type breakdown. A planted statement is found when the model stated it in the right note, on the right shelf, and either quoted the planted sentence or named the statement in its own words. Right sentence on the wrong shelf is counted as neither found nor spurious and listed as mistyped — a wrong type on the reasoning behind a product is the failure the prompt is written to avoid. The same rule is applied in the second phase as in the first — same note, an exact passage, a name inside the passage, or a close-enough name — with only the type check dropped, so a sentence found under another type is mistyped however it was found, not only when the passage was exact. Each type then reports recall twice: the right-type recall, and recall regardless of type, which counts a mistyped sentence as found. The gap between them is what the model got wrong about the type rather than about the reading. Where the gold itself says a sentence honestly reads two ways, it accepts either type — the berth sentence labelled Constraint that every run reads as a Decision is a right type, not a mistype. The primary type is still what the per-type counts are keyed on, and precision is computed as before: a mistyped sentence is not a hit and is not listed as spurious, but it is still one of the statements extracted, so it lowers precision the way any wrong answer does. Across several seeds the report gives each metric a mean and a min–max, and names the planted sentences missed on every seed apart from those missed on some. That spread is the error bar: a type whose recall moves by less than the spread between seeds cannot be read as an effect, and the report says per type how large a change the spread can resolve. A statement the pass withdrew after writing it (a name that is not a proposition, or an assumption or constraint with no real qualifier; see docs/architecture.md §5.7) has no shelf, so it is scored as missing, and so is one the pass retired. The report lists both under statements.lost (unshelved and retired, each with its note, name and qualifier), so a miss the pipeline caused is not read as a miss the model made.
• decoys served: the sentences that read like reasoning and state none — an open question, an option being weighed — that the model reported as a statement even so. The prompt says to report nothing for these; a model that does is the finding, and it is listed sentence by sentence.
• passage exact: of the statements found, how many quoted the planted sentence exactly. The engine drops a passage it cannot find verbatim in the body, so this is the share a reader could check without opening the file.
• noise: concept entities minted per artifact, inferred edges, how many concept names are shaped like a sha, a path, a version or a code identifier (the same table [extraction] concepts = "declared" withdraws by, so this is zero once the pass has run and the model's habit under open), whether any of the planted noise decoys still stands as an entity, and how many of those name neither artifact they join — the same census revetly status keeps between passes.
• cost: tokens in and out, dollars at the rate table in force, seconds, and each per artifact, taken from the pass's own reckoning — the same row revetly status shows for every pass.

Retrieval scores say whether the reads put the answer first. They are reported, not judged: a ranking is what later work changes, and this is the number it is held against. Every question is asked in-process of the same functions the MCP tools call, at the tools' own limits (ten for search, five for the drafting bundle), and the reads make no provider call. Only fields that hold still when a read's payload is reshaped are scored: artifact ids and their order, a passage's artifact id and whether its excerpt carries a planted stretch of text, a fact's endpoints, a binding statement's id, passage and superseded_by, and the drafting bundle's backlog ids.

The nineteen questions come in five kinds. Eight topic questions ask about planted statements in plain words ("Do we keep skippers' card numbers anywhere?"); the answer is the note the seed planted the statement in, so it moves with the seed, and each also asks get_scope(question='binding') for a plain topic ("card numbers"). Four why questions are answered by a decision note and distracted by a commit carrying the same words. One superseded question is answered by the later of two decision notes, and asks binding for its topic. One withdrawn question has a live answer beside a note withdrawn before the reads run. Five authoring questions are what somebody about to draft a story would type, each answered by the story it would duplicate; one of them names a story a later one superseded, and is answered by the successor.

• MRR (retrieval_mrr): over the fourteen search questions, one over the rank of the first answering artifact on artifacts, or zero where none is served; the mean. 1.0 is every answer first.
• P@3 (retrieval_p_at_3): answering artifacts among the first three, over as many as three slots could hold (one for a question with one answer), so a perfect answer scores 1.0.
• excerpt hit (retrieval_excerpt_hit): the share of search questions where a passage from an answering artifact carries the planted stretch of text in its excerpt, case and spacing aside. A passage that only echoes the title is a miss.
• facts precision (retrieval_facts_precision): of every fact served on the search questions, the share with an endpoint among that question's answering artifacts. The count served is beside it.
• authoring top-3 (retrieval_authoring_top3): the share of drafting questions whose story is among the bundle's first three backlog items; and binding non-empty (retrieval_authoring_binding_nonempty), the share whose bundle carried any binding statement at all.
• binding precision (retrieval_binding_precision): of the first ten statements binding served for a topic, the share that belong to it — a statement quoting the planted sentence, or a decision note the topic names. Counted over all nine topics together; a topic answered with nothing is counted apart, as binding_empty.
• retired above successor (retrieval_superseded_outranks): how many times a superseded artifact ranked above the one that replaced it, on search or in the drafting bundle's backlog, including where the successor was not served at all.
• retired served unmarked (retrieval_superseded_unmarked): binding rows, on a topic or inside a drafting bundle, naming a superseded artifact without superseded_by.
• withdrawn served (retrieval_withdrawn_served): every place the tombstoned note appeared on a default read — an artifact row, a passage, a fact endpoint, a backlog row, a binding row. It should be zero.
• commit above the decision (retrieval_distractor_outranks): how many of the four "why" questions ranked the distracting commit above the decision note that answers it.
• lite parity (retrieval_parity): the share of search questions whose lite answer's top three artifacts are all on the full answer's served shelf. "Lite" is the same search with the semantic recipe held back, which is what a lite instance serves: the engine's own keyword leg over the same graph. It is scored inside the one run rather than against a second, lite instance, which would be a second corpus with no statements or concepts in it.
• bytes per read (retrieval_bytes_<read>_median, ..._mean): the byte length of each payload as compact JSON, per read kind (search, lite, authoring, binding). A read that answers as well in fewer bytes is the better context; the median leads because one oversized answer should not stand for a read that is usually small.

Each question's rank, what was served first, and each miss are kept in the report under scores.retrieval.rows, and a read that raised is named under errors and scored as a miss. Every figure above is in the summary that context_health and revetly status read, and in a seed set's spread with its mean, min–max and per-seed values.

A number here is a number for this corpus. It is a regression eval for the engine, not a benchmark of the product: the seed's prose is short, its statements are planted to be findable, and a model that scores well on it has shown it can read the shape, not that it reads your notes. What it settles is the comparison — this model against that one, this graphiti-core against the last, this prompt against yesterday's — on ground that did not move.

The suite runs it too

The graph lane (make graph-check) runs this same eval end to end against a real Neo4j with a deterministic in-process model whose answers are scripted from the gold. A perfect model has to earn link recall 1.0, statement F1 1.0 with every passage exact, zero violations and every slot exact — so the scorer is proven before it is pointed at a paid run, and a change to any read that breaks a slot fails in CI before it reaches a report. One lane run is the full corpus, withdrawal pass included: every planted question has to be asked without a read raising, the withdrawn note must never be served, and a retired decision served as binding must say what replaced it. The ranking figures themselves are only required to be figures; a fake embedding ranks nothing worth holding a number to.

Your own corpus

revetly eval check scores the corpus this instance actually has, against questions you write, at no LLM cost (a search question embeds its query, as the tool does when a client asks it). Put them in eval/golden.json under the instance root (or name a file with --golden):

[
  {"read": "branch",  "artifact_id": "EPIC-000", "expect": {"children": ["FEAT-000", "STORY-000"]}},
  {"read": "drift",   "artifact_id": "EPIC-000", "expect": {"departed": ["STORY-000"]}},
  {"read": "binding", "artifact_id": "STORY-000", "expect": {"contains": ["PBI-000"]}},
  {"read": "artifact", "artifact_id": "STORY-000", "expect": {"status": "done", "parent": "FEAT-000",
                                                              "links": [["cites", "PBI-000"]]}}
]

children is a root's direct children and descendants its whole branch; departed and clean are sets over the scope; contains is what a binding read must not miss — the ids of the decision notes, or of the notes a planted statement was stated in; the artifact slots are equality. The report lands beside the seeded ones with kind: local. Write these the day you notice a read answering wrongly about your own backlog: the question that caught it is the question that keeps catching it.

The same file can hold ranking questions — whether a read puts the answer first on your own corpus, which is the check a ranking change cannot be fitted to the seed's gold against:

[
  {"read": "search",    "query": "why is the page served on loopback", "expect": {"relevant": ["DEC-000"]}},
  {"read": "authoring", "intent": "let a teammate join", "expect": {"backlog": ["STORY-000"], "binding": ["DEC-000"]}},
  {"read": "binding",   "topic": "config page", "expect": {"top10": ["DEC-000", "CONV-000"]}}
]

They are asked at the tools' own default limits and scored with the seeded retrieval arithmetic: for search, the reciprocal rank of the first relevant id on artifacts, hit@3 and P@3 (relevant ids in the top three over min(3, relevant)); for authoring, whether the bundle's top three backlog holds an expected story, and binding recall — the share of the expected notes with a statement on the bundle's binding shelf; for binding, precision and recall of the expected notes among the first ten statements served, a statement being identified by the artifact it is or the note it was stated in. contains keeps working beside top10. The summary is MRR, hit@3, P@3, authoring backlog hit@3 and binding recall, binding precision and recall, the count of empty binding shelves and the median bytes per read, under retrieval in the report, with one row per question. --json FILE also writes the whole report to FILE, so two checks can be diffed. A slot miss still sets a non-zero exit; a ranking figure never does. A file of ranking questions alone prints no slot questions; N question(s) scored by ranking in place of the slot count, and its summary in revetly eval list and context_health carries gold_questions, the number of ranking questions scored, with no questions key, so it never reads as a run that scored 0/0.

revetly eval lint --golden FILE checks a question file before it is scored. An id with whitespace or a comma in it (joined) or an expected id the graph does not hold (missing) is an error and exits non-zero; an expected id a --spent file already expects (spent, so a new blind set does not reuse one a set was tuned on) or two questions of the same read asking the same text (duplicate) is a warning that never fails. Each line names the question, the read, the field and the id, and the last line counts them. --json FILE writes the report. It reads the file and asks the graph about each distinct id once.

revetly eval compare OLD.json NEW.json reads two such files question by question: per metric the old and new figures, then the questions that lost (losses first, they matter most) and the ones that gained, the median bytes per read, and any question present in only one report. --primary read.metric (for example search.rr, or stale.stale_unflagged, where a question going from true to false is the gain; retrieval.stale_rate and retrieval.stale_unflagged_rate name the stale questions' two flags by their summary names) makes it the acceptance instrument for a quality pass: the last line is KEEP and the exit is zero only when that metric gained at least one question and no metric lost one, otherwise REVERT and a non-zero exit. --json FILE writes the comparison. It reads the two files and nothing else — no config, no graph, no project.

A score that explains itself

Every expected id a ranking question got no credit for is given one reason, counted per read on the misses line and listed id by id under retrieval.missed. An id gets credit in the top three for search and authoring backlog, anywhere on the shelf for authoring binding, and in the first ten statements for binding top10.

• not_captured: no artifact by that id is in the graph.
• not_gathered: the artifact is there, but the read never gathered it.
• ranked_out: served, but below the scored window (its rank is given).
• budget_cut: gathered, then cut before serving. cut says whether the top-k or statement cap cut it, or the byte budget.

Telling the last two apart takes what the read gathered before its cuts. eval check collects that on a side channel only it opens, so the payload a client gets is unchanged. Each read's bytes are also split by top-level shelf (shelf_bytes, printed as the median bytes per shelf), and --payloads DIR writes each question's served payload, with what it gathered, to its own file (001-search-<slug>.json). You never need a separate dump script to see why a question missed.

Five more shapes:

[
  {"read": "stale",     "query": "first-run wizard", "expect": {"current": ["DEC-001"], "stale": ["DEC-000"]}},
  {"read": "binding",   "artifact_id": "STORY-000", "expect": {"top10": ["DEC-000"], "newer": ["DEC-002"]}},
  {"read": "authoring", "intent": "let a teammate join", "exclude": ["STORY-000"], "expect": {"backlog": ["PBI-000"]}},
  {"read": "search",    "query": "what is a concept", "tags": ["concept"], "expect": {"relevant": ["DOC-000"]}},
  {"read": "path",      "path": "src/app/billing.py", "expect": {"relevant": ["STORY-000", "DEC-000"]}}
]

A stale question asks search_context the way a search question does. It counts as a stale hit when a retired id ranks at or above the best current id on artifacts, or is served there while no current id is. stale_rate is hits over questions, beside the current ids' MRR. stale_unflagged_rate counts only the hits whose stale row carries no revised_by mark — served above what replaced it with nothing on the row saying it was revised; each row carries the per-question stale_unflagged, and the stale line prints it as unflagged. newer lists decisions recorded about a binding scope after the scope was written: newer_recall is the share of them in the first ten statements, newer_served the share anywhere on the shelf. exclude removes ids from an authoring question's backlog and binding, both expected and served, so a story whose gold came from its own citations cannot score by finding itself. tags repeat the ranking figures per tag, under by_tag and one tag line each. A path question asks get_scope(question='binding', path=...) which stories and decisions are behind a file, and is scored as a search is, on behind: reciprocal rank, hit@3 and P@3, printed on a path line.

Questions harvested from your stories

revetly eval harvest --out FILE [--limit N] writes authoring questions built from your own stories, in the same file shape. Each backlog item whose file carries cited_sources becomes one question:

{"read": "authoring", "intent": "<its title, plus its first sentence when short>",
 "expect": {"backlog": ["PBI-000"], "binding": ["DEC-000"]},
 "exclude": ["STORY-000"], "tags": ["harvested"], "source": "STORY-000"}

backlog holds the cited backlog items and binding the cited decision notes. A citation counts only when the cited record is dated strictly before the story was created, so the question asks for what existed when the story was drafted. A story's creation date is the earlier of its source creation time (git, for a file) and its first review_log entry. A cited record's date is its source creation time (a note's captured_at, otherwise git), or its first review_log entry when it has no source time. A citation with no date is dropped, and so is a question left with no expectation. exclude names the story itself, which is in the graph now but did not exist when it was drafted. Questions come out in id order, numbers compared as numbers (9 before 10), and --limit keeps the first N.

The command reads the files your own record stores hold (the backlog, the conversation notes, the delivery records) the way revetly check reads them: no repository is fetched, no connection floor is written, no graph is opened and nothing is sent. The output says how many citations it did not use, and why: unknown, undated, not dated before the story, or neither a backlog item nor a decision note.

A harvested set is circular. The drafter cited what the drafting read served it, so a high score here proves little. Use it to find misses, report it separately from questions you wrote, and do not use it to decide a change.

Capture health

revetly eval health [--json FILE] compares what your graph holds with what your files say, over this instance's group. It reads the graph over a plain connection: no provider key, no provider call, no write. It reports four counts:

• coverage: how many source-of-record artifacts the connectors list have an anchor node in the graph, and which ones do not;
• link fidelity: every link a file declares (cited_sources, parent, edges, supersedes, same_as, and a note's related), resolved by the linker's own rules, checked against the live declared edges in the graph, counted per relation. A citation or supersession naming nothing the sources list is counted unresolved, because the linker drops it by rule. A note's related produces an edge only when the id is a backlog item. Any other related id is counted not minted;
• statements: live statements, echo-withdrawn statements (a statement node that still names its note, is not retired, and wears none of the Decision, Assumption or Constraint labels), and which notes read for statements have no live one left;
• supersession: how many decision notes say in prose that they supersede, replace, reverse, overturn or override a record they name by id, and how many of those claims have a supersedes edge. The rule is conservative. The verb and the id must be in the same sentence. A sentence containing not, never, no, nor, without or n't is skipped. The Alternatives and Related sections are skipped too, because a rejected alternative often reads "replace X". The id must belong to a record other than the note itself.

The report is saved under .revetly/eval/health/. revetly status prints one line from the latest report, with the date it was measured. status never opens the graph itself, so it runs as fast as before. The line appears once eval health has run, and it goes out of date until you run eval health again.

Raising a cap, changing a model

The procedure is no longer "run the suite, then ingest a real corpus":

1. make check — the hermetic suite and the lint.
2. make graph-check — the claims against a real database, free.
3. revetly eval run on an instance declaring the new model or the new library — and compare the report with revetly eval list's previous one: the structural scores must still be perfect, and the extraction scores and the cost are what the change is judged by.

Two runs of the same model on the same seed will not score identically — extraction is not deterministic — so read a difference of a few points as noise and a difference in a kind of miss (a shelf that empties, a passage rate that halves) as the finding.