Who this is for: whoever pays the provider bill for an instance, and
whoever finds search_context answering with paths and version strings.
You need: an instance with a revetly.toml, and revetly ingest --dry-run
to see what a pass would cost before and after. Not here: how extraction
is scored on a seeded corpus (scoring-extraction.md),
and every option's type and default
(engine-config-surface.md).
Every artifact this engine reads lands in CONTEXT twice over: once as itself — its body on an episode, its node recorded with what the source states, its declared and explicit links written by deterministic passes that cost nothing — and once as whatever a model inferred from it: entities, edges between them, a summary. The first half is what the reads this product is positioned on consult. The second half is what the provider bills for, on every artifact, forever.
This page is about three declarations that decide how much of the second half
an instance pays for, what it gets, and what it takes back. All three live in
[extraction] in revetly.toml, all three are per project, and all three are
the instance's own: nothing here ships a vocabulary.
[extraction.tiers]
commit = "index" # the engine default
delivery-record = "index" # the engine default
delivery-note = "index" # the engine default
revision = "index" # the engine default
comment = "index" # an instance's own choice for a kind it reads
doc = "statements" # an instance's own choice: statements, no entities
Every kind is on one of three tiers. entities is what every kind was
before this existed: the body is extracted, and the model names entities, draws
edges between them, writes a summary, and — for prose documents — reads the
three statement shelves. index is the engine writing the episode itself
and no model reading it. statements is the engine writing the episode
exactly as index does and then reading it once for the three statement
shelves — one provider call per artifact, the only call that produces
decisions, assumptions and constraints — with no Graphiti extraction at all:
no concept nodes, no inferred edges, no summary. A kind whose artifacts carry
no statement shelves (a backlog item, a decision note, a commit) costs exactly
what index costs on this tier. revetly doctor prints the table in force
under extraction tiers, naming every kind declared off the engine default.
Where the three statements land. A prose document the model read may state
a decision, an assumption or a constraint. Those are not entities: the engine
writes each as a :Statement node carrying its one type, joined to the
artifact that stated it by a STATED_IN edge, outside Graphiti's :Entity
namespace — so dedup, contradiction and the two search legs never see one, and
no model can fuse a concept into a statement. A statement the engine retired
because a revised note stopped making it stays on the graph, marked, and is
served only when a read asks for withdrawn work. A lite instance reads for none
of them (there is no model to ask), so on a lite pass no statement is written
and none is served; a statement a corpus already holds is left exactly as it
stands.
What the engine owns, and what Graphiti owns. The graph holds two layers.
The engine-owned core is written by the engine's own Cypher and
nothing else: statements (:Statement, joined by STATED_IN) and declared
links — explicit references, title mentions, supersedes, cites, same-as and the
relations a source declares — as one DECLARED relationship carrying
relation_type, method and provenance, with no embedding. Graphiti's
dedup, contradiction and search legs work over :Entity nodes and RELATES_TO
edges only, so they cannot merge a statement, strip a declared link's tags or
invalidate one: that is structural, not a check the engine repeats. The
inferred layer is what Graphiti's own extraction writes — concept nodes and
RELATES_TO edges — and a model's opinion stays confined to it. The artifact
anchors sit between the two: they stay :Entity, so an extracted mention still
resolves onto the artifact it names, and a pass re-asserts their
source-of-record properties — ids, paths, tombstoned, deleted_at,
deleted_clock and a withdrawn unsourced_id — from the engine's own ledger,
never putting a tombstone on an artifact that is live at its source. It does
so for every anchor an episode touched and every artifact whose facts moved,
and for everything it read once per [watch] full_interval. The heal
that cleared a Graphiti invalidation off a declared link and
repair_shared_edges remain only as cleanup for rows a pre-revision-7 corpus
carried into the migration.
What an indexed artifact keeps:
search_context under passages with an excerpt around the words
that matched, so a commit message is found by what it says;implements on a commit, delivers on a
delivery record, a parent, a supersession) and every explicit tracker-id
mention, citation and identity claim — written by the same passes that write
them for everything else;What it loses: the entities and edges a model would have inferred from the text, and its summary paraphrase. For a commit, a delivery record or a revision, that is ranking noise — every read that answers what shipped against this, what departed, what changed and what binds this is deterministic and consults none of it, and the statement shelves were never read from those kinds. On the instance this engine develops itself against, those three kinds were 63% of provider calls at 31 calls per 1.2 KB commit message. Indexed, a busy day of eighty commits lands in seconds rather than forty minutes, and a repository with a five-year history is read for the price of its embeddings.
The table merges over the engine's defaults: declaring commit = "entities"
says nothing about delivery records. A kind you never declare is entities.
A value outside the three is refused when the file is loaded, naming all
three.
Pricing. revetly ingest --dry-run and the quote printed before a pass
price the tiers apart — indexed episodes at no provider token, extracted
ones against this instance's measured history, and one statements read per
statements-tier artifact that carries shelves, priced from that one call's
own shape (the prompt it sends plus the note, at about four bytes a token), as
revetly repair statements prices a re-read — and the reckoning after the pass
says how many of its episodes the engine wrote at no provider token. The measured history an extraction is priced
from leaves out every row the engine wrote itself (index or statements), so
cheap engine-written episodes never pull an extraction's quote down.
revetly migrate follows the same declaration. A migration that names
documents to be read again for a newer shelf asks each one's tier before it
prices or marks it: a document on index is read by no model either way, so
naming it for a re-read would quote a statement read the pass that follows
would never make. The tier it asks is the one declared now, not whichever one
stood when the document was first sent — the migration and the pass it leaves
to be run both act under today's policy.
And the same step stamps the tier it decided under. When today's policy puts
documents on index, every live document row written before tiers existed —
one whose ledger states no tier — is marked index by the migration itself:
no episode sent, no provider called. The row then reads as what it is, a
document no model has read under this policy, and a provider declared later
promotes it exactly as the rule below says. Without that stamp the row would
stay NULL, the promotion rule would never fire on it, and a document no
model ever read would be the one document a provider could never reach.
Changing a tier. The tiers read a body to three depths: index, then
statements, then entities. Move a kind to a deeper tier — index to
statements or entities, statements to entities — and the next pass
sends exactly the artifacts not yet read that deeply — unchanged at the source,
promoted by the policy — and the dry run lists them under that name. An
index artifact promoted to statements keeps the episode the engine already
wrote for its bytes and is only read for statements. Move a kind to a shallower
tier and nothing is sent and nothing is retired: what was
extracted stays, the dry run says so, and revetly forget <id> is the gesture
that takes an extraction back, priced and confirmed. The one exception is the
statement shelves: a note no model reads for statements is not served
statements, so the pass withdraws them. That happens when its kind moves to
index, and for every note while [extraction] statements = false. The
ledger marks each note it withdraws them from (statements_owed, with the
reason), and the first pass whose policy offers the shelves again reads those
notes back. Every tier move and every switch change is reversible this way: a
note left with fewer shelves than its new tier offers is read again on the
next pass, and the dry run prices it.
statements row moved to index keeps nothing a model read, so the
ledger restates it index. Moving it back promotes it, and the engine's
own episode is reused.entities row moved to index keeps its entities and loses its
statements. Moving it back to entities, or on to statements, reads it
for statements only: one call, no new episode, no extraction re-bought.
The dry run lists these as "unchanged artifact(s) read again for
statements only" and gives the reason.[extraction] statements back on reads every unchanged note it
was off for, for statements only, on either tier.entities to statements re-reads nothing: the note keeps the
statements it has. Moving it back re-reads nothing either. A note first
written while the kind was on statements has no entities, so it is
promoted.A policy that has not changed owes nothing, and a read settles what it owed, so a second pass reads nothing. Rows written before the ledger kept this mark owe nothing until a pass under a policy that withholds shelves marks them, so run one pass after upgrading before you move a tier back.
A ledger written before
tiers existed reads every row as extracted, because every pass before this one
extracted everything — except the documents the revision-4 migration judged
index, which it marks so the promotion above reaches them. A row so marked is
not a row extracted; it is a row the migration priced at no model read, and
leaving it NULL would read as extracted and re-read on no declaration at all.
A lite instance. Declaring [llm] provider = "none" and [embedding] provider = "none" together puts every kind on index, whatever this table
says: there is no model to read the entities tier with, so this table is not
a choice a lite instance gets to make for itself. revetly ingest --dry-run
prices the pass at zero calls, zero tokens and zero cost, and states the
declaration that made it so in the same sentence that would otherwise price
the entities tier. Naming a real provider on both sinks later is what
promotes a kind this table already calls entities: the next pass sends
exactly the artifacts no model has yet read, priced and listed the way a tier
change always is — revetly manual show providers has the mechanics of the
declaration itself.
What "searchable by keyword" means. Until this story, the fulltext query
graphiti-core built for an indexed body ran the query text through an escape
table that mangled more than the Lucene operators it was meant to guard
against — a search for STORY-000 reached the index as S\T\O\RY\-000 —
and it capped a query at 128 words, past which it answered nothing rather
than truncating. Neither that query nor the index it ran against carried any
notion of stemming, so a body that said ingest was invisible to a search
for ingestion. An indexed artifact's "searchable by keyword" promise held
only for a query that happened to survive its own escaping intact.
The engine now builds this query itself, and writes its own index for it. An
id the engine recognises (STORY-000, story 000, story000) gets its own
boosted, exact clause; a quoted phrase stays a phrase; the remaining terms
tolerate one typo at five to eight characters and two at nine and over, none
shorter; a match in the title outweighs the same word in the body; a spelling
declared under [extraction.aliases] expands to its canonical term, the
typed spelling kept beside it; everything else is escaped, so a leading
wildcard is impossible by construction. The title, the body and every id
spelling land on a search_text property the engine writes on every episode
and its anchor node, indexed under Neo4j's own english analyzer — the
stemming that makes ingestion find a body that only ever says ingest.
What a typo does not reach: the tolerance is measured against the stem the
index stores, not against the word the source wrote. A misspelling of a word
the analyzer keeps whole is found; a misspelling of one it stems is not, because
mooring is stored as moor and moorong is three letters away from that. It
is exactness that is lost, never recall — the correctly spelled word finds the
stem through the ordinary clause either way.
What it does not do: it never paraphrases a word you did not type unless that
word is a declared alias, it never expands a synonym it was not told about,
and none of it is semantic — no embedding, no model call. A hit this leg
produces carries method: "keyword (english stemming)", and nothing about it
is ever called semantic, vector or inferred.
A corpus written before this story keeps the old, escaped-and-unstemmed
behaviour until revetly migrate has run its search-text-backfill step —
failure-and-staleness.md has the whole migration
story. context_health.corpus_revision names the step as pending until then,
and the step itself reads nothing from a provider and costs nothing.
[extraction]
concepts = "declared" # the default; "open" is what every pass did before
[extraction.aliases]
Acme = ["acme", "acme_platform", "Acme Platform"]
Left to itself, an extractor names what it likes. Measured on this engine's
own corpus, that meant a graph whose five best-connected nodes were five
spellings of the product's name, and whose concept shelf held
cli._history_on_offer, Commit 8cfcc0f, v0.1.2 and a git identity beside
the customers, vendors and rules a product team would actually name. Two
thirds of the nodes stood for no artifact; a good share of those stood for
nothing at all.
Under declared, the default, every extracted episode carries instructions
rendered from this project: extract only what a product team would name in
conversation about the product — people, teams, roles, customers, products and
product areas, features, systems, vendors, business rules, domain terms — and
never code identifiers, file paths, commit hashes, version strings, git
identities, shell commands, flags, environment variables or URLs. Tracker ids
of this project name artifacts already in the graph: a mention of another
artifact's id is a reference, never an entity. And the declared aliases are one
entity, always named by the canonical spelling.
The instructions govern what is extracted from now on. Two deterministic passes at the tail of every ingest govern what the corpus already holds, at no provider cost:
Entity shelf carrying merged_into. It runs whenever aliases
are declared, under open too, because a name is the instance's statement
rather than a restriction on the model.name <email>
identity, an email, a path, a file name, a dotted or snake-case identifier,
a flag or an environment variable — leaves the Entity shelf with the shape
recorded as the reason. The node and its edges stay, for audit and for the
one SET that would put it back. It never touches a node an artifact stands
on, a statement, a name declared as an alias, or a node whose id was
withdrawn under the unsourced-id rule. It runs only under declared.Leaving the Entity shelf is what makes a node invisible: every read in this
engine and every search Graphiti runs matches :Entity, on both ends of an
edge. Nothing is erased.
Under open no instructions are passed and the noise pass does not run.
Use it where a project's concepts genuinely look like code — or to measure what
declared costs you, which is the next section.
A lite instance names neither. With no model declared, this whole
declaration is moot: nothing is ever extracted, so neither declared nor
open runs, and search_context's concept shelf reads absent with the reason
lite rather than empty — a fact about what the corpus lacks, not about what
it holds nothing of. That holds on a corpus a model had already read before
none was declared, too: neither the alias fold nor the noise pass is
registered on a lite pass, so what an extractor left on the shelf stays exactly
as it was until a provider is named again — a lite pass retires nothing.
[[extraction.retractions.edges]]
source = "Acme"
relation = "COMPETES_WITH"
target = "the marina office"
reason = "no source ever joined these two; the model read one paragraph twice"
[[extraction.retractions.concepts]]
name = "berth fabric"
reason = "no such concept in this product"
The two passes above act on a rule — what a concept is called, what shape a
name has. Neither reaches the single edge or the single concept that is well
formed, product shaped and false: a RELATES_TO drawn between two concepts no
source ever joined, a concept named for something the product does not have.
Until this section there were three ways to deal with one, and all three were
bad: edit Neo4j by hand (which the next pass may undo, and which nothing
records), revetly forget the whole artifact (which takes the true inferences
with the false one, and re-extraction may draw the same edge again), or leave
it.
So the instance declares it, and a third deterministic pass at the tail of every ingest — after the alias fold and the noise withdrawal — takes it back. It costs no provider call, it survives every re-ingest and every tier promotion, and it is the write-side companion of the read-side cap: the cap serves less of the inferred half unasked, and this says which part of it is wrong.
Edges are invalidated; concepts are withdrawn. Two idioms, because the
graph already has two and an edge and a node do not mean the same. An edge is a
claim, so a matching edge gets invalid_at set to the pass clock, its
provenance set to retracted-by-instance, and the declared reason written
beside it — which is the same word every other closed-out fact in this engine
wears, so no read had to be taught a new one. A concept is an entity, so a
matching node leaves the Entity shelf with withdrawn_reason: retracted and
the declared reason, exactly as the noise withdrawal takes a name off it.
An edge is named by its two endpoints and its relation, matched
case-insensitively on the names and on the relation (toUpper on both sides),
in the direction the edge runs. Not by uuid: a re-extraction mints new uuids,
so a uuid retraction would survive exactly until the next time the artifact
changed. The cost of naming is worth saying out loud — a later, genuinely
true edge between the same two names under the same relation is retracted
too. That is a property of the declaration and not a defect in the pass: what
you have written down is "this pair, under this relation, is not something a
model may assert here".
An endpoint is an artifact by id, or a concept by name. A declared string
that equals a ledger artifact's id is matched as that artifact's id and nothing
else; anything else is matched as a concept's name, and a concept match never
lands on a node an artifact stands on. That is what stops a concept named for an
artifact's id and the artifact itself from both answering one declared string —
and it is the safe direction, because an id is exact and a name is not. A
concept whose name is id-shaped is still reachable: the row says
kind = "concept" (on a concept row) or source_kind/target_kind (on an edge
row), and the loader refuses any other word. A string that matches no ledger id
is a concept name, which is what most rows say.
What it never touches. An edge a source stated — retracting a stated fact
is a correction to the source, which is a revision, a delivery record or the
source's own edit. A node an artifact stands on. A statement shelf node. A
declared alias's canonical node — what the instance says a concept is called
outranks what it says about one name of it. And a node whose id was withdrawn
under the unsourced-id rule. A row that names one of this engine's own words —
Decision, Assumption, Constraint, ProductBacklogItem, Document,
Artifact, Entity, Episodic, STATED_IN — is refused at config load by
name, as is a row missing any of its fields. A reason is a field: the
record of why an instance took something back is the whole of what this section
adds over editing the graph by hand.
Retracting a concept leaves its edges standing. The node is off the shelf, so every read misses it and the edges reach nothing — and retracting them too would take back claims the declaration never named. The census counts them, so nothing about it is invisible.
Nothing is erased, and removing a declaration restores nothing. The edge
stays with its reason on it; the node keeps every edge it had; the one SET
that would put either back is the same as for a folded or noise-withdrawn node.
A retraction is a statement the instance made, and deleting the line does not
unmake it — taking an extraction back is revetly forget on the artifact, which
is priced. forget takes back a retracted concept standing on that artifact's
episodes too, through the MENTIONS edge its episode still carries: a retracted
node is off :Entity, so without that a re-extraction would mint a second node
beside it and an orphan would accrete on every cycle. A retracted concept
another artifact's episode also mentions is left standing, as an ordinary
concept is.
Every declaration is reported, row by row. The pass writes one line per
row to .revetly/ingest-health.json — the clock and count of its first
application, whether it has ever matched anything, what stands retracted under
it now, and how many edges it found only a source stating. revetly ingest
prints the same rows, saying matched, standing, matched nothing or
declined as stated, on a quote as well as on a landed pass; revetly audit-links --inferred lists them under their own heading; and revetly doctor
warns on a row that has matched nothing since it was declared, or whose only
matches were stated edges, naming the likely causes. A row with no record yet
is not judged — no pass has applied it, and a warning invented from nothing is
how a reader learns to skip doctor lines. This matters because the pass is
idempotent: zero is the correct answer for a working row on every pass after
the first, so a total cannot tell a dud row from a finished one.
The reads. search_context, get_artifact's links, the drafting bundle's
related and get_scope(question='binding') stop serving a retracted edge or
concept by default. deleted='include' serves them marked retracted, with
the declared reason and a note saying who decided it; deleted='only' serves
the withdrawn alone — the same word everything withdrawn on this surface
already comes back on.
The pass is idempotent and re-ingest-proof. An edge already carrying
invalid_at is outside the match and a withdrawn node is off the shelf the
match runs over, so a second pass over an unchanged corpus reports zero. An
edge the extractor draws again — after a tier promotion, after a body changed
— is a new edge with no invalid_at, and the same pass takes it back in the
same run. A second pass over an unchanged corpus also leaves the per-declaration
record byte-for-byte as it was.
Unlike the two passes above it, this one is registered on a lite instance as well. It applies nothing but the rows an operator wrote down by name, so there is no rule to mis-apply on a shelf nothing is adding to — and an instance that turned its provider off still has to be able to take back what the model wrote while it was on.
revetly audit-links --concepts is the census: how many concept nodes stand
for no artifact and how many per ledger artifact; how many have no edge at all;
how many wear an artifact shelf with no recorded id; how many are noise-shaped,
by shape, with examples; which declared alias spellings still stand apart; what
the two passes have folded and withdrawn so far; how many concepts and edges
[extraction.retractions] has taken back; and the best-connected names.
Read it before a pass and after one, and the effect is a number.
revetly audit-links --inferred lists the retracted edges under a heading of
their own, each with its reason — beside the inferred findings and never among
them, because those are edges a reader is being asked to judge and each of
these is one somebody has already judged and written down. The per-declaration
record leads that heading, so a row that has matched nothing is printed even
where no edge stands retracted. context_health carries the two counts beside
noise_shaped, read from the record the last pass wrote, so a client meeting a
short answer can tell a thin corpus from one an operator took something out of
on purpose.
revetly eval run scores the same measures on a seeded corpus whose answers are
known (revetly manual show scoring-extraction): the seed plants a sha, a
version, a path, an identifier and a git identity in its notes, and the score
reports how many noise-shaped names stand and whether any planted decoy does.
Run it under declared and under open and the difference is what the policy
buys — and what it costs, if a concept you wanted went with the noise. open
stays available for exactly that finding.
revetly eval run is refused by name on a lite instance: it scores what a
model extracts, and a lite instance declares none. revetly eval check still
runs — it scores this instance's own corpus against the questions in
eval/golden.json, slot by slot — and says, above its misses, that nothing it
scored was written by one: a slot whose answer is a concept, an inferred link
or a prose-only statement misses here by declaration rather than by a model
that did badly. Said above the misses rather than after them, because a reader
who learns it afterwards has already read them as a corpus that came out
wrong; and the report it writes carries lite, so a score read back later is
not taken for one a model ever touched.
It does not stop the extractor minting a node for the artifact an episode is
about — the title-node beside the id-node the linker anchors — which is where
the extracted summary lives and what search ranks an artifact on. It does not
retire what a model extracted from a kind that has since moved to index;
that is revetly forget, per artifact. And the shape table is the engine's:
an instance that needs a shape of its own asks, and the answer is a story.