Skip to content

What makes a node findable

An agent working in your codebase does not browse your knowledge base. It issues a query, gets back a handful of nodes, and reasons over those. Whatever did not come back may as well not exist. So the question "what makes a node findable" is not a stylistic one — it decides which of your institutional knowledge is reachable at the moment it is needed.

This page answers that question with measurements rather than intuition, using Hadron's own platform-specification corpus as the subject. It is the first of a series of research notes. Two things make it possible to run at all — a corpus that is a queryable store rather than a folder of files, and an embedding pipeline whose exact behaviour is knowable — and that turns out to be the larger point.

The intuition that failed

The investigation started from a plausible claim: long abstracts dilute into weak vectors.

The reasoning is sound on its face. A node's abstract is embedded as a single 768-dimensional vector (see The RAG tech stack). Embedding models pool over the whole input, so the more distinct material you pack into one vector, the more each individual claim gets averaged away. Pack 1600 characters of specification into one vector and you would expect a short question about any one part of it to match weakly.

The observation that prompted the claim looked like confirmation. One spec with a roughly 1300-character abstract scored 0.59 against a conversational question it answers almost verbatim — down among its neighbours, indistinguishable. Another spec with a roughly 700-character abstract scored above 0.9 on its own query. Two nodes, two lengths, a large gap in the expected direction.

Two nodes, and also two different queries. Nothing in the comparison held anything fixed. That is the flaw the rest of this page is about, and it is the single most common way retrieval intuitions go wrong: the variable you noticed is rarely the variable that moved.

How to measure this properly

Every measurement below ran against the production embedding model, offline: nomic-embed-text v1.5, 137M parameters, 768 dimensions — the same model a Hadron server serves through Ollama in development and SageMaker in production, with the same mandatory task-instruction prefixes (search_document: for indexed text, search_query: for queries; getting those wrong silently degrades recall). Body chunking mirrored the server's own structure-aware chunker.

  • Index — all 271 abstracts in Hadron's live platform-spec corpus. Not synthetic text: the real thing, competing against its real neighbours.
  • Subjects — the 69 rule-tier specs whose abstracts run at least 1000 characters over at least four sentences, leaving room to manipulate.
  • Queries — 476 conversational questions, one generated per abstract sentence, instructed to use everyday wording and to avoid the source sentence's distinctive noun phrases.
  • Metrics — cosine similarity, and the two that actually decide retrieval: the subject node's rank against the real corpus, and its top-1 rate.

Validate the instrument before trusting a null result

A negative finding from an insensitive instrument is worthless. So before testing length, test something known to be harmful: pad a spec's core sentence with off-topic text drawn from unrelated specs.

padding length mean cosine Δ top-1 rate
none 179 0.6912 — 37.7%
off-topic 700 0.6482 −0.043 13.0%
off-topic 1000 0.6419 −0.049 10.1%
off-topic 2000 0.6430 −0.048 6.5%

Dilution is real, it is large, and it is plainly visible at 700 characters. The instrument works. Whatever it now says about length can be believed.

What the data says about length

Truncation. Cut each subject's abstract back sentence by sentence. Balanced panel — only the 25 subjects with eight or more sentences, so every length step contains exactly the same nodes:

sentences kept mean length mean cosine mean rank
1 166 0.6815 11.48
3 579 0.6886 5.64
5 1000 0.6947 3.92
8 1524 0.6950 3.80

Longer is better, monotonically, out to 1500 characters.

Extension. Grow each real abstract past its natural length using on-topic prose from the node's own body — precisely the "the abstract just restates the body" pattern the original claim indicted — holding the queries fixed:

length Δ cosine vs. real 95% CI top-1 rate
1377 (unmodified) — — 42.1%
1728 +0.0004 ±0.0020 35.9%
2109 −0.0015 ±0.0015 36.5%
3099 −0.0082 ±0.0017 31.0%

Nothing measurable until about 2000 characters — which happens to be the platform's own cap on the field.

Compression — the decisive test. Truncation removes content; distillation does not. Rewriting 35 abstracts to preserve every distinct claim in far fewer words, then scoring them on the full per-sentence query set so coverage is held constant:

variant mean length mean cosine Δ vs. unmodified top-1 rate
distilled 722 0.6959 −0.0010 ± 0.0041 45.7%
distilled 918 0.6997 +0.0029 ± 0.0037 48.6%
unmodified 1396 0.6969 — 43.3%

Both confidence intervals straddle zero. Distilling a 1396-character abstract down to 722 while keeping every claim changes retrieval by nothing measurable. Tightening your prose is good writing. It is not a retrieval intervention.

There is a real dilution cost to length — measured paired, same query against the same node, baselined on the shortest variant that already contains the answering sentence:

extra on-topic characters Δ cosine mean rank
+1 to 200 +0.0007 9.56
+400 to 700 −0.0046 6.50
+700 to 1100 −0.0127 5.90
+1100 to 2000 −0.0125 4.83

That is about −0.013 cosine per additional 1000 characters, against a corpus whose median first-to-second-result gap is 0.0152 — so roughly one rank position's worth. But rank improves anyway, because the added text also wins the node queries it would otherwise have missed entirely. Coverage beats concentration.

Where the threshold landed

Hadron's spec linter warns on over-long abstracts, and this is the analysis that set its bound. Applied to the 180 rule- and flow-tier specs in the corpus:

threshold specs flagged verdict
800 115 (63.9%) inside the plateau — no measured basis
1000 71 (39.4%) inside the plateau — no measured basis
1200 43 (23.9%) inside the plateau
1600 19 (10.6%) top of the plateau
2000 0 the platform's own cap; would never fire

1600 is where added length has demonstrably stopped paying for itself — a ceiling, not an optimum. The lint message says so, and points at subject drift rather than brevity, because that is what the data supports. A rule firing at 1000 would have flagged two-fifths of the corpus for no measurable gain, and its guidance would have pushed authors toward the flat-to-worse end of the curve.

The real answer: one node, one subject

Put the two results side by side.

  • 1000 characters of on-topic material: −0.013 cosine, and better rank.
  • 1000 characters of off-topic material: −0.049 cosine, top-1 rate from 38% down to 10%.

Roughly a fourfold difference in cost, in opposite directions on the metric that matters. What dilutes an embedding is subject drift, not word count. A 2000-character abstract about one thing is a strong vector. A 600-character abstract about three things is three weak ones averaged together.

This reframes the authoring rule. "Keep abstracts short" is a proxy — and a poor one: across the corpus, abstract length predicts internal topical coherence at r = −0.29. The rule that actually follows from the data is one node, one subject, and it is a rule about node granularity, not about prose style. If an abstract has drifted, the fix is usually not to trim it. It is to notice that the node is carrying two subjects and split it into two nodes — each with its own address, its own abstract, and its own vector.

Where the original observation went wrong

Re-running the query that started all this reproduces the symptom exactly, and shows the cause is not the node. On that question, the entire corpus bands between 0.68 and 0.70. Corpus-wide, the median gap between the first and second result is 0.0152. Nothing anywhere in the corpus scores above 0.9 on a natural-language question — a 0.9 requires a near-verbatim query, which is what the "good" node in the original comparison happened to get.

So the spec was not weakly embedded. It was competing in a field where nothing was clearly distinguishable from anything else. The failure was a property of the corpus, not of the node — and no amount of rewriting that one abstract would have fixed it.

This is the most transferable finding here. When retrieval disappoints, the instinct is to blame the document you were looking for. Check the spread of the result set first. A narrow band means your corpus has a discrimination problem: too many nodes making overlapping claims, or nodes drawn at a granularity where neighbours are near-duplicates of each other. Both are structural, and both are fixed by re-cutting nodes rather than by rewriting them.

What this means beyond specs

Nothing in the mechanism is specific to specifications. Any node with an abstract — a finding, a meeting record, an imported research page, a runbook — is embedded the same way and competes the same way. The transferable rules:

  • Write the abstract as the questions a reader would ask, in your own words rather than as a restatement of the body. It is a genuinely separate retrieval surface, and phrasing it complementarily to the body buys you coverage the body's own chunks do not already provide.
  • Do not shorten for retrieval's sake. Shorten for a reader's sake. The measured cost of on-topic length, up to the field's cap, is close to zero, and the coverage you lose by cutting a claim is not.
  • Split, don't trim. When a node's subject has broadened, that is a granularity signal. Splitting produces two well-separated vectors; trimming produces one vague one plus a gap in coverage.
  • Judge retrieval by the spread, not the score. An absolute cosine of 0.59 means nothing on its own. A first-to-second gap of 0.002 means your corpus cannot tell two nodes apart, and that is actionable.

Why this experiment needs a managed corpus

The honest argument for keeping specifications and findings in Hadron rather than as markdown files in a repository is not that markdown is bad. It is that the questions that turned out to matter here cannot be asked of a folder of files.

Every step of this investigation depended on properties a flat file tree does not have:

  1. The corpus is enumerable and typed. "Every rule-tier spec with an abstract over 1000 characters" is a query. On a file tree it is a scavenger hunt through whatever the naming convention happened to be that quarter.
  2. The abstract is a first-class field, embedded separately from the body. That separation is what makes a complementary retrieval surface possible at all — and what made it measurable. Markdown has one text field; front-matter summaries are not indexed as anything distinct.
  3. Retrieval is a defined, inspectable pipeline. A known model, known prefixes, known chunking. Every number here is reproducible because the mechanism is specified rather than delegated to whatever an agent's file search does that day.
  4. The corpus has an identity, so it can be compared against itself. The decisive finding — a first-to-second-result gap of 0.0152 — is a corpus statistic. It cannot be computed for a directory, because a directory has no notion of what competes with what.
  5. Conventions are enforceable. The conclusion of this work landed as a lint rule that runs over the whole corpus and reports the specs it applies to. Guidance that is only written down is guidance that decays; guidance that is executable does not.

None of that makes files worthless. If your specifications are read by people, in order, a handful of times, a repository is the right tool and Hadron is overhead. The trade turns when your specifications are read by agents, out of order, by meaning, hundreds of times a week — because at that point retrieval quality is the product, and retrieval quality is something you can only manage if you can measure it.

That is also why this page exists as research rather than as advice. The guidance here is falsifiable, it changed once already when it was tested, and it will change again. A knowledge-management system worth adopting should be able to be interrogated about its own behaviour and be found wrong.

Limits of this result

Stated plainly, because they bound how far you should carry it:

  • One corpus, one domain. 271 platform specifications written to a common rubric by a small number of authors. A corpus of meeting notes or customer research may have different neighbour density and behave differently.
  • One embedding model. All of this is nomic-embed-text v1.5. The qualitative story — drift dominates length — should survive a model change; the exact thresholds will not.
  • Machine-generated queries. Realistic in wording, but generated from the sentences they are meant to retrieve, which likely overstates absolute scores. Comparisons between conditions are unaffected, because every condition faces the same queries.
  • Single-vector abstracts. The results describe today's pipeline. Multi-vector abstracts would change the dilution story materially.
  • Correlations are observational where marked. The truncation, extension and compression experiments are controlled; the corpus-wide correlations are not.

Reproduce it

The whole method fits in a paragraph, which is the point.

ollama pull nomic-embed-text
hadron spec get --prefix cor -m hadronmemory.com:specs --json > specs.json

Embed abstracts with the search_document: prefix and queries with search_query:, L2-normalise, take dot products. Then hold everything fixed except the one variable you are testing — the same query against the same node — and change one thing at a time. Every misleading number in the original investigation came from changing two.

Every number this page relies on is on this page. The engineering write-up that accompanied the resulting lint rule — the same experiments, plus the implementation notes and the rejected alternatives — lives in the CLI repository: docs/plans/spec-abstract-length.md.