Is Your LLM Knowledge Base Getting Messier with Every Update? Use NeuG to Define Concept Boundaries and Pinpoint What Needs Updating

banner

An LLM-generated code wiki can look impressive on the first run. It turns scattered code into readable sections and produces useful explanations in very little time.

But a knowledge base is not generated only once. When we ran the same code through the process again, the LLM did not preserve its previous concept map: sections could be split or merged, and the same code could move elsewhere. For example, code that handles type conversion (cast) might appear under “type system” in one run and “built-in functions” in the next.

A second problem appeared when the code itself changed. The LLM had to infer from the git diff which wiki sections needed updating. When an effect crossed file or directory boundaries, it often updated the most obvious sections and left other affected sections stale.

Each result may look reasonable on its own. Taken together, however, they stop behaving like one coherent knowledge base: concept boundaries drift, affected sections go untouched, and after a few updates it becomes hard to know what can still be trusted.

We ran into this while maintaining NeuG’s code wiki. We eventually moved two decisions out of the LLM prompt and into graph computation: derive concept boundaries from code relationships when building the wiki, and calculate which concepts are affected when the code changes. The LLM still reads and writes; NeuG uses the graph to decide which files belong together and what changed.

The rest of this post shows how that works on NeuG’s own codebase. A complete tutorial is linked at the end.


An LLM Can Write a Wiki, but Keeping It Coherent Is Hard

Build time: concept boundaries are unstable. The LLM reads code and decides how to split sections on its own. But it tends to follow directory structure — and directory structure ≠ concept structure. common/types/, function/cast/, function/comparison/ all belong to “type system”, yet the LLM splits them into two independent sections.

Update time: impact scope is uncertain. Code changes, and the LLM reads the git diff to guess which wiki sections are affected. It often glances at a few sections and starts writing, while the ones that actually changed never get touched. What should have been updated stays stale, and the knowledge slowly rots.

Neither is primarily a writing problem. They are relationship problems: which files belong to the same concept, and which concepts are affected by a code change, cannot be decided reliably by looking at files in isolation.

We use a code graph to give those decisions a stable basis.


Build: Leiden Draws Boundaries, PageRank Picks Files

First, we build NeuG’s codebase into a graph: files and functions as nodes, function call relationships as edges. At the symbol level, that gives us ~28,000 nodes and 52,501 edges.

But symbol-level is too fine-grained for concept grouping — file-level works better. So we aggregate the symbol graph into a file-level view: “file contains functions, functions call each other” collapses into file-to-file relationships, yielding 1,423 file nodes and 3,869 edges.

This view is only needed for computation, so we import it as a temporary graph using NeuG v0.1.3’s COPY TEMP — run the analysis, discard the temp data, no pollution to the persistent graph.

Then we do two things on the file-level graph:

First, Leiden community detection to define concept boundaries. Tightly related files naturally cluster into the same community, independent of directory structure.

CALL leiden('code_graph', {concurrency: 1})
YIELD node, community
RETURN node.name, community

Take a typical community from NeuG’s codebase: type-and-value-system. It contains 187 files spanning 6 path prefixes. The figure below compares the two approaches on 6 representative elements: the graph approach uses Leiden clustering to correctly group types, values, casts, and comparisons into one “type system” section; the LLM approach follows directory structure and splits them into common-types and built-in-functions — severing the semantic connection.

type-and-value-system clustering comparison

Second, PageRank to pick core files per concept group. Once communities are defined, which files should the LLM read to write the wiki? It usually resorts to heuristics — like picking files with short names. The graph approach uses PageRank to rank files by global citation count and directly surfaces the most important ones:

CALL page_rank('code_graph', {max_iterations: 20})
YIELD node, rank
WITH node, rank WHERE node.community = 37
RETURN node.name, rank
ORDER BY rank DESC LIMIT 5;

For the cypher-parser section, PageRank’s top pick is transformer.h (330 lines, declares all transform methods, referenced by many files); the LLM picks use_database.h (39 lines, 1-2 references, chosen because the filename is short and it’s a .h).

The PageRank version covers the Transformer class’s 80+ method dispatch architecture and operator precedence chains. The LLM-selected version gets the overview right but has to infer the core architecture, leaving readers to check the source code. The difference is not writing style but evidence: one version reads the files that support the section, while the other reads files that merely look relevant.

PageRank vs LLM file selection

Third, the same grouping can be reproduced. Graph computation produces the same output every run; LLM-generated wiki drifts each time:

Reproducibility comparison


Update: Leiden Assigns New Concepts, Cypher Computes the Delta

Building handles “how to split the first time.” But code changes daily — a knowledge base can’t be split once and forgotten.

NeuG provides an incremental freeze-assign Leiden algorithm: existing nodes keep their concept assignments, only new content gets re-clustered. A module that was “compiler” before stays “compiler” after. No drift, fully diffable.

CALL leiden('code_graph', {concurrency: 1,
  initial_community_property: 'delta_comm'})
YIELD node, community, previous_community
RETURN node.id, community, previous_community;

Then use Cypher to compute community-level deltas: group by community, precisely measure each community’s change — stable / growth / new. Only communities that actually changed get fed to the LLM; stable ones are reused as-is.

MATCH (n)
WITH n.community AS community, count(*) AS total,
     count(n.previous_community) AS old_members
RETURN community, total - old_members AS new_members,
  CASE WHEN old_members = 0 THEN 'new'
       WHEN old_members = total THEN 'stable'
       ELSE 'growth' END AS change_type;

For example, analyzing a NeuG version update (47 changed files, +14,243 / -61 lines) against the wiki:

LLM approach: reads all 35 changed files, compares against 23 sections one by one. Might flag 4 for update, but two are false positives.

Graph approach: directly identifies 21 concepts (wiki sections) with zero changes, and 2 concepts with new members — only those two sections need incremental updates.

Incremental update delta comparison


Keeping the Same Knowledge Base Across Many Updates

The graph does not write the wiki. It answers two questions that need consistent answers over time: what evidence defines a concept boundary, and which sections actually need to be rewritten after the code changes.

NeuG computes the boundaries and affected areas from code relationships. The LLM then reads the relevant code and writes the explanation. The point is not only to produce a good first version, but to keep the same conceptual structure after several updates without leaving changed knowledge behind.

If you have seen concept groups drift or struggled to decide which sections to update after a code change, tell us what happened in your project.


About NeuG

NeuG is an open-source graph database. v0.1.3 added the GDS graph algorithm extension (9 algorithms including Leiden, PageRank, BFS, SSSP, etc.) and COPY TEMP for ad-hoc analysis without polluting the production graph.

  • GitHub: https://github.com/alibaba/neug
  • GDS extension docs: https://github.com/alibaba/neug/blob/main/doc/source/extensions/load_gds.md
  • Companion tutorial (step-by-step reproduction of this post’s wiki build & update): https://github.com/alibaba/neug/blob/main/doc/source/tutorials/code-graph-wiki-pipeline.md
  • Stars, issues, and PRs welcome