The Compiled Knowledge Base

Sometimes an idea arrives at exactly the right moment — not because the timing was planned, but because the problem it solves had been building long enough that several people were reaching for the same answer simultaneously.

Andrej Karpathy posted his LLM Knowledge Bases gist on April 3rd. I published Metadata Is the New Code on April 4th. I hadn't read his post when I wrote mine. He almost certainly hasn't read mine.

But we were describing the same thing from different directions. And the convergence is worth unpacking — because it points at something more fundamental than a clever workflow trick.

Both arguments converge on the same diagnosis: most AI-assisted systems aren't failing because of retrieval quality or model capability. They're failing because the knowledge they're built on was never given a stable, persistent form.

The Problem RAG Was Always Solving Wrong

Standard RAG has a well-known architecture and a less-discussed philosophical flaw. It treats knowledge as something you retrieve, not something you accumulate. Every query is a fresh excavation. The system doesn't get smarter with use — it just gets faster at finding the same chunks.

Karpathy put it precisely in his gist: the LLM is rediscovering knowledge from scratch on every question. Nothing is built up. Ask something that requires synthesising five documents and the agent has to find and reassemble the fragments every time. NotebookLM, ChatGPT file uploads, most RAG implementations — all of them work this way.

That's not a retrieval problem. It's a memory architecture problem.

In Metadata Is the New Code, I described the same failure from the practitioner side. Most AI-assisted development sessions start with the same re-explanation ritual. You tell the agent which LLM providers you use, paste your infrastructure setup, describe your scheduling patterns. The agent builds something reasonable. You move on. Next session — you do it again, slightly differently, producing outputs subtly inconsistent with everything you built before.

The knowledge isn't lost. It's just trapped — in past conversations, in READMEs, in your own head. It never acquires a form that survives the session that produced it. And each time you re-explain instead of reference, you're accumulating a retrieval tax on your own cognition — one that connects directly to something I've been tracking since The Drift.

What Karpathy Actually Proposed

What makes Karpathy's framing distinctive — and precise — is the word he reaches for: compiler. In standard RAG, the LLM is a reader. It reads chunks on demand and generates answers. In his architecture, the LLM is a compiler. It reads raw source material and produces a structured, interlinked wiki. The compilation creates value. The retrieval is just navigation within what was already built.

The architecture operates in three stages:

Ingest — raw/ directory Raw materials (research papers, GitHub repos, web articles) land here. Web content is converted to Markdown via Obsidian Web Clipper, images stored locally for vision-capable models to reference.

Compile — LLM as author The LLM reads from raw/ and authors structured wiki articles — cross-linked, summarised, backlinked. It doesn't retrieve on demand. It transforms raw material once into something navigable. You rarely edit the wiki manually; it's the LLM's domain.

Lint — Active maintenance Periodic linting passes scan the wiki for inconsistencies, missing links, and new connections. The knowledge base heals and extends itself. The compiled artifact grows more precise with each cycle.

At the scale Karpathy is operating — around 100 articles and 400,000 words — the compiled wiki navigated via summaries and index files outperforms a full RAG stack. The overhead of chunking, embedding, and similarity search introduces more latency and retrieval noise than it removes. The compiled artifact is smarter than the source material it was built from. That's the point. That's what RAG cannot do.

The Connection to the Drift

In The Drift, I described the Cognitive Offloading Loop — the mechanism by which AI-assisted development quietly degrades the developer's own mental model over time. The brain detects that a task is being handled externally and starts de-prioritising the neural pathways associated with doing it. The result:

Better output → less deep thinking → weaker mental models → vaguer prompts → worse output → frustration

The tell is a specific kind of prompt degradation. Six weeks into heavy agent use, you catch yourself writing "refactor this to be more maintainable" and wondering why the result feels off. Earlier, you'd have written: "extract the validation logic into a separate module, make it unit-testable, and don't touch the API contract." Same intent. Completely different outcome. The difference is the mental model you brought to the prompt — and that model had quietly eroded.

Here's what I didn't make explicit enough in that article: unexternalised knowledge accelerates this loop. When domain knowledge lives only in your head, you have to re-engage it conversationally on every build session. That constant re-explanation feels like engagement — but it's a symptom of knowledge that was never properly captured. And each re-explanation session is one step closer to the drift prompt.

Connection to The Drift: Artifacts as a countermeasure to cognitive offloading

When knowledge is compiled into a persistent, human-readable wiki that the agent reasons from — rather than reconstructed from scratch each session — two things happen simultaneously. The agent gets better grounding. And you get a forcing function for keeping your own mental model precise.

You maintain the wiki. You review the linting outputs. You update the compilation when new source material arrives. The mental model stays sharp because it's anchored to something explicit — not reconstructed from memory each time.

Karpathy's compiled wiki is a structural defence against cognitive offloading that RAG was never designed to provide. The artifact is the discipline. The discipline is the architecture.

Connection to Metadata Is the New Code: Artifacts over explanations — the same principle at portfolio scale

In Metadata Is the New Code, I called this the difference between explanations and artifacts. When domain knowledge lives in a versioned, structured artifact, the agent reasons from the artifact. You reason from the artifact too. The mental model stays sharp because it's anchored to something explicit.

My Reference Catalog — an LLM taxonomy, scheduling taxonomy, deployment strategy reference, and article classification hierarchy — operates on exactly this principle. Each artifact was built once and now pays back on every project that consumes it. The LLM taxonomy alone has been injected into every multi-model routing component I've built since MRAG.

Karpathy's wiki is a knowledge artifact at session scale. The Reference Catalog is a knowledge artifact at portfolio scale. The principle is identical: externalise what you know, give it a stable form, and let every downstream build consume the same structured truth.

The Enterprise Implication Nobody Is Saying Loudly Enough

Most coverage of Karpathy's post has treated it as a personal productivity story. That's the least interesting reading.

Responding to his post, entrepreneur Vamshi Reddy wrote something that deserves more attention than it got:

"Every business has a raw/ directory. Nobody's ever compiled it. That's the product."

Think about what that means concretely. Every organisation is sitting on decades of institutional knowledge — Confluence pages, Slack exports, incident post-mortems, architectural decision records, documents authored by people who have since left. It exists. It just has no compiled form. It can't be navigated, cross-referenced, or reasoned over. It can be keyword-searched, at best.

A Karpathy-style enterprise layer wouldn't search this data. It would author a living institutional memory from it — one that self-heals, self-links, and stays auditable because every claim traces back to a plain-text file a human can read, edit, or delete. No vector embedding opacity. No retrieval noise. Full provenance.

Auditability as competitive advantage In regulated environments — pharma, finance, healthcare — GxP compliance, ALCOA+ principles, and audit trail requirements demand traceability from AI output back to source. RAG's opacity is a structural problem in these contexts. A compiled Markdown wiki with explicit provenance is architecturally aligned with regulatory requirements in a way that vector embeddings simply aren't.

The right question for enterprise RAG The enterprise RAG conversation has been dominated by retrieval quality metrics and embedding model comparisons. The conversation that should be happening is whether retrieval is even the right primitive — or whether the organisation's knowledge problem is fundamentally a compilation problem that RAG was never designed to solve.

Prompt Norm Decay at organisational scale In The Drift, I described Prompt Norm Decay — shared prompt libraries frozen in time, quietly leaving capability on the table as models improve. A compiled knowledge base with version control and linting cycles is structurally resistant to this. The artifact forces the update; the prompt library assumes someone will remember to do it.

Where This Gets Interesting for Multi-Agent Systems

A design pattern already emerging from community discussion scales Karpathy's workflow to a 10-agent system with a supervisory model acting as quality gate — scoring and validating draft wiki pages before they enter the live knowledge base, creating a compound loop where raw outputs are refined, checked, and fed back as high-integrity briefings to the broader agent swarm.

This is where the compiled knowledge base becomes a multi-agent memory architecture. In the MRAG architecture I've been building — with GPT, Claude, Gemini, and Grok operating across Broadcast, Orchestrate, and Mediator modes — one of the persistent design challenges is shared state. Agents that share a context window are reset-bound. Agents that share a compiled knowledge base are not. The wiki accumulates. Individual sessions reset; institutional memory compounds.

Broadcast as collective calibration MRAG's Broadcast mode sends the same context to multiple models simultaneously, with outputs synthesised by a Mediator. Individual model drift gets averaged out across the swarm. Karpathy's compiled wiki + a multi-agent quality gate creates the same structural averaging at the knowledge level: no single compilation pass dominates; each linting cycle corrects the last. These aren't separate patterns — they're the same principle at different layers of the stack.

Fighting agent-level drift through persistent state In The Drift, I described how agents themselves drift — early session decisions get summarised or dropped, context degrades, and the agent starts hallucinating consistency it no longer has. A compiled wiki with explicit backlinks and index files is a structural countermeasure: the agent navigates to what it needs rather than assuming it still holds it in context. This is the multi-session analogue of CLAUDE.md hygiene — but at an architecture level, not a file level.

Shared knowledge state vs. shared context window Multi-agent systems built on shared context are fragile at the boundaries — context limits, session resets, and model-specific summarisation all introduce divergence. Multi-agent systems built on a shared compiled knowledge base have a stable source of truth that persists independently of any individual agent's session state. The knowledge doesn't belong to any agent. It belongs to the system.

The Long Game: Fine-Tuning on What You've Built

Karpathy gestures at where this ultimately leads: as the wiki is continuously linted and refined into a high-quality, consistent, cross-referenced corpus, it becomes a candidate for fine-tuning. Eventually, you don't retrieve from the wiki — you train a model on it, and the knowledge lives in the weights.

The RAG-vs-fine-tuning debate has always been framed as a trade-off. Karpathy's architecture dissolves it. They're sequential stages in the same pipeline.

Raw data → compiled wiki → synthetic training corpus → fine-tuned model with domain knowledge at the parameter level. A personal AI that doesn't need to retrieve because it already knows. An enterprise model that embeds institutional memory at the weight level rather than the retrieval layer.

This is also the endpoint implied by the Metadata article's core argument: structured knowledge artifacts aren't a retrieval convenience — they're a compounding asset. The LLM taxonomy I maintain isn't just useful for the next routing component I build. If it's linted, versioned, and cross-referenced with sufficient rigour, it becomes training data. The investment front-loads; the returns compound indefinitely.

What This Means in Practice

Karpathy described his current implementation as "a hacky collection of scripts." That's worth taking seriously — the architecture is available to anyone with a raw/ directory, a large-context LLM, and the discipline to maintain a wiki rather than just accumulate a pile of source documents. The discipline is the hard part.

It's the same discipline The Drift and Metadata Is the New Code are both ultimately arguing for: staying the architect rather than becoming the proxy. Building the habit of externalising what you know into a form that survives the session. Treating knowledge artifacts as engineering artefacts — versioned, reviewed, maintained.

01 — Stop designing around retrieval. Design around accumulation. The question isn't "how do I find the right chunk?" It's "how does my system get smarter with every interaction?" The compiler metaphor is the answer: raw documents → structured knowledge. The transformation creates value. Retrieval just navigates what was already compiled.

02 — Plain text is infrastructure. Markdown's resilience as a format is underrated. In a world where every AI platform is 18 months old, portability is a moat. Files outlast apps. If Obsidian disappears, the wiki remains. That's not a coincidence in Karpathy's design — it's a first principle.

03 — Auditability is competitive advantage in regulated domains. The ability to trace an AI claim to a human-readable source file is worth more than marginal retrieval accuracy. Vector embeddings are black boxes. Compiled Markdown wikis are not. This distinction will matter increasingly as AI-generated outputs enter regulated workflows.

04 — Artifacts are drift countermeasures, not just productivity tools. When domain knowledge lives in a versioned, structured artifact, you maintain the mental model by maintaining the artifact — not by re-explaining from memory. The externalisation is the discipline. The discipline is what keeps you on the right rung of the ladder.

The three articles connect because they're describing the same failure mode at different levels of abstraction. The Drift describes what happens to the individual developer when cognitive offloading erodes the mental model. Metadata Is the New Code describes what happens to a portfolio of AI-driven builds when domain knowledge is never externalised. The compiled knowledge base is the architectural pattern that addresses both — simultaneously, at every scale.

The raw/ directory already exists everywhere. The question is who builds the compiler — and who maintains the discipline to keep it current.

Give your agents something worth reasoning from. And stay the architect.

This article is part of the AI Engineering series on aiOps3000. Part 01: The Drift · Part 02: Metadata Is the New Code · Part 03: this article.

Original article: https://venturebeat.com/data/karpathy-shares-llm-knowledge-base-architecture-that-bypasses-rag-with-an

Francisco Marques da Silva Director-level AI & Data practitioner, 20+ years building and leading transformation programmes across industries and borders. Publishing under the aiOps3000 brand, on aiops3000.com · Hashnode · LinkedIn