Modular Knowledge Architecture
Active ResearchDesign AI systems where knowledge, capabilities, and behaviors are stored in separable, swappable, independently updatable modules — rather than entangled across all parameters.
Modularity in AI means: separate modules for separate capabilities that can be composed, swapped, or updated without retraining the whole model. Implementations: LoRA adapters (lightweight capability modules), MoE (architectural modularity), retrieval-augmented systems (external knowledge modules), tool use (capability delegation), and modular networks (explicitly routed sub-networks). The goal is composable AI: mix and match capabilities like microservices.
Why Is This Hard?
The Core Difficulty
Knowledge in dense transformers is distributed via superposition. Forcing modularity may require architectural changes that reduce the model's ability to leverage cross-domain transfer.
The Fundamental Tension
Dense models achieve strong performance partly because knowledge is entangled — the same parameters participate in many tasks. Enforcing separation may reduce capability.
Who Feels This
Enterprise deployers who need domain customization, safety teams who need to disable specific capabilities, anyone who needs to update a model more often than they can afford to retrain.
What Failure Looks Like
Cannot update a model's knowledge of current events without risking degradation. Cannot remove a capability (e.g. code generation) without retraining. Cannot combine two fine-tuned models reliably. Enterprise customers cannot customize models for their domain without expensive full fine-tunes.
Where Research Stands
Current Approaches
LoRA/QLoRA adapters, MoE routing, RAG for external knowledge, tool use for capability delegation, model merging techniques, adapter composition.
Best Result So Far
LoRA adapters achieve near-full-fine-tune quality at 1% of parameters. MoE models (Mixtral, DeepSeek-V3) achieve frontier performance with sparse activation. Model merging can combine capabilities with surprising effectiveness.
Remaining Gaps
Adapters don't truly separate knowledge — they modify shared representations. MoE expert specialization is emergent, not guaranteed. No architecture ensures that updating module A cannot affect module B. Composition of adapters is unpredictable.
What a Breakthrough Looks Like
Architectures designed from the ground up for modularity — where knowledge boundaries are explicit and enforced, not emergent and approximate.
What Success Looks Like
An AI system composed of independently trainable, updatable, and removable modules, where: (1) each module has a clear knowledge/capability scope, (2) adding/removing a module has predictable, bounded effects on other capabilities, (3) modules can be composed to create new capabilities, (4) knowledge updates are targeted and instant (swap a module, not retrain the model).
Timeline Horizon
3-5 years
Techniques That Address This
Task-specific distillation creates specialized compact models — distill a frontier model's medical knowledge into a small medical expert, its coding ability into a code expert, its reasoning into a reasoning expert. Each distilled model is an independent, deployable module with a clear capability scope. This creates de facto modularity at the model level rather than the weight level, enabling mix-and-match deployment architectures.
LoRA adapters are the most practical implementation of modular capabilities today. Each adapter is a self-contained capability delta (medical terminology, legal reasoning, code style) that can be independently trained, stored, shared, and swapped. Adapter merging (TIES, DARE) enables combining capabilities, and removal is trivial — delete the adapter, restore the base model. This is modularity at the weight level.
Memory hierarchy separates knowledge into independently manageable stores (archival, recall, working) — a form of knowledge modularity at the system level
Externalizing knowledge into a graph structure inherently creates modularity — knowledge domains are subgraphs that can be independently updated
If experts specialize (an open question), MoE provides architectural modularity — capabilities mapped to specific experts that could theoretically be added, removed, or updated independently
ROME's causal tracing reveals that factual associations are spatially localized in mid-layer MLPs — empirical evidence that knowledge CAN be modular, informing the design of architectures with explicit knowledge boundaries
Real-World Pressure
Enterprise demand for customizable AI, deployment cost pressure, safety requirement for capability control.
Key Organisations
Key Benchmarks