Teaching agents
to find, judge & use
Drupal modules
agent-module-documentation & the Agent Module Knowledge MCP
Marcus Johansson
- Senior Developer at FreelyGive
- One of the Tech Leads in the Drupal AI Initiative
- Maintainer of the two projects in this talk
This is a great module.
Shy One-Time
Introduction When requesting a one-time login link (request new password or password forgotten function), it often comes to the fact that the link arrives invalidated/invalid via e-mail. This can…
Categories: Access control
It fixes this one, which every Drupal site behind Outlook has seen:
“You have tried to use a one-time login link that has either been used or is no longer valid.”
Mail security scanners click the reset link first, so the user's own click is already too late.
With the worst title.
…and a description that could be better.
- “Shy”? “One-Time” what?
- Nobody searches for “invalidated/invalid”
- Category: Access control, not “email” or “password reset”
“Build me a Facebook login.”
It happily does it.
No question asked. No module searched for.
A brand-new, custom OAuth flow written from scratch, enabled on your site and maintained by nobody.
There is a covered contrib module for this, and the agent never looked.
“OK, then find me a module.”
drupal.org's module search is an HTML app behind a bot challenge. The structured data is on a JSON:API endpoint that Project Browser knows about. The agent does not:
GET /jsonapi/node/project_module ?sort=-field_active_installs_total &filter[...]=field_core_semver_minimum …
Four candidates. Which one?
Social Auth Facebook
OpenID Connect
miniOrange OAuth Login
fbl
“Minimally maintained”
…for the thing that logs users in. Good or bad?
“Covered”
Means the security team issues advisories for stable releases. It is not a code review.
“Maintenance fixes only”
Finished and stable, or slowly dying?
To use it, the agent reads the code
It reads the module, its parent framework, its dependencies and the vendor library, then guesses the config schema.
Every task, every project, every developer. The same reading, over and over.
Extra cost per task vs. having the docs (long-tail evals, Opus 4.8)
+47.6k tokens · +24s · +$0.10
If 10,000 Drupal developers did that once a day:
0 tokens/day
0 agent-hours/day
0 /year
…to re-learn what one documentation run already wrote down.
Agents are good. Don't let them write your OAuth.
Easy to get wrong in a hand-rolled login
- No
statecheck → login CSRF - App secret committed to the repo
- Linking accounts by unverified email → account takeover
- Not validating who the token was issued for
- Open redirects on the callback
A non-developer can't spot any of these. And nobody gets a security advisory for your custom module.
Drupal's quiet superpower: the Security Team
- Private reporting & coordinated disclosure
- Security advisories for core and covered contrib
- Coverage is opt-in: stable release + vetted maintainer
- Update Status tells every site when it runs a vulnerable release
- Unfixed, abandoned projects get marked unsupported
Reusing a covered module means thousands of sites and a security process behind your login button.
We have to teach the agent to do any of this
# CLAUDE.md (every project, every developer…) Before writing custom code, search drupal.org for an existing contrib module. Prefer modules with security advisory coverage. Check Drupal 11 compatibility and maintenance status. Read the module's docs before using it. Never implement OAuth yourself. …
It should already know.
Every rule we have to write is one that someone else forgot to write.
The npm happy path vs. Drupal
- Public search API, CLI search, README in the package
- Massive training data: the model just knows
- Search behind HTML/JS, docs scattered or in code
- Long tail barely in training data
When agents pick the stack, “it just worked” wins, and Drupal gets left behind.
Drupal Module Finder
Describe the problem, find the Drupal module.
Works
Semantic search over the knowledge base, plus a skill so agents can call it.
But…
- Another server to host, scale and pay for
- …or ask drupal.org's infrastructure to run something unproven
What if every developer
ran it locally?
An MCP server shipped as a DDEV add-on.
One command, and your agent knows the Drupal ecosystem.
Tools: search_modules · get_module · list_docs · read_doc · list_categories
Every Drupal 11 module, read by an agent
In popularity order: install on a real Drupal 11 site in DDEV → read code & config → write compact, agent-first docs.
Kept current: scanners watch drupal.org release feeds and re-document new stable minors.
~1,000 commits · 1,843 modules with eval suites · open on GitHub
Browse it: drupal-ai.github.io/agent-module-documentation ↗
For finding modules
usage.md: short summary, long summary, then 15–30 use cases written in the user's words. Plus categories, subcategories and keywords in data.json.
shy_one_time / data.json "categories": ["Security", "User management"], "subcategories": ["Login", "Email deliverability"], "keywords": ["password reset", "one-time login", "crawler", "bot", "email scanner", …]
shy_one_time / usage.md: use cases - Stop a mail scanner consuming a reset link. - Fix "link already used" complaints. - Protect one-time login links from bots. - Reduce password-reset support tickets. - Survive a corporate email gateway. - Prevent link-preview services burning tokens. - Keep reset links usable in enterprise email. - Handle a security appliance that follows links. - Support users behind a filtering proxy. …
These are the sentences people actually type, so problem #2's search now has something to match.
For evaluating modules
social_auth_facebook / 4.0.x / data.json { "core_version_requirement": "^9.5 || ^10 || ^11", "security_advisory_coverage": "covered", "maintenance_status": "Minimally maintained", "development_status": "Maintenance fixes only", "contributor_count": 17, "dependent_modules": [ … ], "composer_requirements": { … }, "provides_drush_commands": false, "tooling": { "provides_tool_plugins", "agent_skills" } }
The long summary is an honest assessment, not marketing:
“What deserves saying plainly is that this is a trade, not a fix: a link that survives being fetched by a third party is a link that third party could still use…”
shy_one_time / usage.md
The agent gets trade-offs and caveats, not just a feature list, before it recommends anything.
If we say “Drupal is more secure”,
we have to check it.
Every module gets a security read
The agent reading the source to document it also looks for:
- Insecure shipped defaults (placeholder keys, protection off)
- Unauthenticated SSRF, injection, access bypass
- CSRF on state-changing routes
- Unsafe file / archive / eval / unserialize handling
- Credentials stored in plain text in config
Findings stay private
Security notes are git-ignored and never pushed to the public repo. They go through coordinated disclosure, not GitHub.
The Security Team are the true heroes
We automate as much as we can, in a way that makes their work easier. They do the hard part.
For using modules
social_auth_facebook/4.0.x/ ├── data.json ├── usage.md └── agent/ ├── start.md ← cheap index ├── configure/ │ └── settings.md └── api/ └── network-and-manager.md
The rule: every agent doc must be shorter than reading the source. If it isn't, cut it.
# Social Auth Facebook: agent index Facebook (Meta) OAuth2 login for Drupal, built on Social Auth / Social API. Adds `/user/login/facebook`, a settings form, and a Facebook button. - Settings config (client_id, client_secret, graph_version…) → configure/settings.md - Network plugin + FacebookAuthManager auth flow, Rules events → api/network-and-manager.md Key facts: - Config object `social_auth_facebook.settings`. No config/install: the object does not exist until the settings form is saved (or you create it). - Form route → /admin/config/social-api/social-auth/facebook - OAuth callback /user/login/facebook/callback (register it in the Meta app) - `graph_version` is stored without the leading `v`; must match e.g. `17.0`.
The highlighted gotchas are what an agent would otherwise have to dig out of the source, one file at a time.
Agent docs are the source.
Human docs are just a render.
shy_one_time/2.0.x/ ├── agent/start.md 28 lines ├── usage.md 27 lines └── data.json 54 lines
109 lines. Dense, structured, honest, grounded in the real code.
Video
A narrated walkthrough
PDF guide
Everything in one place
Game
Learn it by playing it
Podcast
Three hosts, one terrible module name
Your docs, your format, your language, generated on demand.
Write the knowledge once. Render it for every audience.
🎬 The video
Video coming soon
Expected at presentation/shy.mp4.
📄 The PDF: everything in one place
Open shy-one-time-guide.pdf ↗ · 4 pages · problem → how it works → setup → trade-offs
🎮 The game
Keyboard + mouse inside the game.
Use Next → to leave.
🎧 The podcast
Three hosts. One module. A lot of opinions about the name.
Voices generated with ElevenLabs from a script written from the agent docs.
Jess: Welcome back to Module of the Week! Today's module is my absolute favourite. It's called... Shy One-Time. Charlie: Shy... One-Time? Is that a dating app? Gunnar: It's a Drupal module, Charlie. Although the name does a great job of hiding that. Jess: Okay, picture this. A user forgets their password. They request a reset link, open the email, click it, and Drupal says: "You have tried to use a one-time login link that has either been used or is no longer valid." Charlie: But they didn't use it! They literally just clicked it! Gunnar: Correct. Somebody else clicked it first. The corporate mail gateway. The link-preview bot. A search crawler. They all follow the URL before the human even sees the email. Charlie: So a robot... is resetting my password? Gunnar: No. The robot is spending your password reset. Drupal marks the link as used, and the actual human gets nothing. Charlie: That's the most enterprise thing I've ever heard. … 4:49 · full script in derived/podcast/script.md
Evals: the long tail
Same task, three arms. easy = answer · medium = inspect the live site · hard = build it, verified against live site state.
Correct · Haiku 4.5
Time per task · Opus 4.8
Tokens per task · Opus 4.8
Precision goes up on long-tail modules, most for smaller models. Time drops ~36% with the skill and ~72% with docs in context. Cost: $0.19 → $0.15 → $0.09.
Honest bit: the skill arm spends tokens learning to find the docs. Handing over the right doc directly is what cuts tokens about 3×, and that's the MCP's job.
39 modules with run results (19 outside the top 100) · mostly 1 run per cell · Claude only · arms: vanilla / SKILL.md prepended / distilled docs pre-loaded
Frontier model + top 100 modules:
it already knows.
Correct · Opus 4.8 · top 100
Tokens per task
Cost per task
Pathauto, Token and Webform are all over the internet, so the model learned them in training. The docs add little except a bit of speed (36s → 31s).
Smaller models still benefit, even here: Haiku on hard top-100 builds went from 33% → 78% (only 3 cases, so treat as a hint).
Why does Opus know Pathauto?
Because the internet does.
Top 100 modules: years of blog posts, Stack Exchange answers, docs and issues, all in the training data.
The long tail: a README, and maybe a project page.
So we publish 24,642 agent docs for every Drupal 11 module, openly, on GitHub.
Today: context
Agents read the docs through the MCP.
Tomorrow: knowledge
Public, structured, grounded in real code. Exactly what the next training run can pick up.
Synthetic, yes, but generated from installing and reading the actual module. It is still better Drupal training data than nothing.
A DDEV add-on. GitHub takes the hit.
Corpus repo
agent-module-documentation on GitHub
GitHub Actions
indexer.py builds docs.db + server image
Release + GHCR
docs.db.gz (~200 MB)
container image
Your DDEV project
MCP over HTTP:9130/mcp
Your agent
Claude Code, Claude Desktop, …
drupal.org load: zero
No new infrastructure to approve or scale. Distribution runs on GitHub Releases and GHCR. Search runs on your laptop.
Fits how Drupal devs work
DDEV is already the default local stack. Install the add-on, restart, and register the URL with your agent.
Locked down
Non-root container, read-only DB, doc bodies served from the DB with no filesystem access. Exposed only through the DDEV router.
Lexical + semantic = hybrid
Lexical · BM25
SQLite FTS5. Matches the words you typed.
Great for graph_version, pathauto, exact names and config keys.
Misses “mail scanner eats my reset link”.
Semantic · vectors
sqlite-vec KNN over embeddings. Matches meaning.
“reset link already used in Outlook” lands near “Stop a mail scanner consuming a reset link”.
Fuzzy on exact identifiers.
Fused · RRF
Reciprocal Rank Fusion merges both lists:
score = Σ 1 / (60 + rank)
Everything is in one SQLite file, docs.db: metadata, doc bodies, FTS5 index and vectors. It's built once in CI from the corpus.
Model2Vec + sqlite-vec:
embeddings without a GPU
- Static embeddings distilled from a sentence transformer. Each token has a precomputed vector, and a text is their average.
- Model:
minishlab/potion-retrieval-32M, 512 dimensions, tuned for retrieval - No PyTorch, no ONNX, no GPU. Just numpy. Thousands of docs/second on a CPU.
- Model is baked into the image, so it runs fully offline
# requirements.txt (the whole runtime)
mcp==1.29.0
sqlite-vec==0.1.9
model2vec==0.9.0
numpy>=1.26
Build == query: the indexer and the server share embed.py, and the server refuses to start if docs.db was built with a different model.
It runs on any machine that runs DDEV: laptops, CI, old hardware.
The MCP server: vibe coded in ~4 hours.
The module docs are what's worth something.
Take it, build something better
- The dataset is open on GitHub: use it, fork it, index it your own way
- Hosted search, drupal.org integration, Project Browser, your own agent…
- The MCP server is ~600 lines of Python. Replace it freely, because the docs are the asset.
The DDEV idea might be bad
- One container + model + DB per project
- Memory and disk for every project you run
- ~200 MB download per project install
A shared global service or a hosted endpoint may be the better long-term shape.
Thank you
Questions?
Dataset
github.com/Drupal-AI/agent-module-documentation
DDEV MCP add-on
github.com/Drupal-AI/ddev-drupal-agent-module-knowledge-mcp
Browse the catalog
drupal-ai.github.io/agent-module-documentation
Module Finder
module-finder.marcusmailbox.com