Your Agent's Rules Are Probably Wrong

Here’s a failure mode nobody talks about: you build a set of agent skills that encode your team’s conventions. The skills are accurate on day one. Then engineers merge PRs. Patterns evolve. Someone refactors a service, renames a namespace, deprecates a package.

Nobody updates the skills.

Now your agents are confidently teaching patterns from yesterday’s codebase. Every new story picks up the old conventions. The skill is a knowledge amplifier — and it’s amplifying stale knowledge.

We ran a validation script on our agent framework for the first time and found:

• 10 phantom references across 6 skills — code examples that referenced types, folders, and packages that don’t exist in the codebase. They looked plausible. They compiled in your head. They were completely fabricated.
• 1 wrong security rule — a skill that instructed agents to return HTTP 404 for authorization failures. The correct behavior (masking 403 as 404 to prevent resource existence leakage) was only supposed to apply to 403, not 401. Every agent would have propagated that error to every story it touched.

These were in production skills. Real agents were using them.

Why this happens

Skill content is written by humans based on the codebase at a point in time. But the codebase keeps moving:

• A type gets renamed → the skill’s code example now references a non-existent type
• A folder structure changes → the skill’s path conventions are wrong
• A pattern gets deprecated → the skill still teaches it as best practice
• A security rule gets refined → the old rule is still in the skill

None of this triggers a PR review. Skills live in markdown files. They change less frequently than code. They feel stable. That stability is an illusion.

The closed feedback loop

The fix is a feedback loop that connects codebase changes to skill validity.

Codebase changes
  → scan-patterns.ps1 (counts actual usage)
  → Pattern registry updated (with counts + dates)
  → Skill impact warnings emitted (if deprecated patterns still taught)
  → Engineer updates affected skills

Three checks make this work:

Pattern scanning — a script counts how many times each tracked pattern appears in the codebase. If a “Migrating” pattern drops to zero, it’s ready to be marked “Deprecated”. If an “Active” pattern count drops unexpectedly, something changed.

Example validation — a script checks that every type name, folder path, and package mentioned in a skill’s code examples actually exists in the codebase. This is what caught our 10 phantom references.

Skill cross-reference — the registry links patterns to the skills that teach them. When a pattern is deprecated, the registry surfaces exactly which skills need updating. No manual archaeology required.

The principle behind it

Skills are a knowledge amplifier. That’s their entire value — you write a pattern once and every agent benefits from it, forever.

But amplifiers don’t discriminate. They amplify what’s accurate and what’s wrong with equal fidelity.

Review your skill content like you review production code. Put it in your PR review process. Run validation on every merge. Treat a phantom reference as a bug, not a documentation issue.

The cost of a wrong skill isn’t a single bad response. It’s every story touched by that skill until someone notices.