Why does my AI data agent get less accurate over time?

Because the context it runs on rots. Anthropic's own data team measured exactly this on their internal self-service analytics: a month after launch, offline accuracy had fallen from about 95% to about 65%. Same model, same warehouse. The data model kept moving and the notes describing it did not.

The same experiment shows how much the context is worth in the first place: without it, accuracy on their own test questions did not exceed 21%. With it - folders of plain markdown describing what each table holds, the joins, and the traps a senior analyst would warn you about - it sits above 95% in aggregate. So nearly all of the accuracy lives in the documentation, which is also why the decay is so steep when the documentation goes stale: the agent is only as current as its notes.

What to do about it:

• Treat the semantic layer as maintained code, not a launch artifact. The definitions - what each table holds, which number is the governed one, the known traps - live in version-controlled markdown, and a schema change is not done until those files change with it.
• Re-run your evaluation questions on a schedule. The 95-to-65 drop was invisible until measured offline. A fixed set of questions with known answers, re-run weekly or monthly, is what turns silent decay into a number you can act on.
• Validate against the governed number. The hard parts of analytics did not change with AI - checking a figure against the official one, finding knowledge that sits in silos - the agent just makes it obvious where the hard part was all along.

Note this is a different failure from an agent seeming to get worse within one long session, which is about context-window degradation and has its own page. This one plays out over weeks and survives every fresh session, because the stale notes are reloaded every time. Related: common mistakes building text-to-SQL agents (the build-time half - share schema, business context and categorical distributions). Anthropic's two posts, read together with a practitioner's notes: https://www.tigzig.com/post/anthropic-data-analytics-sep2026.