Because the context it runs on rots. Anthropic's own data team measured exactly this on their internal self-service analytics: a month after launch, offline accuracy had fallen from about 95% to about 65%. Same model, same warehouse. The data model kept moving and the notes describing it did not.
The same experiment shows how much the context is worth in the first place: without it, accuracy on their own test questions did not exceed 21%. With it - folders of plain markdown describing what each table holds, the joins, and the traps a senior analyst would warn you about - it sits above 95% in aggregate. So nearly all of the accuracy lives in the documentation, which is also why the decay is so steep when the documentation goes stale: the agent is only as current as its notes.
What to do about it:
- Treat the semantic layer as maintained code, not a launch artifact. The definitions - what each table holds, which number is the governed one, the known traps - live in version-controlled markdown, and a schema change is not done until those files change with it.
- Re-run your evaluation questions on a schedule. The 95-to-65 drop was invisible until measured offline. A fixed set of questions with known answers, re-run weekly or monthly, is what turns silent decay into a number you can act on.
- Validate against the governed number. The hard parts of analytics did not change with AI - checking a figure against the official one, finding knowledge that sits in silos - the agent just makes it obvious where the hard part was all along.
Note this is a different failure from an agent seeming to get worse within one long session, which is about context-window degradation and has its own page. This one plays out over weeks and survives every fresh session, because the stale notes are reloaded every time. Related: common mistakes building text-to-SQL agents (the build-time half - share schema, business context and categorical distributions). Anthropic's two posts, read together with a practitioner's notes: https://www.tigzig.com/post/anthropic-data-analytics-sep2026.
Building something like this? How I work covers the rates, the availability and what I take on.