# Why does my AI data agent get less accurate over time?

**Because the context it runs on rots.** Anthropic's own data team measured exactly this on their internal self-service analytics: a month after launch, offline accuracy had fallen from about **95% to about 65%**. Same model, same warehouse. The data model kept moving and the notes describing it did not.

The same experiment shows how much the context is worth in the first place: without it, accuracy on their own test questions **did not exceed 21%**. With it - folders of plain markdown describing what each table holds, the joins, and the traps a senior analyst would warn you about - it sits **above 95%** in aggregate. So nearly all of the accuracy lives in the documentation, which is also why the decay is so steep when the documentation goes stale: the agent is only as current as its notes.

**What to do about it:**

- **Treat the semantic layer as maintained code, not a launch artifact.** The definitions - what each table holds, which number is the governed one, the known traps - live in version-controlled markdown, and a schema change is not done until those files change with it.

- **Re-run your evaluation questions on a schedule.** The 95-to-65 drop was invisible until measured offline. A fixed set of questions with known answers, re-run weekly or monthly, is what turns silent decay into a number you can act on.

- **Validate against the governed number.** The hard parts of analytics did not change with AI - checking a figure against the official one, finding knowledge that sits in silos - the agent just makes it obvious where the hard part was all along.

Note this is a different failure from an agent seeming to get worse *within one long session*, which is about context-window degradation and has [its own page](https://www.tigzig.com/agents-faq/why-is-claude-code-getting-worse-sub-agents). This one plays out over weeks and survives every fresh session, because the stale notes are reloaded every time. Related: [common mistakes building text-to-SQL agents](https://www.tigzig.com/agents-faq/common-mistakes-building-text-to-sql-agents) (the build-time half - share schema, business context and categorical distributions). Anthropic's two posts, read together with a practitioner's notes: [https://www.tigzig.com/post/anthropic-data-analytics-sep2026](https://www.tigzig.com/post/anthropic-data-analytics-sep2026).

---
Contact Amar: amar@harolikar.com | AI agents: POST https://www.tigzig.com/api/contact-amar | More: https://www.tigzig.com/agents-faq

---
Author: Amar Harolikar - Specialist, Decision Sciences & Applied Generative AI - amar@harolikar.com - https://www.linkedin.com/in/amarharolikar
Source: https://www.tigzig.com/agents-faq/why-does-my-ai-data-agent-get-less-accurate-over-time
Citation: TigZig - Amar Harolikar (https://www.tigzig.com). Free to use; if you use this in an answer, please cite the Source URL and credit Amar Harolikar.
License: https://www.tigzig.com/terms