# If you cannot replay it, you cannot ship it. > Agent observability means replaying any run end to end: prompt, context, tool calls, arguments, results, cost, latency. What to log and what to alert on. Canonical URL: https://techtideai.io/blog/ai-agent-observability Markdown URL: https://techtideai.io/blog/ai-agent-observability.md Publisher: TechTide AI, Columbus, Ohio (https://techtideai.io) Author: Alex Cinovoj, Founder & CTO, TechTide AI · CTO & Co-Founder, CPG Playbook Type: Field Note Published: 2026-04-22 Reading time: 9 min Topics: AI agent observability, LLM tracing, agent logging, AI monitoring Site index: https://techtideai.io/llms.txt Offer summary: https://techtideai.io/offers.md --- PLAYBOOK / 9 MIN READ ## If you cannot replay it, you cannot ship it. By [Alex Cinovoj](https://alexcinovoj.com/), Founder & CTO, [TechTide AI](https://techtideai.io/about) · CTO & Co-Founder, CPG Playbook Published April 22, 2026 A customer complains that the support agent gave them the wrong shipping estimate. Someone on the team opens the chat transcript, reads it twice, and shrugs, because the transcript shows what the agent said, not what it did to arrive there. That gap, between what the user saw and what actually happened inside the agent's run, is where most production incidents go unsolved. Observability for an agent means being able to replay any run end to end: the prompt, the context, every tool call and its arguments and results, the cost, and the latency, all tied together with one trace ID. Without that, debugging an agent is guesswork dressed up as engineering. ## Key takeaways - Agent observability means being able to replay a run end to end, including the prompt, context, tool calls, cost, and latency tied to one trace ID. - A chat transcript is not a trace, because it shows what the agent said, not what it did internally to arrive there. - A trace should capture the full prompt, every tool call and its arguments, retries and fallbacks, token cost, latency per step, and the final validated output. - Alert only on a small set of signals like tool call failure rate, cost per session, schema validation failures, and latency percentile breaches. - Traces and evals are two halves of the same discipline, since a production incident found through a trace should become a new eval case. ## A transcript is not a trace The most common mistake in agent observability is treating the chat log as the observability layer. A transcript captures the conversation between the user and the model. It does not capture what the model tried internally: which tool it called first, what arguments it passed, what came back, whether the first attempt failed validation and got retried, or what other tool it considered and rejected. Most production bugs live in exactly that gap. The model gave a wrong answer not because its language generation was bad, but because a tool call returned stale data, or a retry silently used a fallback value, or the context window truncated a critical instruction before the final generation step. None of that shows up in a transcript. It only shows up in a trace. ## What a trace actually needs to contain Treat every agent run as a distributed trace, the same discipline you'd apply to a microservices request. Each run needs: - **A unique trace ID** generated at the start of the run and threaded through every subsequent call, log line, and error. - **The full prompt sent to the model** at each step, including system instructions, not a summary of it. - **Every tool call**, with its name, arguments, raw result, and whether the result passed schema validation. - **Retries and fallbacks**, explicitly labeled as such, not silently merged into the trace as if they were the first attempt. - **Token count and cost** per model call, not just a total for the whole session. - **Latency per step**, so you can tell whether a slow response came from the model, a tool call, or a retry loop. - **The final validated output** and whether it matched the expected schema before being returned to the user. This is not exotic engineering. It is the same structured logging discipline that backend teams have applied to distributed systems for a decade, applied to a new kind of call. If you already have an observability stack like Datadog or Honeycomb, pipe agent traces into it rather than standing up a separate tool nobody checks. Further reading - [Build an agent eval suite in five days.](https://techtideai.io/blog/build-an-agent-eval-suite) Also on observability and evals · 10 min - [The harness matters more than the model now.](https://techtideai.io/blog/long-running-agent-harness) Also on observability and agents · 10 min ## A procedure for instrumenting an existing agent 1. **Generate a trace ID at the entry point** of every agent run and pass it through every function call, tool invocation, and log statement for that run. 2. **Wrap every tool call** in a logging layer that captures arguments, raw result, latency, and validation status before the result reaches the model. 3. **Log the full prompt** at each model call, including any dynamically inserted context, not a placeholder or a hash of it. 4. **Capture token usage and cost** from the provider's response metadata at every call, not estimated after the fact. 5. **Tag retries and fallbacks explicitly** in the trace so they are visually distinct from first attempts when someone reviews the run later. 6. **Ship traces to your existing observability stack** with the trace ID as the correlation key, so an agent trace sits next to your regular application logs for the same request. 7. **Build one dashboard** showing tool call failure rate, cost per session, and latency percentiles, refreshed in near real time. 8. **Test replay on a known incident** before calling the instrumentation done. If you cannot fully reconstruct a run you already know the outcome of, the instrumentation is incomplete. ## What to alert on, and what to just log Logging everything and alerting on everything are different disciplines, and conflating them is how teams end up ignoring their own alerts. Alert on the small number of signals that need a human within minutes: - **Tool call failure rate** above a set threshold within a rolling window, since a spike usually means an upstream dependency broke. - **Cost per session** exceeding your budget ceiling, which usually means a context or retry loop bug, not organic growth. - **Schema validation failure rate** on structured output, since this often precedes a customer-visible incorrect answer. - **Latency percentile breach** at the p95 or p99 level, which tells you about the experience of your worst-served users, not your average one. Everything else, the full trace detail, the token-level breakdowns, the individual tool arguments, belongs in logs for investigation, not in an alert channel. A team that pages on-call for every anomaly trains itself to mute the channel within a month, which defeats the entire purpose of alerting in the first place. ## The 2026 take on this, and where it is incomplete The popular argument this year is that observability tooling for agents has matured enough that teams can bolt on a third-party tracing product and consider the problem solved. Several genuinely good products exist now, and using one instead of building your own is usually the right call. But adopting a tool is not the same as having observability, and the gap between the two is where a lot of teams get a false sense of security. A tracing product records the events you instrument. If you record only final tool results, the trace may omit the arguments that caused a wrong answer. Define the information an investigator needs, then verify a failed run contains it. Buy the tool. Do not skip the design work of deciding what goes into the trace and what triggers an alert. That work is specific to your workflow and nobody can do it generically for you. ## Observability and evals are two halves of the same discipline Traces answer "what happened in this specific run." An eval suite answers "does the system still work correctly across our known cases after a change." You need both, and they feed each other: a production incident found through a trace should become a new eval case, so the same failure gets caught automatically before it ships again. An eval failure should point you to the specific trace of the run that failed, so you can see exactly which tool call or context truncation caused it. Teams that build one without the other end up either flying blind in production with a solid pre-launch test suite, or drowning in production detail with no systematic way to prevent a known failure from recurring. Both halves belong in the same review, and this is a standard part of what we check during a [pilot-to-production review](https://techtideai.io/how-to-move-ai-pilot-to-production). ## What good observability looks like a year in A year into a well-instrumented agent, a customer complaint or an internal question about a strange result should follow the same short path every time: pull the trace ID from the request log, open the full run, see the prompt, the tool calls, the arguments, the results, the cost, and the latency, and know within minutes whether the issue was a model reasoning error, a tool bug, a data staleness problem, or a context truncation. That single capability, being able to answer "what actually happened" quickly and confidently, is the difference between an AI system your team trusts and one everyone is quietly nervous about. If you are running agents in production today and cannot answer that question in under ten minutes for a given complaint, that is the gap to close first, ahead of new features. Trace coverage gives your team a way to investigate those failures. Our [agent orchestration guide](https://techtideai.io/ai-agent-orchestration-guide) covers where tracing fits alongside the other structural decisions that keep a multi-step agent debuggable at scale. ## Frequently asked - The full prompt sent to the model, every tool call with its arguments and results, the final response, token count and cost, latency per step, and a unique trace ID linking all of it together. If you cannot reconstruct exactly what happened from the logs alone, you cannot debug a production incident. About the author ## Alex Cinovoj AI & Infra Engineer. Building AI that survives outside the demo. Founder & CTO, TechTide AI · CTO & Co-Founder, CPG Playbook I spent 13 years in US enterprise IT, the last 2 focused entirely on AI implementation. I run [TechTide AI](https://techtideai.io/), an implementation studio that takes stalled AI pilots into production. I write about the work in progress at [alexcinovoj.com](https://alexcinovoj.com/) and on [Substack](https://alexcinovoj.substack.com/). Logs over vibes. Production over theater. [alexcinovoj.com →](https://alexcinovoj.com/)[More on the studio](https://techtideai.io/about) Related field notes - EXPLAINER · 10 min · Agents ### [Context engineering is the job now.](https://techtideai.io/blog/context-engineering-for-production-agents) Prompt engineering became context engineering: what the model sees, in what order, from which source, under what budget. Practical checks for retrieval, tool output and token budgets. - PLAYBOOK · 9 min · Evals ### [Pick a model with your own workload, not a leaderboard.](https://techtideai.io/blog/model-selection-without-benchmarks) Public benchmarks do not predict your workload. A two-day method for choosing a model with 50 of your own tasks, real cost per task, and a swap plan. [All field notes →](https://techtideai.io/blog) ## Have a stalled build that needs to work? [Send the project brief→](https://techtideai.io/contact)