AWS ships CloudWatch Omni to observe and evaluate AI agents

September 22, 2026. Amazon CloudWatch launches Omni, a unified observability experience for applications and AI workloads. September 22, 2026. AWS positions it as app-centric, AI-powered, built on open standards, and delivered off-console. September 22, 2026. The service is announced as free and usable without an AWS account. Why it matters: for the first time, AWS moves agent observability into the developer’s IDE, with an evaluation engine built in — rather than a monitoring console you consult after the fact.

What CloudWatch Omni changes

The launch post opens on a problem AWS states bluntly: agent behavior is non-deterministic. A prompt change can degrade response quality while classic metrics — latency, error rate — stay green. Teams burn hours manually cross-referencing logs spread across systems, unable to pin down what changed or why.

CloudWatch Omni responds by capturing every trace and pairing it with built-in evaluators that score correctness, coherence, retrieval quality, and tool selection, among others. That is the heart of the product: turning agent observability into a measure of quality, not just availability.

The positioning is deliberately cross-cutting. Omni works with any model provider, framework, or runtime, and leans on OpenTelemetry for instrumentation. “Off-console” is not marketing fluff — it means the tool lives where developers already work, not inside the AWS Management Console.

Two surfaces for two jobs

The product ships as two complementary surfaces. Developers get a native extension for VS Code and Kiro — the two IDEs currently supported — where traces appear as the agent runs, with a playground and evaluators one click away. Operators get a standalone web experience, separate from the AWS console, reachable through SSO with no console account needed.

Both surfaces share the same data: the trace a developer debugs is exactly the trace an operator investigates in production. That unification is rarer than it sounds — most gen-AI observability tools still force a choice between a developer-only silo and an ops-facing dashboard.

The Cloud Login feature then connects the local environment to the AWS account, sending telemetry to CloudWatch, sharing traces with the team, and opening production dashboards. The connection is optional: you can run Omni fully locally during development, then link to the cloud when it is time to monitor agents in production.

Evaluation becomes the reference metric

The most consequential part of the product is its eval-driven workflow. Omni ships 17 built-in evaluators covering dimensions such as coherence, helpfulness, faithfulness, and routing correctness. You select traces in the Trace Explorer, pick evaluators, and get per-example scores and aggregate metrics — no homegrown evaluation framework required.

The Trace Explorer renders a hierarchical timeline of every agent step: LLM calls, tool invocations, reasoning steps. You can drill into any span to inspect inputs, outputs, token usage, and latency. Compare mode places two traces side by side to isolate the effect of a prompt change, and the Ask Assistant feature has an agent analyze anomalies for you — answering questions like “why did the agent call this tool twice?”

Above the traces, three pieces close the engineering loop. The Playground compares prompts and models in real time. The Experiments view runs the same dataset against two agent variants and compares scores, latency, and token usage. Prompt Management versions configurations and enables rollback. The Session Explorer replays multi-turn conversations, and Agent Topology visualizes the agent’s architecture — sub-agents, tools, and interconnections.

# Run a local GGUF model through transformers serve (OpenAI-compatible API)
# then point CloudWatch Omni at the same endpoint to trace it
pip install -U "transformers[serving] @ git+https://github.com/huggingface/transformers.git" kernels
transformers serve "unsloth/Qwen3.5-4B-GGUF:Qwen3.5-4B-Q4_K_M.gguf"

The golden dataset mechanic is what separates the product from a trace viewer: you freeze a reference corpus, replay the agent, and any prompt regression raises a quantified alert instead of a hunch.

Open instrumentation and a free model

Omni plugs into the agent frameworks teams already run: LangChain, LangGraph, CrewAI, the OpenAI SDK, Strands, and the Vercel AI SDK, in both Python and TypeScript. Instrumentation has two paths: auto-instrumentation via Kiro, which detects the framework and configures tracing, or manual instrumentation with ready-made snippets.

The product also leans on coding assistants — Kiro, Claude Code, Codex — to configure the dev server, install dependencies, and set up instrumentation on your behalf. The stated goal is to go from install to the first traced agent session in minutes.

The business model is worth pausing on: CloudWatch Omni is free. No AWS account is required to start — only Bedrock credentials or API keys for OpenAI or Anthropic if you want hosted models. For a product this rich, that signals less generosity than an adoption play: capture the agent developer before a competitor does, then monetize CloudWatch telemetry and adjacent services. The result is a product that feels less like a monitoring add-on and more like a developer tool that happens to emit telemetry — a deliberate category shift.

The competition and the off-console bet

The launch lands in an already crowded market. On agent evaluation and tracing, LangSmith, Langfuse, and Braintrust own the niche. On classic observability, Datadog, New Relic, and Grafana Cloud have all bolted an LLM module onto their dashboards. AWS’s answer is two structural choices: fold evaluation into observability rather than keep it a separate silo, and deliver the tool inside the IDE rather than a distant dashboard.

The off-console bet is the boldest part. Historically, observability was built for operators — consoles you consult after the fact. By putting tracing at the center of VS Code and Kiro, AWS targets the moment agent quality is decided, during development, rather than the moment it is noticed, in production. If the bet pays off, agent observability becomes a step in the development cycle, on par with linting and tests — and the console is reserved for fleet supervision.

Consider a concrete case. A team rewrites the system prompt to fix a format bias, and the correct tool-selection rate drops from 92% to 71%. Without Omni, the regression only surfaces in user tickets weeks later. With a golden dataset, it is caught before deploy, with the exact score that moved and the trace pointing at the faulty prompt. That loop — prompt, trace, evaluation, rollback — is what agent teams were missing.

Verdict

CloudWatch Omni moves the front line of agent observability: it no longer lives in a console you check after an incident, but in the IDE, at the moment the agent runs and quality is decided. If you build agents on AWS, or with LangChain/CrewAI in Python, try it now — free pricing and OpenTelemetry support make it a credible replacement for fragmented stacks like LangSmith plus a homegrown dashboard. If you are outside AWS, take the mechanics of its 17 built-in evaluators and golden datasets as the standard your own tools must match. Either way, treat a prompt regression as a production incident, with a quantified metric — not a feeling.

References

• AWS News Blog — Introducing Amazon CloudWatch Omni: AI-powered observability for generative AI and agentic workloads (September 22, 2026)
• AWS News Blog — Introducing Amazon CloudWatch Omni: collaborative AI-powered observability for your applications (September 22, 2026)
• Amazon CloudWatch Omni documentation