Observability and Operations (O)
This page corresponds to §7 of Agent Harness Engineering: A Survey. Observability is the first layer the taxonomy promotes from “lifecycle hook side effect” to first-class architectural concern.
Agent observability is not just logging model input and output. It is the operational record of an evolving control system.
Tracing And Monitoring
The foundation is structured trace capture: each LLM call, tool call, retrieval step, context assembly operation, retry, error, and state transition becomes inspectable as part of a span tree.
| Layer | Systems | Signal |
|---|---|---|
| Interactive trace platforms | Langfuse, Opik, Arize Phoenix, MLflow | Trace trees, token/cost breakdowns, latency flame graphs, prompt versions. |
| Instrumentation standards | OpenTelemetry, OpenLLMetry, OpenInference | Standard genAI spans and backend interoperability. |
| Deep/process-external monitoring | AgentSight, AgentTrace | eBPF/system-level observation and multi-plane logs that an agent cannot easily bypass. |
Process-external monitoring is especially relevant for compromised or misconfigured agents because the observed process cannot simply omit the telemetry.
Agent-Specific Operations
Agent operations platforms add concerns that generic LLM tracing misses:
- multi-step session identity;
- agent and role identity;
- tool-selection strategy;
- cross-session handoff;
- workflow-level state;
- harness-module contribution.
NLAH is particularly important because it treats the harness itself as an experimental object: modules such as tool registries, permission gates, hooks, skills, and context folding become ablatable interventions.
Cost Tracking And Optimization
Harnesses amplify cost because one task may trigger many model calls, each with assembled context and tool definitions. The survey separates tracking from optimization:
| Direction | Systems | Signal |
|---|---|---|
| Tracking | TensorZero, Helicone | Gateway-level cost and latency attribution. |
| Routing | FrugalGPT, QC-Opt | Cascades and quality-aware routing. |
| Caching | GPTCache | Semantic reuse for repeated or paraphrased requests. |
| Serving-side optimization | Dual-Pool Routing, TALE | Token-budget-aware pools and token elasticity. |
The warning is that cost optimization can silently change evaluation fidelity. Infrastructure settings alone can shift benchmark scores, so cost needs to be observed at API, application, and infrastructure levels.
Reliability Engineering
Long-running agents fail through more than exceptions. Anthropic identifies recurrent coding-agent failures: trying to complete the entire task in one pass, declaring completion too early, leaving damaged environments across sessions, and marking work done without tests.
The paper connects reliability to harness design:
- initializer agents that decompose work and create progress artifacts;
- coding agents that work incrementally and leave clean handoff state;
- planner/generator/evaluator separation;
- checkpoint/resume;
- component virtualization, where “brain,” “hands,” and “session” are independently recoverable resources.
Harness complexity should change with model capability. If a stronger model no longer needs a context reset or sprint contract, keeping that machinery may add cost without adding reliability.
Toward Unified Observability
The survey identifies a recurring gap: teams often use observability but do not connect it to offline evaluation. Observability shows what happened; evaluation judges whether it was correct.
Unified observability should close that loop:
- convert anomalous production traces into regression cases;
- compute trajectory metrics over spans;
- attribute failures across model, tool, context, sandbox, orchestration, judge, and policy layers;
- track which harness interventions still carry quality, safety, or reliability.
The long-term principle is harness-as-assumption: every reset, verifier, permission gate, context rule, or recovery loop encodes an assumption about model limitations. Observability should tell us when those assumptions stop being true.