In this blog post I articulate a product vision for continuously analyzing and optimizing production AI applications.

The distinction at the heart of this vision is between two problems:

  • Observability: What happened during an individual agent execution?
  • Optimization: Which behaviors, workflows, models, and memories produce the best business outcomes, and how should the application improve?

The vision is a closed-loop optimization architecture: operational state and telemetry become analytical intelligence, that intelligence informs governed changes, and the outcomes of those changes feed back into the next cycle.

This post develops my Agent Analytics and Optimization product vision document into a vendor-neutral vision: Open Agent Analytics and Optimization. It retains the architecture, principles, analytics pillars, and optimization maturity model while separating the required capabilities from the products used to implement them. It describes a product direction, not an announcement of a released product or a delivery roadmap.

The repository for the original document also provides a rough first cut at implementing this vision in the form of a full running sample and a full-day workshop to walk users through the concepts and build it themselves. That reference implementation uses specific products; the broader vision is not restricted to them.

Executive Summary

AI applications are rapidly evolving from single-prompt experiences into complex agentic systems composed of collaborating agents, tools, memories, workflows, and long-running stateful processes.

As organizations deploy agentic applications into production, they need more than debugging tools and execution traces. They need a way to understand how agents behave over time, how memories influence outcomes, how workflows evolve, where costs accumulate, and which patterns drive business value.

More importantly, they need a way to use that intelligence to continuously improve their agent systems.

Open Agent Analytics and Optimization is a product vision for enabling organizations to analyze telemetry, state, memory, workflow, evaluation, and business outcome data from agentic systems at enterprise scale and transform those insights into measurable improvements.

The vision is a framework- and vendor-neutral closed-loop optimization architecture. Compatible operational stores capture agent execution, state, and memory through framework integrations. A data lake or lakehouse with Spark-based compute transforms that data into analytical intelligence and optimization recommendations. Reverse ETL returns insights and proposed changes to the operational layer, where governed interfaces make them available to applications, agents, and presentation tools.

The architecture separates three responsibilities:

  • The operational layer is the system of record for application state, execution records, memory, and active optimization policies.
  • The analytical layer maintains historical evidence, evaluates outcomes, and generates optimization recommendations.
  • The governed activation layer approves and applies changes, records their effects, and supports validation and rollback.

Presentation is independent of these responsibilities. A dashboard, application interface, or agent tool can consume the same insights without owning the optimization loop.

Problem Statement

Execution-focused observability helps teams inspect and debug individual executions.

It answers questions such as:

  • Why did this execution fail?
  • What prompt was sent?
  • Which tool was called?
  • How many tokens were consumed?
  • What happened during a specific workflow execution?

These capabilities are essential for developers.

However, organizations increasingly need to answer higher-level questions:

  • Which agents deliver the highest business value?
  • Which workflows produce the best outcomes?
  • Which memories improve success rates?
  • Which memories are stale or ineffective?
  • How does agent behavior evolve over time?
  • What is the cost per successful outcome?
  • Which workflows should be optimized?
  • Which patterns correlate with business success?

Even more importantly:

  • How can these insights be used to improve agent behavior?
  • Which optimizations should be applied?
  • Which optimizations can be automated safely?
  • How can agent systems continuously improve over time?

These questions require analytics and a governed optimization loop beyond execution debugging alone. Observability remains an essential input to that loop.

Vision

Open Agent Analytics and Optimization aims to enable organizations to continuously improve agentic systems through data-driven optimization.

The architecture combines framework-integrated operational state with enterprise-scale lakehouse analytics and Spark compute to create a continuous learning loop for agent fleets.

Agents
   |
   v
Operational State, Execution Events, and Outcomes
   |
   v
Incremental Ingestion and Schema Normalization
   |
   v
Lakehouse Analytics and Spark Compute
   |
   v
Optimization Intelligence
   |
   v
Reverse ETL to the Operational Layer
   |
   v
Governed Activation
   |
   v
Agents

Subsequent executions produce new evidence about the effect of each change, closing the loop. Publishing a recommendation back to the operational layer does not activate it; activation is a separate, authorized decision.

This pattern is similar to optimization systems that have been successfully applied for years in recommendation engines, search ranking systems, advertising platforms, and machine learning systems.

Those systems continuously learn from outcomes and improve future decisions.

Open Agent Analytics and Optimization brings the same optimization model to agentic applications.

The goal is not simply to understand how agents behave.

The goal is to help agents become more effective over time.

Organizations should be able to continuously optimize:

  • Agent quality
  • Workflow efficiency
  • Memory effectiveness
  • Routing effectiveness
  • Tool utilization
  • Model selection
  • Cost efficiency
  • Business outcomes

through a combination of analytics, recommendations, governance, and automation.

Optimization Maturity Model

Open Agent Analytics and Optimization supports a progressive path from observation to autonomous optimization.

Level 1: Visibility

Organizations gain visibility into agent behavior through dashboards, reports, semantic models, and analytical experiences.

Examples include:

  • Agent performance analysis
  • Workflow bottleneck identification
  • Memory effectiveness analysis
  • Cost optimization opportunities
  • Quality and evaluation reporting

Humans identify opportunities and implement improvements.

Level 2: Recommendations

The platform generates recommendations based on observed behavior.

Examples include:

  • Routing recommendations
  • Memory recommendations
  • Cost recommendations
  • Evaluation recommendations
  • Workflow recommendations

Humans review and approve recommendations before implementation.

Level 3: Assisted Optimization

The platform automatically generates proposed changes and impact analysis.

Examples include:

  • Memory salience adjustments
  • Retrieval weighting recommendations
  • Routing threshold optimization
  • Model selection optimization
  • Tool usage optimization

Human reviewers approve or reject changes before deployment.

Level 4: Autonomous Optimization

For approved optimization domains, the platform can automatically apply and validate improvements.

Initial autonomous optimization scenarios focus on lower-risk domains such as:

  • Memory salience tuning
  • Memory retention policies
  • Retrieval weighting
  • Routing thresholds
  • Tool selection policies
  • Model selection policies

All changes remain measurable, reversible, and auditable.

The platform continuously measures outcomes and validates whether changes improve quality, latency, cost, or business outcomes.

Level 5: Adaptive Agent Systems

Organizations operate fleets of agents that continuously learn from operational outcomes.

Higher-risk optimizations remain human-governed, while lower-risk optimization domains can be continuously improved through automation.

The result is an ecosystem of agents that becomes more effective, more efficient, and more valuable over time.

Key Principles

Framework and Vendor Agnostic

Open Agent Analytics and Optimization defines a common model for agentic systems without requiring a particular orchestration framework, operational database, analytical platform, or presentation product.

Rather than coupling analytics and optimization to a specific orchestration framework, the platform normalizes telemetry, workflow, memory, evaluation, and agent interaction data into a common analytical model.

The intended framework integrations include:

  • LangGraph
  • Microsoft Agent Framework
  • OpenAI Agents SDK

Portability depends on explicit integration contracts:

  • Framework persistence adapters capture checkpoints and state.
  • Execution instrumentation captures model calls, tools, transitions, errors, usage, and outcomes.
  • Memory adapters expose supported retrieval, lifecycle, and update operations.
  • Ingestion and reverse-ETL adapters connect operational stores to analytical storage and compute.
  • OpenTelemetry and OpenInference provide telemetry conventions to align with where applicable.
  • Model Context Protocol (MCP) can expose memory and optimization operations as tools; it does not replace persistence or execution instrumentation.

Organizations should be able to adopt new frameworks without redesigning their analytical model. Each framework and storage combination must still demonstrate the required capture and writeback capabilities; compatibility with one framework does not establish compatibility with every other framework.

Operational State First

Modern agent systems generate far more than telemetry.

They generate persistent operational state including:

  • Checkpoints
  • Agent state
  • Workflow state
  • Long-term memory
  • Conversation history
  • Tool outputs
  • Evaluation artifacts
  • User context

Open Agent Analytics and Optimization treats this operational state as a first-class analytical asset.

Compatible operational stores provide durable state through framework integrations and make analytically useful records available through change data capture, replication, or incremental export. A checkpointer alone is not sufficient: persisted recovery state and analyzable execution history serve different purposes.

Rather than analyzing only what agents do, the platform analyzes what agents know, remember, learn, and persist over time.

Analytics First

Analytics serves as the foundation for optimization.

The platform is not intended to replace developer observability or debugging tools.

Developer-focused tools are optimized for inspecting traces, prompt executions, and workflow failures.

Open Agent Analytics and Optimization focuses on organizational intelligence, with the goal of enabling teams to analyze agent behavior across millions of executions, workflows, memories, evaluations, and users.

The platform answers questions such as:

  • Which agents deliver the highest business value?
  • Which workflows produce the best outcomes?
  • Which memories improve success rates?
  • Where are cost and latency concentrated?
  • How is quality evolving over time?

These insights become the basis for future optimization.

Human-Governed Optimization

Not all optimization opportunities carry the same level of risk.

The platform distinguishes between optimization domains that are appropriate for autonomous improvement and those that require human oversight.

Lower-risk optimization domains include:

  • Memory salience tuning
  • Retrieval optimization
  • Routing optimization
  • Tool selection optimization
  • Model selection optimization
  • Cost optimization

Higher-risk optimization domains include:

  • Prompt modifications
  • Workflow redesign
  • Agent instruction changes
  • Agent capability changes
  • Code generation
  • Production deployment changes

These higher-risk domains remain human-governed, with the analytical and governance layers providing recommendations, impact analysis, validation, and approval workflows.

Risk depends on the application. Model selection, retrieval changes, and memory retention are not inherently safe to automate. Each optimization domain requires explicit authorization, evaluation criteria, and operational safeguards.

Open Agent Analytics Schema

The vision proposes an Open Agent Analytics Schema representing the core execution primitives common across agent frameworks. This is a proposed common model, not an already-established industry standard.

The schema should align with versioned OpenTelemetry GenAI and OpenInference conventions where they apply, while defining the additional operational-state, memory-lifecycle, and outcome relationships needed for optimization.

Core entities include:

  • AgentRun
  • AgentStep
  • AgentTransition
  • ToolInvocation
  • MemoryEvent
  • Checkpoint
  • EvaluationResult
  • TokenUsage
  • UserSession
  • WorkflowExecution

Each source adapter maps framework-specific records into these primitives, preserving identifiers, timestamps, provenance, and schema versions. Relationships to business outcomes and the policy versions used during execution make comparisons meaningful.

By standardizing on these primitives rather than framework-specific objects, the architecture aims to provide consistent analytics and optimization across current and future agent ecosystems.

Architecture

Operational Data Layer

Agent frameworks use compatible operational databases through persistence and memory integrations. The operational layer remains the system of record for application state and the policies agents actually use.

The operational layer stores:

  • Agent checkpoints
  • Agent state
  • Thread and session history
  • Long-term memory
  • Memory salience metadata
  • Tool outputs
  • Evaluation results
  • Agent events
  • Workflow state
  • Business outcome records and their execution links
  • Published insights, proposed optimizations, and active policy versions
  • Approval and activation audit records

This operational data provides a rich analytical foundation beyond traditional telemetry.

Operational Integration Requirements

A database qualifies through its integration capabilities, not simply because it can store documents, vectors, or checkpoints. The required contract includes:

  1. Framework persistence: durable checkpoints and state for the relevant agents, including nested or delegated executions where the framework supports them.
  2. Execution capture: analytically usable records of model calls, tool invocations, transitions, retries, failures, timing, and token usage.
  3. Correlation: stable identifiers linking tenants, sessions, runs, steps, memories, evaluations, policy versions, and business outcomes.
  4. Analytical export: incremental ingestion with defined handling of updates, deletions, historical versions, and schema changes.
  5. Controlled writeback: supported interfaces for publishing insights and proposals and applying authorized policy or memory changes.

Capturing agent execution means capturing the framework-visible behavior and state needed to reconstruct and analyze an execution. It does not imply access to unobservable model internals or require retaining every sensitive payload.

Checkpoints may contain opaque serialized state useful for recovery but unsuitable for direct analysis. Framework instrumentation and adapters must expose the execution records and checkpoint metadata the analytical model needs.

Capture coverage must be explicit. Missing events, sampling, or unsupported memory operations should be visible rather than interpreted as zero activity or successful outcomes.

Analytical and Optimization Layer

The analytical layer consists of a data lake or lakehouse and Spark-based compute. Storage and compute are separate responsibilities and do not need to be provided by the same vendor.

Change data capture, replication, or incremental export brings operational records into the analytical layer. Source adapters normalize those records into the Open Agent Analytics Schema, where they can be joined with evaluations and enterprise business data.

Spark jobs compute historical trends, agent and workflow metrics, memory intelligence, and optimization recommendations. Analytical history must preserve the evidence required for these comparisons: a replica of the latest operational state alone does not provide a complete execution or memory-lifecycle history.

Agent Frameworks
  - LangGraph
  - Microsoft Agent Framework
  - OpenAI Agents SDK
  - Custom Frameworks
           |
           v
Framework Persistence and Instrumentation
           |
           v
Compatible Operational Stores
  - Checkpoints
  - Agent State
  - Memory
  - Conversations
  - Tool Outputs
  - Evaluations
  - Agent Events
  - Business Outcomes
           |
           v
Incremental Ingestion and Source Adapters
           |
           v
Data Lake / Lakehouse
           |
           v
Open Agent Analytics Schema
           |
           v
Spark Analytics & Optimization
           |
           v
Insights and Proposed Optimizations
           |
           v
Reverse ETL to Operational Serving and Policy Stores
           |
           v
Governed Approval and Activation
           |
           v
Agent Systems

Telemetry aligned with OpenTelemetry or OpenInference can enrich the same model through adapters. The ingestion path must define freshness expectations, deduplication, deletion handling, and access controls rather than assuming every source offers immediate, lossless mirroring.

Combining operational state with execution telemetry enables scenarios that are difficult to support with execution traces alone:

  • Memory effectiveness analysis
  • Memory lifecycle analytics
  • Checkpoint health analysis
  • Agent state evolution
  • Workflow replay analytics
  • Longitudinal behavior analysis
  • Optimization policy generation
  • Continuous agent improvement

Reverse ETL and Governed Activation

Reverse ETL publishes analytical outputs back to the operational layer, including:

  • Application-facing metrics and analytical views
  • Optimization opportunities and supporting evidence
  • Proposed routing, model-selection, retrieval, and memory policies
  • Evaluation results and measured effects of previous changes

These outputs belong in explicit serving and policy records, separate from the source execution history. Re-running a job should not duplicate recommendations, overwrite an active policy without authorization, or make derived records look like new agent executions.

Publishing a proposed policy does not change agent behavior. A governed activation service validates permissions, scope, evidence, and the expected current version before applying an approved change through the operational database or memory service’s supported interface.

The application then consumes the active policy during execution. Subsequent execution records identify the policy version used, allowing the analytical layer to measure its effect on quality, latency, cost, and business outcomes.

Changes must be auditable, with appropriate rollback or compensating actions. Reverting a policy cannot undo every external action or recover deleted data, so irreversible operations require additional safeguards.

Presentation-Independent Serving and Interaction

The optimization loop does not depend on a particular dashboard or BI tool. Applications can read published insights from the operational layer through authorized APIs, and analytical tools can query curated lakehouse data where appropriate.

Presentation surfaces can include:

  • Business intelligence reports and dashboards
  • Application-native analytics and optimization consoles
  • APIs and programmatic consumers
  • Agent-accessible tools, including MCP interfaces

Any surface that requests an optimization action must use the same governed backend contract. The surface presents evidence and captures a decision; it does not bypass approval, authorization, or audit requirements.

Power BI is one possible presentation and interaction layer, not a required component of the analytical engine or the writeback loop.

Agent Memory and Lifecycle Integration

The architecture supports memory services and toolkits that integrate with multi-agent applications and expose durable memories through a defined lifecycle contract.

Storage alone is not sufficient. A vector index or checkpoint adapter does not necessarily extract facts, reconcile conflicts, maintain summaries, or record how retrieved memories influence outcomes.

Depending on the implementation, memory services may manage conversation history, thread summaries, facts, episodes, procedures, user profiles, and other derived memory artifacts. To support the corresponding analytics and optimization scenarios, adapters need to expose:

  • Stable memory identifiers, types, ownership, and source provenance
  • Creation, update, supersession, expiration, and deletion events
  • Retrieval events linked to the requesting agent, run, and step
  • Available confidence, salience, retention, and retrieval-policy metadata
  • Supported operations for authorized memory and retrieval-policy changes

Not every memory implementation exposes every capability. An adapter should declare what it supports, and optimization features should require the relevant evidence and controls rather than inventing missing metrics.

This creates a foundation for advanced memory intelligence scenarios including:

  • Memory effectiveness analysis
  • Memory salience optimization
  • Memory lifecycle management
  • Memory decay analysis
  • Memory reuse analysis
  • Memory conflict detection
  • Memory-driven outcome analysis

Retrieval frequency is not proof of effectiveness. Outcome linkage and evaluation are needed to distinguish a frequently recalled memory from one that actually helps the application succeed.

There are several implementation paths. Redis Agent Memory provides session memory, background summarization and extraction, and searchable long-term memories. LangMem provides extraction, consolidation, and memory-management capabilities that can be composed with a database-backed LangGraph store, such as PostgreSQL. MongoDB’s LangGraph integration provides checkpointing and long-term memory storage and retrieval; those capabilities alone do not establish a complete derived-memory lifecycle.

These are alternative foundations, not claims of drop-in compatibility or equal optimization coverage. Each integration must demonstrate the required lifecycle visibility, outcome correlation, and supported writeback operations.

Reference Implementation: Cosmos DB and Fabric

The travel multi-agent workshop provides an initial implementation of the loop using Azure Cosmos DB, Microsoft Fabric, and application and Power BI presentation surfaces.

Its concrete components illustrate the responsibilities above:

  • Operational persistence: the application uses the Cosmos DB integration for LangGraph checkpointing and adds explicit execution instrumentation for analytics.
  • Memory lifecycle: the Azure Cosmos DB Agent Memory Toolkit provides durable conversation history and derived memories, including summaries, facts, and user profiles.
  • Analytical ingestion and compute: Fabric mirroring makes operational data available to analytical processing, and Spark notebooks compute insights and optimization outputs.
  • Reverse ETL: analytical outputs are written back to Cosmos DB, including records in OptimizationInsights.
  • Governed activation: operational policy records distinguish proposed changes from active policies that influence application behavior.
  • Presentation: the Analytics Portal and Power BI provide interfaces over the resulting insights; neither defines the vendor-neutral architecture.

The analytics implementation demonstrates the pattern, not every capability in the long-term vision. The current notebook, authentication, deployment, ingestion, and writeback code includes Fabric- and Cosmos-specific dependencies.

Using a different operational database or Spark environment requires compatible adapters and validation. The architecture is intended to be portable; the existing sample is not presented as a drop-in implementation for every vendor.

Analytics Pillars

Pillar 1: Agent Performance Analytics

Metrics include:

  • Agent latency
  • Success rates
  • Failure rates
  • Throughput
  • Retry behavior
  • Queue time
  • Workflow duration

Example questions:

  • Which agent is the bottleneck?
  • Which workflow step causes most failures?
  • How has latency changed over time?

Pillar 2: Agent Collaboration Analytics

Metrics include:

  • Agent-to-agent transitions
  • Routing frequency
  • Routing confidence
  • Collaboration patterns
  • Cyclic routing
  • Handoff success rates

Example questions:

  • Which agent pairings are most successful?
  • Which routing decisions increase cost?
  • Which collaboration patterns produce the best outcomes?

Pillar 3: Cost Intelligence

Metrics include:

  • Token consumption
  • Model consumption
  • Cost per workflow
  • Cost per user
  • Cost per agent
  • Cost per successful outcome

Example questions:

  • Which workflows are most expensive?
  • Which agents deliver the highest ROI?
  • Which models drive the highest costs?

Pillar 4: Memory Intelligence

Memory is one of the most valuable and least understood components of agentic systems.

Metrics include:

  • Memory creation
  • Retrieval frequency
  • Reuse frequency
  • Memory aging
  • Memory decay
  • Memory salience
  • Memory conflicts
  • Memory effectiveness

Example questions:

  • Which memories are never used?
  • Which memories contribute most to successful outcomes?
  • What is the effective half-life of memory?
  • Which memories should be consolidated or removed?

Pillar 5: Evaluation Intelligence

Metrics include:

  • Correctness
  • Groundedness
  • Relevance
  • Helpfulness
  • User satisfaction
  • Human review scores
  • Automated evaluator scores

Example questions:

  • Which agents provide the highest quality responses?
  • Has quality regressed after deployment?
  • Which workflows perform best?

Pillar 6: Workflow Intelligence

Metrics include:

  • Workflow completion rates
  • Abandonment rates
  • Workflow duration
  • Outcome quality
  • Workflow cost
  • User success rates

Example questions:

  • Which workflows create the most business value?
  • Which workflows should be redesigned?
  • Which workflows fail most frequently?

Differentiation

Beyond Execution-Centric Analysis

Execution-centric tools provide essential capabilities:

  • Trace inspection
  • Prompt debugging
  • Execution monitoring
  • Evaluation execution

These capabilities are valuable, and products may also span evaluation, experimentation, and optimization. The distinction here is not an exclusive market category; it is the scope of the feedback loop.

Open Architecture Differentiation

Open Agent Analytics and Optimization brings together:

  • Enterprise-scale analytics
  • Historical trend analysis
  • Unified lake architecture
  • Presentation-independent serving and interaction
  • Cross-framework reporting
  • Explicit operational-database and memory integration contracts
  • Agent collaboration analytics
  • Advanced memory intelligence
  • Correlation with enterprise business data
  • Operational-state-aware analytics
  • Recommendation-driven optimization
  • Reverse ETL of insights and proposed changes into the operational layer
  • Human-governed optimization workflows
  • Progressive autonomous optimization

The platform answers not only:

“What happened during this execution?”

but also:

“What should we do next?”

and ultimately:

“How can agent systems continuously improve over time?”

Long-Term Outcome

The future of agent operations will look increasingly similar to the evolution of recommendation systems, search relevance systems, advertising optimization systems, and machine learning platforms.

These systems continuously learn from outcomes and improve future decisions.

Agent systems will follow the same pattern.

Compatible operational stores provide the durable foundation for stateful agent systems through framework persistence, execution instrumentation, and memory integrations.

A lakehouse and Spark-based analytical layer provide the intelligence to understand behavior and identify improvements. Reverse ETL and governed activation connect that intelligence back to the applications that can act on it.

The ultimate goal of Open Agent Analytics and Optimization is an interoperable architecture in which agent systems not only report their behavior, but improve from measured outcomes, without binding that loop to one database, analytical platform, or presentation product.

Agent telemetry, memory, workflow state, evaluations, and business outcomes become the foundation for continuous improvement, enabling agents to become more effective, more efficient, and more valuable with every execution.