AI agents and agentic workflows are no longer experimental - they're becoming core infrastructure. For platform teams, that shift brings an urgent challenge: telemetry volume is growing faster than budgets, faster than tooling, and faster than most architectures were designed to handle. Understanding how to build an AI telemetry platform architecture that scales without spiraling costs is now a foundational platform engineering concern.

Main insights

• AI agents generate 2-5x more telemetry per transaction than human-driven workflows, pushing annual telemetry growth from 30% to 40% or more.
• Both solution-first and platform-first approaches create inefficiencies - data duplication, vendor lock-in, and integration burden are common failure modes.
• A modern telemetry platform must deliver four core capabilities: data quality, elastic scale, context fusion, and governance.
• Data tiering strategies that align storage cost with business value are essential for managing full-fidelity telemetry at scale.

This session was led by James Conway, Field CTO at Cribl, and Bill Emmett, Senior Director of Technical Product Marketing at Cribl. Both brought practitioner experience and strategic perspective to the challenge of telemetry management in the agentic era.

You can watch the full discussion here if you missed it.

Why telemetry economics are breaking

Observability data has historically grown at roughly 30% year over year - already a significant challenge when budgets remain flat. AI is accelerating that curve in ways traditional approaches weren't built to absorb.

The core issue is that AI agents aren't just consumers of telemetry; they're prolific producers of it. A single agentic prompt can spawn multiple sub-agents, each running tens of iterative queries. That machine-to-machine query pattern looks nothing like the human-driven traffic most pricing models assume.

As James put it: "The economics of telemetry management will start to break far before the data stops growing, and if the only approach is to simply pay more as that data increases, that approach is going to break very quickly."

The numbers make the pressure concrete:

• Telemetry growth has increased from 30% to 40% year over year with AI adoption
• Agentic workflows can emit 2-5x more telemetry than traditional applications
• Most vendor pricing models are calibrated for human users, not machine-to-machine query volumes

The platform engineering gap

Platform teams want to own telemetry management as a service - absorbing complexity so developers don't have to. But automation has lagged behind that goal. Research cited in the session found that 39% of platform engineers report that instrumentation and configuration of telemetry management adds 24 hours or more to delivery timelines.

That gap stems from how the market has historically structured telemetry tooling. James outlined two dominant approaches, each with real drawbacks:

Solution-first approach: Opinionated products like Datadog, Dynatrace, or Microsoft offerings provide prescriptive workflows but create data silos. "Each of these solutions often comes with its own data engine, its own storage model, its own compute layer, a different pricing model and proprietary workflow or proprietary formats," James noted. "So the same telemetry is being collected, shipped, indexed and ultimately paid for multiple times."

Platform-first approach: Building-block providers like Databricks offer flexibility but shift the integration burden onto platform teams. "The challenge then is that these platform teams become the integrators and the operators and really are responsible for building new solutions out of cloth," James explained.

Most organizations land in a hybrid: a handful of expensive solution-first products stitched together with DIY layers. The result is the swivel chair problem - teams pivot between disconnected tools to assemble a complete picture, slowing incident response and increasing cognitive load.

Four capabilities for AI-ready telemetry

To handle agentic workloads without breaking the budget, your telemetry platform architecture must deliver four capabilities:

Data quality: Garbage in, garbage out still applies - especially with AI. "Your agents are only going to be as reliable and productive as the data that they're consuming," James emphasized. Pre-processing data, normalizing to consistent schemas, and enriching with metadata reduces token waste and makes agent outputs more reliable.

Elastic scale: Agents can burst to large numbers of parallel queries as they spawn sub-agents. Your ingestion and processing layers must autoscale and prioritize critical signals so high-volume bursts don't overwhelm the pipeline.

Context fusion: Telemetry explains what happened; context explains why. Operational context - tickets, runbooks, commits, chat history - must be available alongside traces and logs so neither agents nor humans need to manually stitch data together during an incident.

Governance: Autonomous agents taking autonomous actions require traceable, auditable trails. This means role-based authorization, in-flight redaction of sensitive data, and immutable audit logs that capture agent access, decisions, and data lineage.

A practical checklist for evaluating your telemetry pipeline

Use these six areas to assess your current architecture or evaluate new tooling:

1. Collection and connection: Collect as close to the source as possible; avoid proprietary formats; capture human context alongside machine telemetry.
2. Process and transform: Normalize data in flight; convert high-volume logs to metrics where appropriate; use AI-assisted parsers to reduce manual configuration.
3. Deliver: Collect once, route to any destination, driven by configuration rather than custom code.
4. Security and governance: Redact PII in flight; enforce agent-level permissions before data reaches storage.
5. Capture and comply: Audit agent activity; maintain lineage records and enforce retention policies automatically.
6. Search and access: Enable federated search across tiers; expose open APIs; support search-in-place to avoid unnecessary data movement.

Data tiering: Aligning storage cost with business value

Not all telemetry deserves the same storage tier. A tiered approach maps use cases to cost-appropriate storage:

• Hot (detection): Recent, high-frequency data requiring sub-second query response
• Warm (investigation): Data accessed during active incident response, typically days to weeks old
• Cool (compliance): Data retained for regulatory requirements, infrequently queried
• Cold (archive): Long-term retention at minimal cost, rarely accessed

Define policies that move data through tiers automatically based on access patterns and retention requirements. This is where significant cost reduction is achievable - teams using this approach have reported storage cost reductions of 40-60%, alongside roughly 60% reduction in collection and delivery effort through AI-assisted pipelines.

Building a modular telemetry platform

The architectural pattern that emerges from these requirements is modular by design: edge shaping to reduce volume at the source, stream pipelines for in-flight reduction and enrichment, a cost-efficient data lake for full-fidelity retention, and a lakehouse layer for fast indexed queries. Applications and workflows sit on top, giving teams consistent experiences without locking into a single vendor's data model.

Model routing is an emerging lever worth watching - routing telemetry data to the best-fit inference model per workload type can meaningfully reduce inference costs as AI-assisted analysis becomes standard practice.

​

If you enjoyed this, find more great insights and events from our Platform Engineering Community.

For more comprehensive guidance, check out the Platform Engineering Certified Architect Course and learn best practices from industry experts.

​

Key takeaways

• Design for 10x telemetry growth now: Agentic workflows generate 2-5x more telemetry than traditional applications. Elastic scale, pre-processing, and tiered storage are not future concerns - they're current requirements for any team adopting AI agents.
• Implement data tiering to control costs: Mapping detection, investigation, compliance, and archive use cases to appropriate storage tiers can reduce storage costs by 40-60%. Automate tier transitions based on access patterns so cost management doesn't require manual intervention.
• Normalize and enrich data in flight: Consistent schemas and enriched metadata reduce token waste in AI pipelines and make agent outputs more reliable. Pre-processing at the pipeline layer is more efficient than cleaning data after it reaches storage.
• Make governance a first-class capability: Autonomous agents require traceable, auditable behavior. Role-based authorization, in-flight PII redaction, and immutable audit logs must be built into your telemetry architecture from the start - not retrofitted after an incident.