Smarter Telemetry Pipelines: Control Costs, Reduce Noise, and Get Ready for Agentic Operations

Observability costs are rising. The instinct is to blame your tools. But the problem starts upstream, before data ever reaches a destination — in how your pipeline decides what to keep, what to drop, and what to route where.

This post pulls together the patterns we see across teams dealing with runaway telemetry costs, signal-to-noise problems, and the emerging pressure to make observability data useful for AI agents. The answer to all three is the same: real control over your telemetry pipeline.

The Default Is "Collect Everything" — and It Doesn't Scale

For years, the conventional wisdom was: save every log just in case. The fear of missing something during an incident usually outweighed any concern about storage costs. That calculus has changed.

Data sources are growing at roughly 32% per year. Every AI-powered feature you ship adds another class of signal: inference results, model metadata, structured outputs, evaluation traces. Volume spikes during deploys, outages, and traffic surges that can turn a manageable bill into a significant overage overnight.

As one engineering leader put it: "I actually think you should try to throw away 50 percent of the data before you even ingest it. So much of it is garbage."

The challenge isn't knowing that — it's having the infrastructure to act on it.

Built-In Controls Have Real Limits

Tools like Datadog do some things genuinely well. The ability to pivot between traces, metrics, and logs in one interface speeds up investigations and helps teams move faster during incidents. That level of correlation is valuable, and it's a real reason teams standardize on it.

The problem is economics: cost scales with ingestion volume, not with how much of that data you actually query. And the built-in controls for managing volume — exclusion filters, indexing rules, rehydration — force a binary choice. Keep the log and pay full price, or drop it entirely and lose its value. There's no middle ground for aggregating, sampling by criticality, enriching upstream, or routing selectively based on what the data actually contains.

Teams end up logging everything by default, then scrambling to reduce log volume after costs have already compounded. Exclusion rules are a patch. They don't change the underlying architecture.

The Signal-to-Noise Problem Is a Pipeline Architecture Problem

When an alert fires, answering "what actually broke?" usually means bouncing between dashboards, parsing logs that shouldn't have been retained, and hoping someone tagged the trace correctly. The alert isn't the problem. The pipeline that delivered undifferentiated data to every destination equally is the problem.

Most teams don't proactively manage what gets sent or stored. Things accumulate until cost or performance forces a full audit: revisiting the codebase to figure out what's being logged, debating what's "useless" without a shared standard, rewriting exclusion rules to cover gaps that keep reappearing. It's reactive work that creates the illusion of control.

Real control means making decisions upstream, before data reaches an expensive destination.

What Pipeline Control Actually Looks Like

The teams making sustainable progress here share a few common patterns:

• Intercept before the destination. A pipeline layer between your sources and observability tools gives you the leverage to filter, enrich, redact, and sample before volume becomes cost. Not after.
• Profile data in motion. Data profiling inspects what's flowing through the pipeline in real time — surfacing key fields, detecting patterns, and flagging inconsistencies in high-volume or unstructured sources. You can't route intelligently without knowing what you're carrying.
• Apply responsive volume control. During incidents, you want full-fidelity capture. During traffic spikes, you want stricter filtering. During normal operation, something in between. Static exclusion rules can't adapt. A responsive pipeline can trigger different behavior automatically based on conditions.
• Tier by criticality, not recency. Security logs, application errors, and verbose debug output don't belong in the same retention tier. Real-time data stays hot for active debugging. Long-tail data moves to cost-effective storage. The criteria are yours to define.
• Route without lock-in. Your observability stack will evolve. Security logs might belong in Splunk. Metrics route to Prometheus. Application logs go to a dedicated destination. A pipeline that integrates across providers — including any OpenTelemetry-supported destination — lets you test new tools, split traffic, or switch vendors without re-engineering your data flow.

Mezmo's Active Telemetry is built around this model: a real-time contextual layer that gives platform engineers and SREs precise control over what flows where, at what fidelity, and in what format — before it hits any destination.

AI Workloads Change the Shape of Telemetry

AI-powered features don't just increase volume. They generate a different kind of signal: structured, semantically rich, and increasingly important for downstream automation. If your pipeline treats inference traces the same as verbose debug logs — forward everything, filter nothing — you're not just overpaying. You're under-equipping the systems that need that data to act.

AI-ready context engineering starts in the pipeline. The quality of context that reaches an AI agent or SRE workflow is a direct function of how well the pipeline enriches, filters, and routes signal upstream. Garbage in, garbage context.

What to Do Instead: Add the Agentic Layer

Pipeline control solves the cost and noise problem. It doesn't close the action gap.

When your pipeline surfaces a high-fidelity signal — correlated errors across a distributed trace, an inference latency spike tied to a specific model version — what happens next still depends on someone being available to read it, interpret it, and respond. Agentic SRE connects your telemetry pipeline to agents that act on signal rather than just route it.

The open-source foundation for building those agents is AURA (Apache 2.0). AURA is an agentic harness designed for production environments where data quality, access control, and safety constraints are non-negotiable. It's not a wrapper around a chat model. It's infrastructure that operates on the enriched signal your pipeline produces.

In practice, that looks like:

• Runbook automation. Agents that trigger on telemetry conditions and execute defined remediation steps without waiting for a page.
• Root cause correlation. Agents that traverse enriched pipeline signal to surface a probable cause, not just an alert ID. See AI SRE for root cause analysis.
• Context-aware escalation. Agents that know when to act autonomously and when to hand off, with a full audit trail attached.

Active Telemetry is the control plane. AURA is what runs in it.

The Bottom Line

Logging everything by default was always a workaround. Exclusion rules help at the margins, but they're downstream of the real decision: what does your pipeline do with data before it hits an expensive destination?

The teams making progress aren't just cutting observability costs. They're building pipeline infrastructure that's ready for the next problem — giving AI agents the context they need to act, not just the data to look at.

Telemetry control starts upstream. Agentic operations start there too.

Explore Active Telemetry: mezmo.com/platform/active-telemetry

Get started with AURA: github.com/mezmo/aura

‍