The Ops Community ⚙️: Ashwini Dave

Trace Sampling in Practice: Keeping the Signal Without Paying for the Noise

Ashwini Dave — Fri, 09 Oct 2026 09:18:08 +0000

Distributed tracing is one of the most useful tools in an SRE's kit, and one of the easiest to overspend on. Once a handful of services turn into a few dozen, trace volume grows faster than traffic. Storage bills climb, queries slow down, and teams quietly start turning tracing off.

Most traces in a distributed tracing setup are boring. A healthy GET /health or a cached product lookup that returns in 8 ms tells you very little, and you may be generating millions of them per hour. The traces that matter are the slow ones, the failed ones, and the ones touching a recently deployed service. The goal of sampling isn't to keep less data, but to keep an honest picture of normal behavior while retaining nearly all of the interesting outliers.

No distributed tracing strategy is perfect, but the worst choice is no sampling at all, followed closely by a policy nobody revisits. Start simple, measure what you're losing, and tighten the rules as you learn which traces your team actually opens during incidents.

Why 100% Retention Rarely Makes Sense

Most traces are boring. A healthy GET /health or a cached product lookup that returns in 8 ms tells you very little, and you may be generating millions of them per hour. The traces that matter are the slow ones, the failed ones, and the ones touching a recently deployed service.

The goal of sampling isn't "keep less data". It's to keep a statistically honest picture of normal behavior while retaining nearly all of the interesting outliers.

Head-Based Sampling

Head-based sampling decides at the start of a trace, usually in the SDK or at the entry-point service, whether to record it. The decision is then propagated to downstream services through trace context.

Strengths

Cheap and simple. Unsampled requests never generate span data.
Predictable cost, since you control the percentage directly.
Works well with OpenTelemetry's built-in samplers, such as TraceIdRatioBased and ParentBased.

Weaknesses

The decision is made before you know how the request will turn out. A 1% sample means you keep roughly 1% of your errors too.
Rare, high-value failures are the most likely to go missing.

Head-based sampling is a reasonable default for high-volume, low-risk traffic, but it's a poor fit if you rely on traces for debugging rare issues.

Tail-Based Sampling

Tail-based sampling waits until a trace is complete, then decides whether to keep it. That lets you write rules based on what actually happened:

Keep every trace containing an error
Keep every trace slower than a latency threshold (say, p95)
Keep a small random percentage of everything else

Strengths

You retain the traces that matter most.
Policies can be expressive: by service, endpoint, status code, or custom attributes.

Weaknesses

It's more expensive to run. All spans for a trace must be buffered somewhere until the decision is made.
All spans from one trace have to reach the same decision point. In the OpenTelemetry Collector, that means load balancing by trace ID before the tail-sampling processor.
Late-arriving spans, from async jobs or message queues, can be missed if your decision window is too short.

A Practical Hybrid

Most mature setups end up combining both:

Drop the obvious noise early. Filter health checks, readiness probes, and synthetic pings at the SDK or Collector level. This is a filtering decision, not sampling.
Apply light head sampling at the edge to cap raw volume.
Run tail sampling in a Collector tier with policies for errors, high latency, and a baseline random sample.
Always keep certain traffic. Traces tied to a feature flag rollout, a canary deployment, or a specific customer under investigation shouldn't be left to chance.

This gives you cost control from the head stage and precision from the tail stage.

Common Pitfalls

Broken traces. If services make independent sampling decisions, you'll get traces with missing spans. Always honor the parent's sampling decision with ParentBased samplers.

Skewed metrics. If you compute request counts or error rates from sampled traces, the numbers will be wrong unless you correct for the sampling rate. Generate metrics from the full stream before sampling, for example with span-metrics in the Collector, and use traces for investigation only.

Set-and-forget policies. A latency threshold chosen last year may no longer reflect reality. Review your tail-sampling rules whenever traffic patterns or SLOs change.

Ignoring the sampler's own health. Collectors that buffer traces can run out of memory under load. Monitor buffer size, dropped spans, and decision latency like any other production system.

Cardinality surprises. Policies keyed on high-cardinality attributes, such as user IDs or full URLs, can blow up memory use in the sampling tier.

Storage and Query Considerations

Sampling is only half the story. How retained traces are stored affects how quickly you can query them:

Index selectively. Indexing every attribute is expensive. Index the ones you filter on most (service, status, duration, a few business keys) and leave the rest searchable but unindexed.
Tier your retention. Keep recent traces on fast storage and move older ones to cheaper storage.
Partition by time. Most investigations focus on a narrow window, so time-based partitioning keeps scans small.

How to Decide

Ask these questions:

Do I need to debug rare failures? → Tail sampling.
Is my main constraint cost at very high volume? → Head sampling plus aggressive filtering.
Do I need accurate rate and error metrics? → Derive them before sampling.
Do I have the operational capacity to run a stateful Collector tier? If not, start with head sampling and graduate later.

Final Thoughts

No sampling strategy is perfect, but the worst choice is none at all, followed closely by one nobody revisits. Start simple, measure what you're losing, and tighten the policies as you learn which traces your team actually opens during incidents.

What sampling strategy does your team use, and what's the one trace you wish you'd kept? Share your experience in the comments.

How to Evaluate an AI SRE Before You Trust It With Production Incidents

Ashwini Dave — Tue, 29 Sep 2026 11:30:16 +0000

AI SRE is becoming a real software category.

The pitch is attractive: connect an AI system to alerts, logs, traces, deployment history, runbooks, source code, and infrastructure, then let it investigate incidents before an engineer manually jumps across several tools.

That can save time.

But AI SRE products are easy to demo and much harder to evaluate.

A polished demo can show an agent correlating an alert with a deployment and producing a root-cause hypothesis in seconds. Production incidents are rarely that tidy. Telemetry may be incomplete. Signals may correlate without being causal. The required value may never have been logged. The agent may generate invalid queries, overstate confidence, or recommend unsafe actions.

That is why AI SRE should not be evaluated like another dashboard.

The useful question is:
“How much of our real incident investigation workflow can this system shorten without sacrificing evidence quality or operational safety?”

When comparing AI SRE tools, that matters more than the length of a feature list.

Start With the Incident Problem You Actually Have

Not every AI SRE product solves the same problem.

Some focus on reducing alert noise. Others automate incident coordination. Some correlate observability data and generate root-cause hypotheses. Others specialize in Kubernetes or production debugging.

Those are different use cases.

Incident Bottleneck	Capability That Matters
Too many alerts	Event correlation and noise reduction
Slow incident coordination	Routing and response workflows
Too much telemetry	Automated investigation and synthesis
Kubernetes failures	Cluster-aware diagnosis
Difficult root cause analysis	Evidence-backed hypothesis generation
Missing code-level state	Runtime evidence capture
Risky remediation	Verification and approval controls

This distinction matters because a strong product can still be wrong for your team.

If your engineers already know where a failure happened but repeatedly need to add more logging to understand why, another telemetry summarizer may not solve the problem.

Test Against Real Incidents, Not Vendor Demos

The best AI SRE evaluation dataset is probably sitting in your incident history.

Select five to ten representative production incidents, including:

deployment regressions
dependency failures
intermittent application errors
resource exhaustion
Kubernetes failures
configuration issues
incidents where the first hypothesis was wrong

For each one, preserve the information that was actually available at the time:

Initial alert
Logs
Metrics
Traces
Deployment history
Source code
Runbooks
Infrastructure state

Then give the AI SRE the same starting point.

The workflow becomes:

This reveals much more than a staged demo.

A system may perform very well when the incident is explained by a clear deployment regression. It may struggle when signals conflict or when the decisive evidence was never collected.

Those are exactly the cases worth testing.

Measure Evidence Quality, Not Just Root Cause Accuracy

Root-cause accuracy matters, but it is not enough.

Imagine two systems both arrive at the correct diagnosis.

System A makes several unsupported claims, performs 30 unnecessary tool calls, and suggests restarting a production database.

System B reaches the same conclusion with fewer queries, cites the relevant evidence, rejects one incorrect hypothesis, and keeps remediation behind an approval gate.

Calling both equally accurate misses most of what matters.

A useful evaluation should measure:

Time to first useful hypothesis

How quickly does the system reduce the search space?
The first answer does not need to be final. It needs to be useful.

Evidence precision

Does the evidence actually support the claim being made?
A deployment and an error spike happening close together may justify investigation. They do not prove causality.

Unsupported-claim rate

How often does the system present an assumption as fact?
Compare:
Connection pool exhaustion caused the incident.
with:
Database latency increased during the incident, so connection pool exhaustion is one hypothesis worth testing.

The second statement preserves uncertainty correctly.

Invalid tool-call rate

How often does the agent request a nonexistent metric, malformed query, incorrect field, or unavailable resource?

This matters because a failed query can easily be misread as evidence that the underlying signal does not exist.

Ability to reject its own hypothesis

A useful AI SRE should not only generate explanations. It should test them.

If the system suspects connection pool exhaustion but sees 34 active connections out of a maximum of 100, it should reject that hypothesis and continue.

The Most Important Test: What Happens When Telemetry Is Incomplete?

This is where AI SRE evaluations become much more interesting.

Imagine a checkout service begins returning intermittent HTTP 500 errors.
The investigation shows:

14:03 checkout-api deployed
14:07 HTTP 500 rate increases
14:08 InvalidRegionException appears

Distributed tracing narrows the problem to:
checkout-api
|
+-- validateCustomerRegion()
|
+-- createOrder()
|
+-- InvalidRegionException

The deployment changed region-validation logic.
The AI SRE generates two hypotheses:
H1: customer.region is null
H2: enabled_regions contains stale configuration

Now ask:
“Where is the evidence that distinguishes those two explanations?”

Suppose neither value was logged.

The traces do not contain them. The metrics do not expose them. The error tracker did not capture them.

At this point, a reasoning-only system has reached its limit.
It can rank hypotheses. It cannot confirm them.

This is an important evaluation point.

Does the AI SRE only reason over existing telemetry, or can it gather new evidence?

HyperProbe is relevant specifically at this boundary.

HyperProbe is an AI on-call agent, and sits alongside an existing observability and alerting stack.

It can capture live runtime state from running services using read-only probes when logs, traces, and dashboards stop short of the evidence needed to verify root cause.

That means the investigation could continue with:
Need:
customer.region
enabled_regions

    |
    v

Capture runtime state

    |
    v

customer.region = null
enabled_regions = ["US", "CA", "GB"]

Now one hypothesis has direct evidence.

That is materially different from asking the model to become more confident about incomplete telemetry.

Check the Safety Boundary

An AI SRE should not become another production risk.

The ability to inspect or act on production systems needs clear boundaries.

Useful controls include:
read-only operations by default
role-based access control
query validation
capture limits
automatic expiry
sensitive-data redaction
auditability
approval requirements for remediation

The key architectural principle is simple:
“The AI can decide what evidence it needs. Deterministic policy should decide what it is allowed to inspect or change.”

That separation matters far more than a generic claim of “human in the loop.”

Evaluate Category Fit, Not a Universal Winner

There probably is not one AI SRE that is best for every team.

If your biggest problem is alert noise, prioritize correlation and suppression.

If your team struggles with incident coordination, focus on response workflows.

If your infrastructure is heavily Kubernetes-based, cluster-aware diagnosis may matter most.

If your engineers repeatedly reach the point where they know which function failed but still need another log line to understand runtime state, live production debugging becomes much more relevant.

This is why comparing AI SRE tools by use case is more useful than looking
for one universal winner.

Very few products will be equally strong at all of them.

The Best AI SRE Is the One That Shortens Your Actual Incident Workflow

AI SRE demos can look impressive.

An alert arrives. The agent runs a few queries. A root cause appears.
Production is rarely that clean.

A useful AI SRE should be evaluated on how it behaves when:
signals conflict
the first hypothesis is wrong
telemetry is incomplete
an important value was never logged
remediation carries production risk
the system needs to admit that it does not yet know

That is where the difference between a convincing demo and a useful reliability tool becomes clear.

The best evaluation process starts with your own incidents.
Replay them. Measure evidence quality. Track unsupported claims.

Test whether the agent can reject its own hypotheses. See what happens when telemetry runs out.

Because the best AI SRE is not the one that produces the fastest answer.

It is the one that reduces the specific work your engineers actually perform during production incidents while preserving the evidence and control required to trust the result.

The Evolving Role of Observability in Autonomous SRE Agents

Ashwini Dave — Fri, 08 May 2026 07:11:22 +0000

The Observability Imperative for Modern SRE

Site Reliability Engineering (SRE) has always demanded deep system understanding, but today's distributed architectures—microservices, serverless functions, multi-cloud setups—generate telemetry at petabyte scales. Traditional monitoring answers "what broke," but observability reveals "why" through correlated logs, metrics, and traces.

Enter SRE agents: AI-driven systems that ingest observability data to automate toil-heavy tasks like anomaly detection, root cause analysis (RCA), and remediation. These agents don't replace SREs; they amplify them by handling repetitive investigations, letting humans focus on architecture and innovation.

Recent benchmarks show SRE agents reducing mean time to resolution (MTTR) by 40-70% in production environments. Yet their success hinges on robust observability foundations—without high-fidelity signals, agents hallucinate or miss subtle issues.

Core Components of an SRE Agent Pipeline

Effective SRE agents follow an OODA loop (Observe-Orient-Decide-Act), powered by observability:

1. Observation Layer: Multi-Signal Fusion

Agents pull from unified pipelines using OpenTelemetry (OTel) standards. Metrics provide quantitative baselines (e.g., RED: Rate, Errors, Duration); traces map causal chains across services; logs add qualitative context for errors.

Key Insight: Agents excel with semantic conventions. OTel's GenAI extensions tag LLM inputs/outputs, enabling agents to monitor token usage and latency in AI workloads—critical as SRE teams manage inference pipelines. linkedin

In practice, fuse signals via vector databases for semantic search. A latency spike isn't just a P95 metric; it's correlated with trace spans showing a slow database query in 80% of affected requests.

2. Orientation: Contextual Reasoning

Raw data overwhelms; agents use knowledge graphs to contextualize. Nodes represent services/pods; edges show dependencies weighted by blast radius.

Example: During a 2025 outage analysis at a major e-commerce platform, an SRE agent correlated a 3x error rate in checkout (metrics) with increased payment-gateway spans (traces) and "connection pool exhausted" logs, pinpointing a config drift—all in under 2 minutes.

Agents employ retrieval-augmented generation (RAG): Query observability stores, retrieve relevant telemetry, then reason via LLMs like GPT-4o or Llama 3.1. Guardrails prevent overconfidence—e.g., confidence scores below 80% trigger human escalation.

Diagnosis: From Patterns to Root Causes

SRE agents shine in RCA, moving beyond correlation to causation.

Pattern Recognition

Anomaly Baselines: Use statistical models (e.g., Prophet for seasonality) on metrics; graph neural networks on traces.
Dimensional Drill-Down: Auto-slice by high-cardinality fields like user_id or region without predefined queries.

Action and Learning: Closing the Loop

Autonomous agents don't stop at diagnosis—they act:

Remediation: Generate runbooks (e.g., "scale pod replicas to 5") or execute via APIs (Kubernetes HPA).
Feedback Loops: Post-incident reviews update agent memory via RLHF (reinforcement learning from human feedback).

Phase	Observability Role	Agent Capability
Observe	Real-time telemetry ingestion	Multi-modal fusion (logs+metrics+traces)
Diagnose	Dimensional analysis + traces	Causal graph reasoning
Act	Alert enrichment + runbook context	API orchestration
Learn	Incident replay datasets	Model fine-tuning

Long-term, agents build "system memory": Vector stores of past incidents enable proactive hunting for recurring patterns.

Challenges and Practical Mitigations

1. Data Quality and Cost

High-volume traces explode storage costs. Mitigate with:

Adaptive sampling: 1:1000 on happy paths, 1:1 on errors.
Aggregation: Use PromQL/OTel processors for pre-agent filtering.

Stat: Teams retain 90-day metrics, 30-day traces, 7-day logs—agents query efficiently via indexes.

2. Explainability and Trust

Black-box LLMs erode confidence. Counter with:

Chain-of-thought prompting: Agents verbalize reasoning ("Latency spiked due to X because Y").
Human-in-loop: Escalate >5% blast radius incidents.

3. Security and Scope

Agents with API access risk privilege escalation. Implement RBAC + audit logs; start narrow (read-only RCA) before expanding to writes.

Benchmark Tip: Test agents on Chaos Engineering scenarios—inject faults, measure detection accuracy.

The Future: Toward Fully Autonomous Operations

SRE agents mark Observability 3.0: From reactive dashboards to proactive autonomy. By 2027, Gartner predicts 50% of enterprises will deploy agents handling 80% of incidents.

Yet humans remain essential for error budgets, SLO design, and ethical oversight. Observability evolves from "three pillars" to an AI-ready lakehouse: Petabyte-scale, queryable at sub-second latency.

Call to Experiment: Start small—prototype an agent on Grafana Loki + LlamaIndex. Instrument a toy microservices app, simulate faults, iterate on prompts. The signal-to-noise ratio in your telemetry will dictate success.

Observability isn't just data; it's the nervous system empowering agents to keep systems resilient.