gRPC Load Balancing Breaks the Kubernetes Mental Model Faster Than Most Teams Expect

gRPC load balancing on Kubernetes can fail in ways that surprise teams because long-lived HTTP/2 connections do not behave like short, independent HTTP requests. A normal ClusterIP Service may distribute connections, but once a gRPC client establishes one, many calls can remain pinned to the same backend pod.

That becomes especially visible when one pod degrades instead of failing outright.

The benchmark discussed compared vanilla Kubernetes with Linkerd, Istio, and Cilium under healthy and unhealthy conditions. The authors said the goal was to measure what happens when gRPC traffic encounters normal pods, slow pods, and different data-plane balancing algorithms.

The results generated interest because vanilla Kubernetes appeared to perform poorly when one backend slowed down, while all three service meshes handled the condition better. Linkerd appeared to show the strongest latency behavior in the tested scenarios.

That conclusion came with an important disclosure. Everyone involved in the benchmark worked at Buoyant, the company behind Linkerd. The authors said the results were reproducible with open source releases, but readers still challenged the design, the vanilla baseline, the load model, and the missing client-side load-balancing comparison.

Those objections do not make the benchmark useless. They show how service-mesh benchmarks should be read.

Kubernetes Services balance connections, not gRPC calls

The core issue is connection pinning.

A Kubernetes ClusterIP Service can distribute new connections across backend pods. With HTTP/1.1 traffic, clients often create enough connections over time that requests spread reasonably well.

gRPC normally runs over HTTP/2 and multiplexes many calls over a smaller number of long-lived connections. If a client opens one connection and keeps using it, all the calls on that connection may continue reaching the same backend.

The Service did its job when it selected the endpoint for the connection. It does not reselect a pod for every gRPC request flowing through that established channel.

This creates uneven traffic distribution even when every pod is healthy.

It becomes much worse when one pod is degraded. The connection can remain pinned to the slow backend, and the client continues experiencing high latency even though other replicas are healthy and underused.

That is why the source discussion described vanilla Kubernetes as “keeling over” when one pod became slow. The problem was not that the Service stopped functioning. It was that connection-level balancing was too coarse for the request pattern.

A degraded pod is more dangerous than a dead pod

Hard failures are often easier for infrastructure to handle.

When a pod crashes, fails readiness, or disappears from the endpoint set, new traffic can be sent elsewhere. Existing connections may fail, forcing clients to reconnect.

A degraded pod is still alive.

It may continue accepting requests while responding slowly. If the readiness check does not reflect the degradation, Kubernetes still considers it available. A client with a pinned connection has no reason to move.

This creates a classic tail-latency problem.

Most requests may remain fast because they reach healthy pods. A smaller share may become dramatically slower because they continue landing on the degraded endpoint. Average latency may look acceptable while p95, p99, or maximum latency becomes painful.

That is where load-aware balancing becomes valuable.

A data plane that observes endpoint latency can reduce traffic to the slow pod without waiting for it to fail a health check completely. The benchmark discussion said Linkerd’s peak-EWMA load score caused the degraded endpoint to lose most power-of-two-choices comparisons, sharply reducing its request share.

The key difference is that the balancer reacted to observed performance, not just endpoint membership.

Linkerd, Istio, and Cilium were not using the same algorithm

One of the most useful clarifications in the discussion was that none of the tested meshes simply used round robin.

The Envoy-based meshes used LEAST_REQUEST.

Linkerd used EWMA with power-of-two choices.

The vanilla setup operated at the connection level.

These algorithms react differently under skew.

Least-request approaches prefer endpoints with fewer active requests. This can move traffic away from a slower backend because slow requests remain in flight longer, increasing the apparent load on that endpoint.

Linkerd’s approach used observed latency as part of the load score. As one pod slowed, the algorithm increasingly avoided it.

The distinction matters because “service mesh load balancing” is not one behavior.

Two meshes can both distribute traffic at Layer 7 while producing different tail latency under the same failure condition. The proxy implementation, endpoint selection algorithm, retry behavior, connection reuse, and measurement window all affect the result.

That is why the benchmark’s strongest claim should not be reduced to “meshes are better.” The more precise conclusion is that request-aware and load-aware balancing handled a slow backend better than the default connection-pinned ClusterIP path in the tested setup.

The vanilla baseline was the biggest methodological challenge

One commenter argued that the vanilla comparison was unfair because it used a ClusterIP Service instead of a headless Service with gRPC client-side load balancing.

That is a serious question.

With a headless Service, DNS can return multiple pod addresses. A gRPC client configured with a dns:/// target and round_robin strategy can create channels to multiple endpoints and distribute calls across them.

The benchmark authors acknowledged that this arm was not included.

They explained that the benchmark focused on the default ClusterIP setup used in many clusters and on how service meshes handle connection pinning. They also checked the benchmark tool behavior and said the tested ghz version used grpc-go with pick_first when no load-balancing strategy was set.

They expected client-side round robin to produce distribution similar to Layer 7 meshes in a uniform healthy scenario, potentially with the best p50 latency because there is no extra proxy hop.

The limitation appears under degradation.

Round robin is not load-aware. If one of five pods is slow, the client may continue sending roughly one-fifth of calls to it. The traffic is distributed, but the slowest responses remain close to the degraded pod’s delay.

That does not make client-side balancing useless. It means it answers a different question.

A fair follow-up benchmark should compare:

• ClusterIP with default gRPC behavior
• Headless Service with pick_first
• Headless Service with round_robin
• Linkerd
• Istio
• Cilium

That would show the difference between connection distribution and adaptive load awareness more clearly.

Healthy-state latency still matters

A mesh that handles degradation well may also add latency during normal operation.

One reader asked for a clearer comparison of added latency against a one-pod baseline with a known static delay. They wanted to know how much each solution added before considering failure behavior.

That is the right instinct.

A service mesh inserts more work into the request path. Sidecars or node proxies process connections, collect telemetry, apply policy, and make routing decisions. Even when overhead is small, it is not zero.

The benchmark discussion suggested that Linkerd’s healthy baseline added less latency than the other compared meshes, and one commenter said the delta looked strong.

Still, the source discussion did not provide enough numerical detail to reproduce or independently validate that claim inside this article.

The correct operational question is not simply which mesh wins under failure. It is whether the healthy-state cost and degraded-state benefit match the application’s latency budget.

For some services, a small increase in median latency is acceptable if p99 latency improves dramatically during partial failure. For others, the normal-path overhead may matter more.

Both conditions should be measured.

Closed-loop load changes how results should be interpreted

The benchmark authors said the tests used a closed-loop model.

In a closed-loop benchmark, the client sends new work as previous requests complete. That couples throughput and latency. If responses slow down, the client generates less new traffic.

This can make each system operate at a different effective request rate.

One commenter asked for constant RPS and connection counts so each data plane could be compared under the same load. The authors agreed that a fixed-load result would be interesting but explained that the chosen model made each percentile reflect the system’s own operating point.

Neither method is universally correct.

Closed-loop tests reflect client backpressure and can resemble systems where callers wait for responses before sending more work. Open-loop or fixed-RPS tests are better for showing what happens when arrival rate remains constant regardless of latency.

A strong benchmark suite should often include both.

The risk of using only closed-loop load is that a slower system may appear to remain stable partly because it processes fewer requests. The risk of using only fixed RPS is that the generator may push some systems beyond a realistic operating point and turn the result into a saturation test.

The methodology must match the production question.

Isolation and repeatability strengthen the result

The benchmark authors described several controls intended to reduce noise.

Each mesh ran in its own isolated, identically configured EKS cluster. The setup used anti-affinity rules, a dedicated load-generation node, a fixed connection count, a fixed number of in-flight streams, and a timed warm-up period that was discarded.

Those choices are good benchmarking hygiene.

Separate clusters reduce the chance that one mesh leaves behind state or resource pressure that affects the next. Anti-affinity limits accidental pod concentration. A dedicated load node reduces interference from application workloads. Warm-up removal avoids measuring startup behavior as steady state.

The authors also named the benchmark tool: ghz.

That transparency helps reproduction.

The discussion still left open questions about backend baseline performance, exact latency definitions, networking details, EKS CNI configuration, and whether other cloud load-balancing paths were involved. Those gaps do not invalidate the setup, but they identify where a reproduction should add detail.

A benchmark earns trust through enough information for someone else to rerun it and obtain a similar shape of result.

Replica count moves the problem deeper into the tail

The authors made another useful point about scaling replica count.

If one of five pods is degraded, a non-load-aware algorithm may send about 20 percent of calls to the slow endpoint.

If one of one hundred pods is degraded, only about one percent of calls may be affected.

The problem has not disappeared. It moved further into the tail.

That means p50 and p95 may look excellent while p99 or p99.9 still reveals the slow backend.

Large replica counts can therefore hide poor balancing if the benchmark does not collect detailed enough tail metrics.

The same principle applies to production monitoring.

Teams that only watch averages may miss a severe user experience problem affecting a small fraction of requests. gRPC systems with many replicas should track high-percentile latency and endpoint-specific behavior.

The benchmark debate correctly suggested testing several degraded-pod ratios such as 1/5, 1/20, 1/50, and 1/100.

That would show whether the algorithm advantage remains visible as the bad endpoint becomes a smaller part of the fleet.

Client-side balancing is powerful but shifts responsibility

One commenter questioned whether client-side load balancing is practical at very large microservice counts.

That concern is less about whether the mechanism works and more about operational ownership.

Client-side balancing requires clients to discover endpoints, understand naming, choose a policy, handle updates, and maintain compatible behavior across languages and libraries.

In a small homogeneous system, that can be efficient and avoid proxy overhead.

In a large platform with many teams and languages, it can create inconsistency. Some clients may use round robin, others pick_first, others custom logic, and some may never update when the recommended policy changes.

A service mesh centralizes more of that behavior in the platform.

The tradeoff is clear:

• Client-side balancing can reduce hops and keep logic close to the application
• Mesh-based balancing can standardize behavior across services and languages
• Client-side round robin may distribute traffic but remain blind to degraded endpoints
• Meshes add infrastructure complexity and operational cost

The right choice depends on organizational scale, latency sensitivity, language diversity, and how much networking behavior the platform team wants to own.

Licensing and release cadence affect technical decisions

The discussion also shifted briefly into Linkerd licensing and release practices.

One reader said they had stopped considering Linkerd after changes around 2024. Another replied that open source releases still existed and argued that the concern was overstated, while noting that frequent edge releases could require more updating for teams that want the latest features.

This issue is important because service mesh selection is not based on benchmark performance alone.

Teams need to evaluate:

• License terms
• Open source release availability
• Support model
• Upgrade cadence
• Security patch access
• Long-term maintenance
• Commercial dependencies
• Community health

The source discussion did not resolve the licensing debate fully, and this article should not pretend it did.

What it established is that some operators still view release and licensing changes as a barrier, while others consider the project usable and openly available.

A benchmark win does not remove procurement, governance, and lifecycle concerns.

The benchmark was useful because people challenged it

The strongest part of the discussion was not the conclusion that Linkerd performed well.

It was the quality of the objections.

Readers asked why the vanilla baseline did not include headless Services. They requested a one-pod latency baseline, constant RPS, different degraded-pod ratios, clearer latency definitions, backend benchmarks, and more detail about cluster isolation.

The authors responded with specifics about grpc-go behavior, ghz, pick_first, round_robin, EWMA-P2C, least-request, isolated EKS clusters, anti-affinity, warm-up periods, and closed-loop load.

That exchange made the benchmark more useful.

Vendor benchmarks should be treated as hypotheses with reproducible methods, not as neutral verdicts. The disclosure that the authors worked for the company behind Linkerd was necessary. The willingness to explain methodology mattered just as much.

The result worth carrying forward is not “Linkerd wins.”

It is this:

Default Kubernetes connection balancing is a poor fit for long-lived gRPC traffic when one backend degrades. Request-aware balancing helps. Load-aware balancing helps more. The exact winner depends on the workload, algorithm, failure mode, and benchmark design.

FAQ

Why does gRPC load balancing behave poorly with a Kubernetes ClusterIP Service?

gRPC reuses long-lived HTTP/2 connections. Kubernetes usually balances the initial connection, so many calls can remain pinned to one backend pod.

What happens when one gRPC backend becomes slow?

If the connection remains pinned and the pod still passes readiness checks, calls may continue reaching the degraded backend, creating severe tail latency.

How does a service mesh improve gRPC balancing?

A mesh can make Layer 7 routing decisions for individual requests or streams and use load-aware algorithms to reduce traffic to slower endpoints.

What algorithm did Linkerd use in the discussion?

The benchmark authors said Linkerd used power-of-two choices with a peak-EWMA latency-based load score.

What did Istio and Cilium use?

The authors said the Envoy-based meshes used LEAST_REQUEST in the tested configurations.

Was the vanilla Kubernetes comparison complete?

No. The benchmark used a ClusterIP baseline and did not include a headless Service with gRPC client-side round robin. That was the main methodological criticism.

Would client-side round robin solve the problem?

It can improve distribution in healthy conditions, but it is not load-aware and may continue sending a fixed share of calls to a degraded pod.

What should teams benchmark before choosing a mesh?

They should test healthy latency, degraded endpoints, fixed-RPS and closed-loop load, different replica counts, multiple degraded-pod ratios, resource overhead, and operational complexity.

CTA

Service behavior under partial failure is difficult to understand without visibility across latency, endpoints, traffic distribution, and infrastructure health. Request an online trial and explore how Sensaka supports clearer monitoring across complex Kubernetes environments.