Kubernetes resource tuning — avoiding fake precision

Kubernetes resource tuning is not about finding the perfect CPU and memory numbers. It is about understanding what each setting actually controls.

A common mistake is treating requests and limits as if they are just "small" and "large" versions of the same value. They are not.

• Requests are mostly scheduling signals.
• Limits are runtime boundaries.
• Memory limits can kill the container.
• CPU limits can throttle the container.
• Missing requests can lead to bad placement and noisy-neighbor behavior.

The goal is not to guess. The goal is to measure, set sane initial values, observe failure modes, and then adjust.

Start with what the workload actually does

Before changing resources, I want to know the shape of the workload:

• Is it CPU-bound or I/O-bound?
• Is memory stable or slowly growing?
• Are there startup spikes?
• Are there batch jobs or traffic bursts?
• Does latency matter more than throughput?
• Is the service horizontally scalable?

A pod serving HTTP traffic has different tuning needs from a Kafka consumer, a cron job, a database, or a JVM service with large heap usage.

Useful starting points:

kubectl top pods -n production --containers
kubectl describe pod <pod-name> -n production
kubectl get events -n production --sort-by=.lastTimestamp

For Prometheus, CPU is usually treated as a rate because it is accumulated over time:

sum by (pod) (
  rate(container_cpu_usage_seconds_total{namespace="production", container!="POD"}[5m])
)

Memory is different. It is a current value that goes up and down, so I usually look at current usage, max over a window, or quantiles over time:

sum by (pod) (
  container_memory_working_set_bytes{namespace="production", container!="POD"}
)

max_over_time(
  container_memory_working_set_bytes{namespace="production", container!="POD"}[1h]
)

The exact metric names depend on your monitoring setup, but the important idea is the same: CPU usage is usually rate-based, memory usage is not.

Requests: tell the scheduler what the pod needs

A CPU or memory request tells Kubernetes: "place this pod on a node that can afford this much."

For example:

resources:
  requests:
    cpu: "500m"
    memory: "512Mi"

This does not mean the container can only use 500m CPU or 512Mi memory. It means Kubernetes uses those numbers when deciding where the pod should run.

If requests are too low, the scheduler may pack too many pods onto the same node. Everything looks fine at deploy time, but under real traffic the node becomes noisy and latency gets worse.

If requests are too high, pods may stay Pending even though the cluster looks underutilized from a real usage perspective.

So requests are a trade-off:

• too low: risky placement
• too high: wasted capacity and scheduling failures
• roughly right: predictable placement without unnecessary waste

Limits: define the failure boundary

Limits are different.

A memory limit is a hard boundary. If the container exceeds it, it can be OOMKilled.

resources:
  limits:
    memory: "1Gi"

That makes memory limits useful, but dangerous if set blindly. Increasing the limit may hide the symptom, but it does not explain the cause.

For CPU, limits are softer in a different way. The container is usually throttled rather than killed.

resources:
  limits:
    cpu: "2000m"

CPU throttling can be especially painful for latency-sensitive services. A service may have free CPU available on the node, but still get throttled because it hit its configured CPU quota.

For that reason, I avoid setting tight CPU limits on many application services unless there is a specific reason to isolate them. I care much more about setting realistic CPU requests.

A better rule than P50/P95

I do not like blindly saying "requests at P50, limits at P95." It sounds precise, but workload behavior matters more than a single percentile rule.

A more practical starting point:

• CPU request: normal sustained usage, plus enough room for predictable traffic.
• CPU limit: avoid it or set it generously unless you specifically need hard isolation.
• Memory request: close to normal working set.
• Memory limit: above observed peaks, with enough room for runtime overhead.
• Critical workloads: tune more conservatively and validate under load.

For small stateless services, being slightly under-requested may be acceptable if horizontal scaling is fast and the service is resilient.

For critical services, databases, brokers, or anything expensive to restart, I would rather over-request slightly than create constant eviction or OOM risk.

JVM memory needs extra care

Java containers are easy to misconfigure because JVM memory is not only heap.

A container with a 1Gi memory limit should not usually have:

-Xmx1g

The heap is only one part of the process. You also need room for metaspace, direct buffers, thread stacks, code cache, GC overhead, native allocations, agents, and the process itself.

A safer fixed-heap example:

resources:
  requests:
    memory: "768Mi"
  limits:
    memory: "1Gi"
env:
  - name: JAVA_OPTS
    value: "-Xmx700m -XX:MaxMetaspaceSize=128m"

For newer JVMs, I often prefer percentage-based tuning:

-XX:MaxRAMPercentage=70

That lets the JVM size the heap based on the container memory limit, while still leaving space for non-heap memory.

The exact percentage depends on the application. A simple Spring Boot API, a Kafka consumer, and a high-throughput Netty service may have very different off-heap behavior.

OOMKilled is a symptom, not the diagnosis

When a pod is OOMKilled, the lazy fix is to increase the memory limit.

Sometimes that is correct. Often it is not enough.

I usually check:

kubectl describe pod <pod-name> -n production
kubectl logs <pod-name> -n production --previous
kubectl get events -n production --sort-by=.lastTimestamp

Then I look at the memory graph.

A sharp spike may mean a large request, batch, cache warmup, decompression, or temporary allocation.

A slow upward trend may indicate a leak, unbounded cache, stuck references, too many threads, or poor lifecycle cleanup.

For JVM services, heap dumps, GC logs, and allocation profiling are more useful than guessing.

QoS classes matter, but do not cargo-cult them

Kubernetes assigns pods a QoS class based on requests and limits:

• BestEffort: no requests or limits
• Burstable: some requests/limits, but not fully equal
• Guaranteed: CPU and memory requests equal limits for all containers

Guaranteed pods are the last to be evicted under node pressure, but that does not mean every pod should be Guaranteed.

For example, setting CPU request equal to CPU limit may reduce scheduling flexibility and cause avoidable throttling.

I usually reserve Guaranteed-style tuning for workloads where eviction would be very expensive, or where predictable resource isolation matters more than efficient bin-packing.

Stateful workloads need more conservative tuning

Stateful workloads deserve more caution because restarts and rescheduling are more expensive.

For databases, queues, and storage-heavy services, I care about:

• stable memory behavior
• predictable disk I/O
• enough CPU for compaction/background work
• avoiding eviction
• graceful shutdown time
• disruption budgets
• node placement and anti-affinity

For memory-heavy stateful workloads, setting memory request close to memory limit can make sense. But I still avoid blindly applying "requests = limits" to everything, especially CPU, without understanding the workload.

Validate with real load

The best resource config is the one that survives realistic traffic.

I prefer testing with actual application-level load rather than running random stress commands inside the container.

Useful signals:

• p95/p99 latency
• CPU throttling
• memory working set
• restart count
• OOMKilled events
• HPA behavior
• GC pauses for JVM apps
• node pressure events
• pending pods due to insufficient resources

The resource config should be tested together with the scaling config. A pod that looks fine alone may behave badly when HPA adds replicas, the node fills up, or startup CPU spikes overlap.

Practical checklist

Before calling a resource config "good enough", I check:

• requests are set for every production container
• memory limits leave room for runtime overhead
• CPU limits are intentional, not copied from a template
• JVM heap does not consume the whole container limit
• OOMKilled events are investigated, not just patched
• memory graphs distinguish spikes from leaks
• critical workloads have safer QoS and eviction behavior
• load tests include startup, steady state, and burst behavior
• dashboards show throttling, restarts, and node pressure

Final thought

Kubernetes resource tuning is less about perfect numbers and more about failure behavior.

Bad requests create bad scheduling. Bad memory limits create OOMKills. Bad CPU limits create throttling. Bad observability creates guessing.

The safest approach is to measure the real workload, understand what Kubernetes does with each value, and tune based on evidence instead of defaults copied from another service.