Introduction

A Go service on EKS can be OOMKilled even when runtime.MemStats.Alloc looks small. This is a common observability mismatch: Go heap metrics describe managed heap objects, while Kubernetes enforces memory limits on total process memory (RSS/cgroup usage), which includes stacks, mmap regions, cgo allocations, page cache effects, fragmentation, and runtime overhead.

If you diagnose only Alloc, you can miss the real pressure and keep raising pod memory blindly. The right approach is to correlate Go runtime metrics with OS-level memory and cgroup limits, then tune runtime behavior and workload patterns accordingly.

Core Sections

1. Understand the metric gap: Go heap vs RSS

runtime.MemStats.Alloc is live heap bytes managed by the Go GC. Kubernetes OOM decisions are based on container memory usage, usually close to RSS from the kernel perspective.

var m runtime.MemStats
runtime.ReadMemStats(&m)
log.Printf("Alloc=%d HeapInuse=%d Sys=%d", m.Alloc, m.HeapInuse, m.Sys)

m.Sys is usually closer to total Go-managed memory than Alloc, but still does not include all native allocations.

2. Check cgroup/container memory directly

On Linux containers, inspect cgroup stats to compare runtime metrics with enforced limit.

cat /sys/fs/cgroup/memory.current        # cgroup v2
cat /sys/fs/cgroup/memory.max

In EKS, also inspect pod events:

kubectl describe pod <pod-name>

If reason is OOMKilled, the kernel terminated the process because cgroup memory exceeded limit.

3. Profile non-heap growth with pprof

Expose pprof and analyze memory profiles during peak traffic.

1import _ "net/http/pprof"
2
3func main() {
4    go func() {
5        log.Println(http.ListenAndServe("0.0.0.0:6060", nil))
6    }()
7    // app startup...
8}

Capture heap profiles and compare with RSS trends. If heap remains stable while RSS grows, investigate cgo/native libraries, large buffers, file mappings, or goroutine stacks.

4. Tune Go runtime for container limits

Recent Go versions support memory limit awareness. Set GOMEMLIMIT near but below container memory limit.

1env:
2  - name: GOMEMLIMIT
3    value: "700MiB"
4  - name: GOGC
5    value: "100"

If pod limit is 1Gi, a GOMEMLIMIT around 60-75% often leaves room for non-heap memory. Tune with production load tests, not guesses.

5. Reduce peak allocations and buffer retention

Common causes of memory spikes in Go services:

• Unbounded caches.
• Huge request/response buffering.
• Large slice growth with delayed reuse.
• Goroutine leaks keeping references alive.

Example mitigation with streaming instead of full buffering:

1func proxy(w http.ResponseWriter, r *http.Request) {
2    resp, err := http.DefaultClient.Do(r)
3    if err != nil {
4        http.Error(w, err.Error(), http.StatusBadGateway)
5        return
6    }
7    defer resp.Body.Close()
8
9    w.WriteHeader(resp.StatusCode)
10    io.Copy(w, resp.Body) // stream, avoid loading entire body in memory
11}

6. Set realistic Kubernetes resources

Configure requests and limits based on measured peak behavior:

1resources:
2  requests:
3    memory: "512Mi"
4  limits:
5    memory: "1Gi"

Too-tight limits plus bursty allocations produce recurring OOMKills even when average memory looks safe.

Common Pitfalls

• Treating runtime.MemStats.Alloc as total process memory and ignoring RSS/cgroup metrics.
• Running cgo-heavy dependencies without visibility into native allocations.
• Leaving GOMEMLIMIT unset in constrained containers, allowing heap growth that crowds out non-heap memory.
• Ignoring goroutine and buffer retention patterns that keep large objects alive longer than expected.
• Increasing pod memory limits repeatedly without profiling root memory consumers.

Summary

Low Alloc with high RSS is expected in many Go workloads and does not contradict OOMKills in Kubernetes. Diagnose memory at both runtime and container levels: heap metrics, Sys, pprof, cgroup usage, and pod events. Then tune GOMEMLIMIT, reduce non-heap pressure, and size resources from real load behavior. With this full-stack view, Go memory incidents on EKS become explainable and fixable instead of mysterious.

A practical way to keep this issue from returning is to turn the fix into a lightweight runbook. Capture the exact environment assumptions (tool versions, runtime flags, cluster or platform settings, and required dependencies), then store a short verification command sequence that any teammate can run from a clean setup. This makes troubleshooting deterministic instead of person-dependent and reduces rework during on-call incidents.

It also helps to add one automated guardrail in CI or pre-deploy checks that validates the critical assumption described above. That guardrail might be a linter rule, a smoke test, a schema check, a policy validation step, or a minimal integration test. When the same class of failure is caught before release, teams spend less time on emergency debugging and more time on controlled improvements.