Skip to main content
Version: 2.3

Microservice Deployment Settings

Most of our microservices are Spring Boot Java applications with Actuator monitoring enabled. This guide describes recommended CPU and memory settings, probe configuration, monitoring practices with Grafana, and troubleshooting steps with kubectl.

CPU Settings​

Principles​

  • Limit up to 4 CPUs Helps startup time (faster JIT warm-up and class loading). ⚠️ If all services start in parallel this may throttle the node – but in production we always use rolling updates, so this is not a problem.

  • Constrained CPU (1 vCPU) Works fine, but you must extend the startupProbe timeout/threshold, because startup will be slower.

  • Normal operation Microservices typically consume very little CPU most of the time. → Set requests.cpu: 100m.

Kubernetes Example​

resources:
  requests:
    cpu: "100m"    # low steady consumption
  limits:
    cpu: "4"       # allow faster startup

Memory Settings​

What Java Memory Consists Of​

  • Heap (-Xmx) – application objects.
  • Metaspace – typically ~300 MB.
  • Thread stacks – ~1 MB per thread (~100 MB for ~100 threads).
  • JVM internals & native buffers – ~200 MB (GC structures, NIO, Netty, glibc arenas).

👉 Rule of thumb: Pod memory limit = Xmx + ~600 MB.

Guidelines​

  • Set both -Xms and -Xmx to the same value to avoid dynamic heap resizing.
  • at least 200 MB for small microservices, 300 MB for microservices with process engine
  • at most 1-5 GBs for large workloads. For larger workloads scale horizontally (add more replicas).
  • Always leave headroom above -Xmx for Metaspace, threads, and native overhead.
  • You should never see OOMKilled. If it happens → either increase memory limit or even decrease -Xmx!.
  • Use Grafana monitoring to calibrate real usage.

Kubernetes Example​

resources:
  requests:
    memory: "900Mi"    # Example: Xmx(300Mi) + ~600Mi overhead
  limits:
    memory: "1200Mi"   # Add 300Mi headroom for spikes. For safety, avoid OOMKilled.

JVM Options Example​

env:
  - name: JAVA_TOOL_OPTIONS
    value: -Xms300m -Xmx300m

Probes (Startup, Liveness, Readiness)​

Spring Boot with Actuator provides endpoints for Kubernetes probes.

Recommendations​

  • Startup Probe: /actuator/health Allows slow startup without failing the pod. Increase failureThreshold if CPU is constrained.
  • Liveness Probe: /actuator/health/liveness Detects JVM deadlocks or crash states.
  • Readiness Probe: /actuator/health/readiness Ensures the pod only receives traffic when ready.

Typical Settings​

  • Startup Probe: initialDelaySeconds: 20, periodSeconds: 10, failureThreshold: 30
  • Liveness/Readiness Probe: periodSeconds: 5, timeoutSeconds: 5, failureThreshold: 5

Example Deployment Descriptor​

A typical Spring Boot microservice (tsm-catalog) Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: tsm-catalog
  name: tsm-catalog
  namespace: tsm-datalite
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: tsm-catalog
  template:
    metadata:
      annotations:
        linkerd.io/inject: disabled
      labels:
        app: tsm-catalog
    spec:
      containers:
        - name: tsm-catalog
          image: registry.datalite.cz/tsm/tsm-catalog:2.3
          imagePullPolicy: Always
          env:
            - name: JAVA_TOOL_OPTIONS
              value: >
                -Xms300m -Xmx300m
            - name: spring.config.import
              value: configserver:http://tsm-config-server
            - name: MALLOC_ARENA_MAX
              value: "2"
          ports:
            - containerPort: 8099
              protocol: TCP
          resources:
            requests:
              cpu: 100m
              memory: 800Mi
            limits:
              cpu: 3000m
              memory: 1100Mi
          livenessProbe:
            httpGet:
              path: /actuator/health/liveness
              port: 8099
            initialDelaySeconds: 0
            periodSeconds: 5
            timeoutSeconds: 5
            successThreshold: 1
            failureThreshold: 5
          readinessProbe:
            httpGet:
              path: /actuator/health/readiness
              port: 8099
            initialDelaySeconds: 0
            periodSeconds: 5
            timeoutSeconds: 5
            successThreshold: 1
            failureThreshold: 5
          startupProbe:
            httpGet:
              path: /actuator/health
              port: 8099
            initialDelaySeconds: 40
            periodSeconds: 10
            timeoutSeconds: 1
            failureThreshold: 30
      imagePullSecrets:
        - name: regcred
      securityContext:
        runAsUser: 1001
        runAsGroup: 1001

Monitoring with Grafana​

  • JVM metrics (via Actuator/Micrometer):

    • jvm.memory.used{area=heap} vs. jvm.memory.max{area=heap}
    • jvm.memory.used{area=nonheap}
    • jvm.threads.live

  • Pod metrics (via cAdvisor / kubelet):

    • Pod memory usage (RSS / working set)
    • Pod CPU usage and throttling

👉 Always compare JVM vs Pod metrics: if Pod RSS ≫ JVM heap+nonheap → overhead from threads, native memory, or allocator arenas.

Troubleshooting with kubectl​

Check resource usage​

kubectl top pod <pod> -n <namespace>

Inspect pod state and events​

kubectl describe pod <pod> -n <namespace>

Access logs​

kubectl logs <pod> -n <namespace>

Exec into pod​

kubectl exec -it <pod> -n <namespace> -- sh

Check memory details​

cat /proc/1/smaps_rollup

In case you need detailed native memory tracking (NMT) for troubleshooting, add: -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary and after restart add jcmd 1 VM.native_memory detail.

Or with JDK tools:

jcmd 1 VM.native_memory summary

Checklist​

  • CPU requests = 100m, limits = 4 (or 1 if constrained + probe adjusted).
  • Memory limit = Xmx + ~600 MB overhead.
  • JVM options include -Xms -Xmx and optional NMT.
  • Optionally MALLOC_ARENA_MAX=2 set to reduce native memory overhead.
  • Probes configured (startup, readiness, liveness).
  • Monitor JVM and Pod metrics in Grafana.
  • Never accept OOMKilled events.