spring-boot-api GitOps

ArgoCD ApplicationSet + Helm chart deploying the Spring Boot API to two Kubernetes clusters (dev/prd). Secrets generated by Helm (randAlphaNum) and preserved across upgrades via lookup. TLS via cert-manager + Let's Encrypt.

---

Repository Structure

spring-boot-api/
├── argocd/
│   ├── argocd-project.yaml       # AppProject scoping the app to its own repo and namespace
│   ├── applicationset.yaml       # Deploys the app to both clusters
│   └── cluster-issuers.yaml      # Tells cert-manager how to issue TLS certs (Let's Encrypt)
├── helm-chart/
│   └── spring-boot-api/
│       ├── Chart.yaml
│       ├── values.yaml           # Shared defaults (all envs inherit these)
│       └── templates/
│           ├── _helpers.tpl         # Shared name/label helpers
│           ├── configmap.yaml       # app config.json mounted as a file
│           ├── secret.yaml          # Helm-generated random credentials (lookup-stable)
│           ├── deployment.yaml      # Pods: rolling update, probes, security, graceful shutdown
│           ├── service.yaml         # ClusterIP, 3 ports (api/logs/soap)
│           ├── ingress.yaml         # nginx ingress, TLS, 3 path routes
│           └── pdb.yaml             # Minimum pods guaranteed during node drains
└── environments/
    ├── dev/values.yaml           # Dev: 3 replicas, staging TLS, DEBUG logging
    └── prd/values.yaml           # Prd: 5 replicas, prod TLS

Separation of concerns: helm-chart/ is how to run the app (owned by platform team). environments/ is what each environment looks like (owned by app team). Different teams, different review gates, same repository.

---

Prerequisites

Infrastructure is provisioned by the terraform repo. After terraform apply completes, apply the ArgoCD config in order:

# 1. Create the AppProject (scopes the app to its own repo/namespace)
kubectl apply -f argocd/argocd-project.yaml

# 2. Create the cert-manager ClusterIssuers (required for TLS)
kubectl apply -f argocd/cluster-issuers.yaml

# 3. Create the ApplicationSet (triggers ArgoCD to deploy the app)
kubectl apply -f argocd/applicationset.yaml

Both clusters must be registered in the central ArgoCD instance. The --name must match clusterName in applicationset.yaml:

argocd cluster add <context-name> --name dev-global-cluster-0
argocd cluster add <context-name> --name prd-global-cluster-5

---

Deploy

kubectl apply -f argocd/applicationset.yaml -n argocd

ArgoCD creates one Application per cluster and syncs automatically. Watch progress:

argocd app list
argocd app get spring-boot-api-dev
argocd app get spring-boot-api-prd

---

How It Works

ArgoCD ApplicationSet (List Generator)

One ApplicationSet creates two Applications, one per cluster:

generators:
  - list:
      elements:
        - env: dev
          clusterName: dev-global-cluster-0   # matches --name from argocd cluster add
          chartRevision: HEAD                 # dev tracks latest chart commit
        - env: prd
          clusterName: prd-global-cluster-5
          chartRevision: v1.0.0               # prd pinned to a tag (explicit promotion)

destination:
  name: "{{ .clusterName }}"   # references cluster by registered ArgoCD name, not URL
  namespace: spring-boot-api

List Generator: Explicit, auditable. You see exactly which clusters exist and what revision each runs.

Pinning prd to a tag: With HEAD, any chart commit would reach prod immediately. A tag means prod only changes when someone bumps the tag in a PR: deliberate promotion, no accidental rollouts.

Multi-Source (values outside the chart directory)

sources:
  - path: helm-chart/spring-boot-api
    targetRevision: "{{ .chartRevision }}"   # chart at tag or HEAD
    helm:
      valueFiles:
        - $values/environments/{{ .env }}/values.yaml
  - ref: values                              # second source acts as $values
    targetRevision: HEAD                     # env values always at HEAD

$values keeps helm-chart/ and environments/ fully independent: separate ownership, separate review gates. Values always come from HEAD so config changes (replicas, hosts) deploy without requiring a chart version bump.

Secret Management (Helm-generated, lookup-stable)

Credentials are generated randomly by Helm on first install and preserved on subsequent upgrades via lookup. The flow:

helm install (first time)
  randAlphaNum 32 generates APP_SECRET_KEY, DB_PASSWORD, JWT_SECRET
          |
          v
  K8s Secret (same name as the app, wave 0)
          |
          | envFrom.secretRef
          v
  Pod environment variables

helm upgrade (subsequent)
  lookup reads existing Secret from the cluster
  -> values are reused unchanged
  -> no credential rotation, no rolling restart

How lookup works: lookup "v1" "Secret" .Release.Namespace <name> returns the live Secret object. dig "data" dict extracts the .data map (empty dict on first install). index $data "KEY" | default (randAlphaNum 32 | b64enc) generates a fresh value only when the key is absent. On all subsequent upgrades the existing base64 value is returned and the default branch never fires.

Sync wave ordering: Secret is wave 0, Deployment is wave 1. ArgoCD applies wave-0 resources first and waits for them to reach Healthy status before applying the Deployment. Without this ordering, pods fail with CreateContainerConfigError because the Secret they reference does not exist yet.

Known limitations of this approach vs a secrets manager:

• helm template has no cluster access; CI dry-runs always generate ephemeral values. The rendered output cannot be applied directly to the cluster.
• Secret values live in Helm release history (stored as Kubernetes Secrets in the release namespace). Anyone with kubectl get secret access can decode them.
• No audit trail for when a value changed or who changed it.
• Rotation requires manually deleting the Secret (or patching it out-of-band) to force regeneration on the next sync. There is no built-in rotation mechanism.

Manual rotation:

# Delete the secret to force Helm to regenerate values on next sync
kubectl delete secret spring-boot-api -n spring-boot-api

# Trigger an ArgoCD sync (or wait for auto-sync)
argocd app sync spring-boot-api-dev

This approach trades operational simplicity (no external dependency, no IRSA setup) for weaker auditability. The limitations above are the known trade-offs of that choice.

TLS (cert-manager + Let's Encrypt)

Ingress annotation: cert-manager.io/cluster-issuer: letsencrypt-staging
  --> cert-manager creates a CertificateRequest
  --> Let's Encrypt sends HTTP-01 challenge to /.well-known/acme-challenge/<token>
  --> cert-manager serves the challenge response via a temporary Ingress
  --> Let's Encrypt verifies ownership and issues the certificate
  --> cert-manager writes the cert+key into spec.tls[].secretName
  --> nginx-ingress reads that Secret for HTTPS termination
  --> cert-manager renews automatically before expiry

Staging vs prod issuer:

• letsencrypt-staging: untrusted cert (browser shows warning), ~3000x higher rate limits. Use for dev and when iterating on TLS config to avoid hitting prod limits.
• letsencrypt-prod: browser-trusted. Rate-limited to 50 certs/week per domain. Switch to this only after staging works end-to-end.

HTTP-01 requirement: The domain must be publicly resolvable and port 80 must be reachable by Let's Encrypt servers. Private/internal domains require the DNS-01 solver instead. See the DNS section below.

Rolling Updates (Zero Downtime)

rollingUpdate:
  maxUnavailable: 0   # never terminate an old pod before its replacement is Ready
  maxSurge: 1         # allow one extra pod during the rollout

Kubernetes default is 25% maxUnavailable, floored. With prd's 5 replicas: floor(5 * 0.25) = 1, allowing one pod to be terminated before its replacement is Ready. Explicit 0 is safe regardless of replica count.

maxSurge: 1 is required: maxUnavailable: 0 + maxSurge: 0 deadlocks the rollout cannot add a pod (no surge) and cannot remove a pod (none available to spare).

Graceful Shutdown

When Kubernetes deletes a pod, two things happen simultaneously:

• The container receives SIGTERM
• The pod is removed from Service endpoints (no new traffic)

Endpoint removal takes a few seconds to propagate through kube-proxy. Without a delay, SIGTERM fires before propagation completes and in-flight requests land on a terminating pod.

preStop:
  exec:
    command: ["sleep 5 && wget -q -O- http://localhost:8080/service/shutdown || true"]

sleep 5 covers the propagation lag. The shutdown endpoint then triggers Spring Boot's graceful shutdown (drains active requests). || true prevents a non-zero exit from blocking termination for the full terminationGracePeriodSeconds (60s).

Probes

Three probe types work together:

• startupProbe: runs during startup only. Liveness and readiness are paused until it passes. Budget: failureThreshold: 30 * periodSeconds: 10 = 5 minutes. Handles slow Spring Boot starts (DB migrations, heavy initialization) without initialDelaySeconds guesswork. Once it passes, it never runs again.
• livenessProbe: "is the process alive?" Failure restarts the container.
• readinessProbe: "should this pod receive traffic?" Failure removes it from Service endpoints without restarting. Pod keeps running but receives no requests.

Because startupProbe handles the startup window, liveness and readiness need no initialDelaySeconds; they activate immediately after startupProbe succeeds.

Security

Pod-level (applies to all containers):

• runAsNonRoot: true: rejects images that run as UID 0.
• runAsUser: 1000: explicit UID, does not rely on image default.
• seccompProfile: RuntimeDefault: enables the container runtime's default syscall filter, blocking ~100 dangerous syscalls. Requires K8s >= 1.22.

Container-level:

• readOnlyRootFilesystem: true: the container cannot write anywhere except explicitly mounted volumes. Spring Boot needs /tmp for Tomcat work files, so an emptyDir is mounted there.
• allowPrivilegeEscalation: false: prevents setuid/setgid privilege escalation.
• capabilities.drop: ["ALL"]: removes all Linux capabilities. A plain HTTP server needs none of them.

High Availability

Replicas: 3 in dev, 5 in prd.

topologySpreadConstraints: distributes pods across nodes so a single node failure loses at most one or two pods. maxSkew: 1 = at most 1 pod difference between any two nodes. DoNotSchedule blocks new pods if the constraint cannot be met; switch to ScheduleAnyway in clusters with fewer nodes than replicas.

PodDisruptionBudget (policy/v1): guarantees minimum 1 pod stays Running during voluntary disruptions (node drains, cluster upgrades).

The critical detail: Deployment's maxUnavailable: 0 only governs rollouts triggered by the Deployment controller. Node evictions use the Eviction API directly against the ReplicaSet, bypassing the Deployment strategy entirely. Without a PDB, all pods can be evicted simultaneously during a kubectl drain, regardless of maxUnavailable.

policy/v1 requires K8s >= 1.21. policy/v1beta1 was removed in K8s 1.25.

---

Assumptions

Decision	Choice	Reason
Image	busybox:stable	Placeholder image. Override image.repository and image.tag in environment values.
Image pull policy	Always	Mutable tag stable requires a fresh registry check on every pod start. Use IfNotPresent only with immutable tags (SHA digests).
ArgoCD generator	List	Explicit, auditable. Every cluster and its revision is visible in one place.
Chart revision (dev)	HEAD	Fast feedback: chart changes reach dev immediately.
Chart revision (prd)	tag pin	No accidental rollouts. Prod changes on explicit version bump.
Values revision	HEAD	Config changes (replicas, hosts) deploy without a chart version bump.
Namespace	spring-boot-api	Created automatically by ArgoCD CreateNamespace=true.
ArgoCD project	spring-boot	Logical grouping. Must exist in ArgoCD before the ApplicationSet is applied.
Ingress class	nginx	Standard. Change via ingress.className value.
Secret storage	Helm randAlphaNum + lookup	Generated on first install, preserved on upgrades. No external dependency. See limitations in Secret Management section.
TLS issuer (dev)	letsencrypt-staging	High rate limits. Safe for iteration. Cert is browser-untrusted.
TLS issuer (prd)	letsencrypt-prod	Browser-trusted. Rate limited. Use after staging validates.
PDB maxUnavailable	1	At most 1 pod evicted at a time. minAvailable:1 with replicaCount:1 deadlocks node drains.

---

DNS

cert-manager's HTTP-01 challenge requires the app hostnames to be publicly resolvable and port 80 reachable by Let's Encrypt servers before TLS issuance works. Create DNS A records pointing to the nginx ingress LoadBalancer IP before deploying:

• api.dev.inpost.pl -> ingress LoadBalancer (dev cluster)
• api.prd.inpost.pl -> ingress LoadBalancer (prd cluster)

---

Local Chart Testing

# Render templates without deploying
helm template dev helm-chart/spring-boot-api -f environments/dev/values.yaml
helm template prd helm-chart/spring-boot-api -f environments/prd/values.yaml

# Lint for errors
helm lint helm-chart/spring-boot-api -f environments/dev/values.yaml

---

FAQ

Why Helm lookup + randAlphaNum instead of External Secrets Operator?

Operational simplicity. ESO requires an OIDC provider, an IRSA IAM role, a SecretStore manifest, and an ExternalSecret manifest per secret - all of which need to exist and be healthy before the first pod starts. lookup has no external dependencies and no operator to run. The trade-offs are documented in the Secret Management section above: no audit trail, rotation is a manual delete + sync, and CI dry-runs (helm template) always generate ephemeral values. For production workloads with compliance requirements, ESO + AWS Secrets Manager is the right answer. For this scope, the simpler approach was chosen deliberately.

Why maxUnavailable: 0 instead of the Kubernetes default (25%)?

With 5 production replicas, floor(5 x 0.25) = 1 - the default still allows one pod to be terminated before its replacement is Ready, creating a brief capacity dip. Explicit 0 guarantees no pod is removed until its replacement passes the readiness probe, regardless of replica count. It requires maxSurge >= 1 to avoid deadlock (can't add a pod, can't remove a pod).

Why is there a startupProbe in addition to liveness and readiness?

Spring Boot startup (DB migrations, bean initialization) can take 30-60 seconds. Without a startup probe you either set a large initialDelaySeconds on liveness (guesswork, wastes time on fast restarts) or risk the liveness probe killing a healthy pod that is still initializing. The startup probe suspends liveness and readiness until the port opens, giving the app a 5-minute budget (failureThreshold: 30 x periodSeconds: 10) with no guesswork required.

Why is preStop needed if Kubernetes already sends SIGTERM?

Pod deletion triggers two async operations simultaneously: SIGTERM to the container, and removal of the pod from Service endpoints. Endpoint removal takes a few seconds to propagate through kube-proxy. Without the sleep 5 delay, SIGTERM fires before propagation completes and in-flight requests hit a terminating pod. The sleep covers this lag before the shutdown endpoint is called.

How do I rotate a secret?

kubectl delete secret <release-name> -n spring-boot-api
argocd app sync spring-boot-api-<env>

ArgoCD triggers a Helm upgrade. lookup finds no existing Secret, randAlphaNum generates fresh values, and the Deployment rolls out because the checksum annotation changes.

Why are values in environments/ and the chart in helm-chart/ in the same repo?

The requirement allows "different place or repository." Using a single repo with separate paths satisfies the intent: chart and values have independent revision tracking. The chart is pinned to a tag in production (chartRevision: v1.0.0) while values always track HEAD. Config changes (replica count, hostnames) deploy without bumping the chart version. In a larger team this would split into two repos with separate ownership and review gates.