updating missing bitnami images

Quick Navigation

• Context
• Primary Symptoms
• Root Cause Pattern
• Upstream Distribution Context (Bitnami/Broadcom, 2025)
• Incident Sequence
• Terms Used In This Incident
  • ImagePullBackOff
  • Multi-Attach (RWO PVC)
  • Recreate Rollout Strategy
• Important Operational Guardrail
• Step 1: Update WordPress/MariaDB Images In Values
• Step 2: Force Argo Sync And Verify Templates
• Step 3: Resolve PVC Rollout Deadlock
• Symptom Detection Commands
• Exact Recovery Commands (Generic)
• Validation Checklist
• Follow-Up

Context

This runbook captures WordPress incident recovery where:

• wordpress-mariadb-0 failed with Init:ImagePullBackOff
• WordPress app pods failed to pull bitnami/wordpress tags
• rollout got stuck because old and new ReplicaSets competed for one RWO PVC

This workflow assumes WordPress is managed by Argo CD + Helm and values are stored in Git.

Primary Symptoms

• Init:ImagePullBackOff on wordpress-mariadb-0
• Failed to pull image ... not found for bitnami/* images
• Multi-Attach error ... volume is already used by pod
• WordPress deployment template updated, but old ReplicaSet pod still running old image
• Argo app shows synced/progressing while pods still fail startup

Root Cause Pattern

There were two related issues:

1. Bitnami removed/retired old tags from the main bitnami/* namespace and kept them under bitnamilegacy/*.
   • Our chart asked for fixed old tags in docker.io/bitnami/....
   • Docker returned not found for those tags because they were no longer published there.
   • The same versions were still available in docker.io/bitnamilegacy/..., which is why switching repository fixed pulls.
   • Why they did this (practical reason): keep the main namespace focused on actively maintained images, while legacy tags stay in a separate namespace for backward compatibility.
2. Rolling update deadlock with a single RWO PVC:
   • old pod keeps PVC attached
   • new pod cannot attach/start
   • rollout remains stuck with mixed ReplicaSets.

Upstream Distribution Context (Bitnami/Broadcom, 2025)

The image pull issue in this incident is tied to a broader upstream packaging/distribution change:

• Bitnami (under Broadcom ownership) changed how public images/tags are distributed.
• Many previously used version-pinned tags were no longer available under docker.io/bitnami/*.
• Legacy-compatible tags remained available under docker.io/bitnamilegacy/*.
• Production-oriented maintained image channels were shifted toward paid/commercial offerings.

Operational impact for clusters like this one:

• Pinned bitnami/*:<version> tags can return not found.
• Existing Helm/Argo configs can fail at runtime with ErrImagePull / ImagePullBackOff even when manifests remain syntactically valid.
• Recovery often requires repository/tag migration in values plus controlled rollout handling.

Incident Sequence

• Image source changed for pinned tags and pulls started failing.
• Argo updated templates, but rollout overlapped old/new pods.
• Old pod held the RWO PVC, so new pod could not attach.
• Result was a second failure mode (Multi-Attach) after image-pull failures.

Terms Used In This Incident

ImagePullBackOff

Kubernetes repeatedly tries to pull the container image and backs off after failures.
Typical reasons: image tag missing, registry access failure, auth issues.

Multi-Attach (RWO PVC)

A PVC with ReadWriteOnce can be mounted by one node/pod at a time.
If an old pod still holds the volume, a new pod scheduled elsewhere gets Multi-Attach failure.

Recreate Rollout Strategy

Recreate stops old pods before creating new ones.
For single-writer volumes (RWO), this avoids old/new pod overlap and attach deadlocks.

Important Operational Guardrail

Keep fixes Git/Argo-first:

• update values in repo
• force sync Argo app
• verify rendered templates
• only then restart pods if necessary.

Avoid ad-hoc drift unless explicitly needed for emergency rollback.

Step 1: Update WordPress/MariaDB Images In Values

Update /home/orest/repos/k3s/wordpress/values.yaml with valid legacy images:

mariadb:
  image:
    registry: docker.io
    repository: bitnamilegacy/mariadb
    tag: 11.4.4-debian-12-r4
 
image:
  registry: docker.io
  repository: bitnamilegacy/wordpress
  tag: 6.7.0-debian-12-r4

Step 2: Force Argo Sync And Verify Templates

Force a hard refresh and sync so Argo re-renders Helm with latest values.

Use commands from Exact Recovery Commands (Generic).

Verification for step completion:

• StatefulSet wordpress-mariadb uses bitnamilegacy/mariadb:*
• Deployment wordpress uses bitnamilegacy/wordpress:*

Step 3: Resolve PVC Rollout Deadlock

If old + new WordPress pods overlap and one pod blocks the RWO PVC:

1. Set deployment strategy to Recreate
2. Delete current WordPress pods
3. Let deployment create one clean pod from latest template

Then verify:

• one active WordPress pod
• 1/1 Running
• no recurring Multi-Attach warning events.

Symptom Detection Commands

Argo application state

kubectl -n argocd get application wordpress -o wide
kubectl -n argocd get application wordpress -o jsonpath='sync={.status.sync.status}{"\n"}health={.status.health.status}{"\n"}op={.status.operationState.phase}{"\n"}'

MariaDB pod details

kubectl -n wordpress get pod wordpress-mariadb-0 -o wide
kubectl -n wordpress describe pod wordpress-mariadb-0

Image pull failures

kubectl -n wordpress get events --sort-by=.lastTimestamp | rg 'Failed to pull image|ErrImagePull|ImagePullBackOff|BackOff'

Multi-attach failures

kubectl -n wordpress get events --sort-by=.lastTimestamp | rg 'Multi-Attach|FailedAttachVolume'

ReplicaSets and pod images (old vs new)

kubectl -n wordpress get rs -l app.kubernetes.io/name=wordpress -o wide
kubectl -n wordpress get pods -l app.kubernetes.io/name=wordpress -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.initContainers[0].image}{"\t"}{.status.phase}{"\n"}{end}'

Exact Recovery Commands (Generic)

# 0) Namespace shortcut
kubectl config set-context --current --namespace=wordpress
 
# 1) Force Argo app refresh + sync
kubectl -n argocd annotate application wordpress argocd.argoproj.io/refresh=hard --overwrite
kubectl -n argocd patch application wordpress --type merge -p '{"operation":{"sync":{"syncStrategy":{"apply":{"force":true}},"prune":true}}}'
 
# 2) Verify rendered workload templates now use expected images
kubectl -n wordpress get statefulset wordpress-mariadb -o jsonpath='init={.spec.template.spec.initContainers[0].image}{"\n"}db={.spec.template.spec.containers[0].image}{"\n"}'
kubectl -n wordpress get deployment wordpress -o jsonpath='init={.spec.template.spec.initContainers[0].image}{"\n"}app={.spec.template.spec.containers[0].image}{"\n"}'
 
# 3) If WordPress rollout is stuck with old pod holding RWO PVC:
kubectl -n wordpress patch deployment wordpress --type='json' -p='[{"op":"remove","path":"/spec/strategy/rollingUpdate"},{"op":"replace","path":"/spec/strategy/type","value":"Recreate"}]'
kubectl -n wordpress delete pod -l app.kubernetes.io/name=wordpress
 
# 4) Watch startup
kubectl -n wordpress get pods -l app.kubernetes.io/name=wordpress -w

Validation Checklist

• wordpress-mariadb-0 is 1/1 Running
• WordPress app pod is 1/1 Running
• No repeating ImagePullBackOff for bitnami/* legacy-migrated pods
• No repeating Multi-Attach warnings for WordPress PVC
• Argo app status:
  • sync: Synced
  • health: Healthy (or steadily converging without failures)

Follow-Up

• Keep image overrides in Git values and avoid manual-only cluster patches.
• If using RWO app PVC with single replica app, keep Recreate strategy to avoid overlap.
• Monitor for drift between Argo desired state and running workloads after chart upgrades.
• Consider documenting approved bitnamilegacy/* tags per app to reduce on-call guesswork.

Related:

• k3s

Domain

• runbook