Node Pool Upgrade: g4s.kube.medium to g4s.kube.large

Upgrade the Civo Kubernetes node pool from g4s.kube.medium (2 vCPU, 4GB RAM) to g4s.kube.large (4 vCPU, 8GB RAM) for the SpektraBot pilot cluster.

Why: Postgres OOM'd on 2026-02-10 with a 1Gi memory limit. The larger node provides headroom for increased Postgres memory (2Gi limit, 512MB shared_buffers) and future growth.

Downtime: Expect 5-10 minutes during drain and reschedule. Schedule during low-usage period.

---

Prerequisites

1. Verify current node pool:

   civo kubernetes show spektrabot-pilot -o json | jq '.pools'

   Find the id field in the JSON output and export it:

   export OLD_POOL_ID="<your-old-pool-id>"

2. Take a fresh Postgres backup before starting:

   kubectl create job --from=cronjob/pg-backup-s3 manual-pre-upgrade -n spektrabot-pilot

3. Verify backup completed successfully:

   kubectl logs job/manual-pre-upgrade -n spektrabot-pilot

   Look for "Backup completed successfully" and verify the S3 upload size is > 1MB.

4. Note: Civo does NOT support in-place node resizing. You must create a new pool, migrate workloads, then delete the old pool.

---

Procedure

Step 1: Create new node pool with larger nodes

civo kubernetes node-pool create spektrabot-pilot \
  --size g4s.kube.large \
  --count 1 \
  --wait

The new pool ID will be shown in the command output. Export it:

export NEW_POOL_ID="<your-new-pool-id>"

Step 2: Wait for new node to be Ready

kubectl get nodes -w

Wait until the new node shows Ready status. This typically takes 1-3 minutes.

Step 3: Cordon the old node

Prevent new pods from scheduling on the old node(s):

kubectl cordon -l "civo.com/node-pool=$OLD_POOL_ID"

Step 4: Drain the old node

Evict all pods gracefully:

kubectl drain -l "civo.com/node-pool=$OLD_POOL_ID" \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --grace-period=120 \
  --timeout=300s

This will cause brief downtime as pods reschedule to the new node.

Step 5: Wait for all pods to be Running on new node

kubectl get pods -n spektrabot-pilot -o wide -w

Watch for the CNPG postgres pod -- it may take longer due to PVC reattachment. All pods should show Running status within 2-5 minutes.

Step 6: Verify PVC reattachment

kubectl get pvc -n spektrabot-pilot
kubectl get volumeattachments

If a PVC is stuck (FailedAttachVolume event), check for a stale VolumeAttachment from the old node and delete it:

# List volume attachments and find the one referencing the old node
kubectl get volumeattachments -o wide

# Delete the stale attachment
kubectl delete volumeattachment <stale-attachment-name>

The pod will then reattach the volume to the new node automatically.

Step 7: Verify services are healthy

kubectl get pods -n spektrabot-pilot
curl -s https://spektrabot.co.uk/api/health | jq .

All pods should be Running and the health endpoint should return a 200 response.

Step 8: Apply updated Helm values

Deploy right-sized resources and postgres tuning for the larger node:

helm upgrade spektrabot ./helm/spektrabot \
  -f ./helm/spektrabot/values-pilot.yaml \
  -n spektrabot-pilot \
  --set certManager.enabled=false

Note: CNPG postgres parameter changes trigger a rolling restart of the postgres pod. This causes a brief (~30s) postgres restart.

Step 9: Verify postgres picked up new parameters

PG_POD=$(kubectl get pods -n spektrabot-pilot \
  -l 'cnpg.io/cluster=spektrabot-postgres,cnpg.io/role=primary' \
  -o jsonpath='{.items[0].metadata.name}')

if [ -z "$PG_POD" ]; then
  echo "Error: Primary postgres pod not found. Check CNPG cluster status." >&2
else
  kubectl exec "$PG_POD" -c postgres -n spektrabot-pilot -- \
    psql -U spektra -d spektrabot -c "SHOW shared_buffers; SHOW maintenance_work_mem;"
fi

Expected output:

 shared_buffers
----------------
 512MB

 maintenance_work_mem
----------------------
 256MB

Step 10: Delete old node pool

Once everything is confirmed working:

civo kubernetes node-pool delete spektrabot-pilot "$OLD_POOL_ID" --yes

Step 11: Take post-upgrade backup

kubectl create job --from=cronjob/pg-backup-s3 manual-post-upgrade -n spektrabot-pilot

Verify it completed: kubectl logs job/manual-post-upgrade -n spektrabot-pilot

---

Rollback

If the new node has issues before deleting the old pool:

1. Uncordon the old node(s):

   kubectl uncordon -l "civo.com/node-pool=$OLD_POOL_ID"

2. Delete the new pool:

   civo kubernetes node-pool delete spektrabot-pilot "$NEW_POOL_ID" --yes

3. Pods will reschedule back to the old node automatically.

If old pool was already deleted: Create a new pool with the original size:

civo kubernetes node-pool create spektrabot-pilot \
  --size g4s.kube.medium \
  --count 1 \
  --wait

---

Expected Resource Budget on g4s.kube.large (4 vCPU, 8GB RAM)

Service	Memory Request	Memory Limit	CPU Request	CPU Limit
Postgres	768Mi	2Gi	200m	500m
API	256Mi	512Mi	100m	400m
Web	128Mi	384Mi	50m	200m
Redis	64Mi	128Mi	25m	100m
Umami	192Mi	384Mi	100m	300m
Total requests	1408Mi	-	475m	-
System/DaemonSets	~500Mi	-	~200m	-
Available	~6.1Gi	8Gi	~3.3	4

Headroom: ~4.2Gi memory and ~2.6 CPU cores free for spikes and system processes.