Monitoring & Observability Guide

SpektraBot pilot cluster monitoring stack: what's deployed, how to access it, and what to do when alerts fire.

Cluster: Civo Kubernetes (2 nodes, k3s) Namespace: monitoringHelm charts: helm/monitoring/, helm/loki/, helm/alloy/

Architecture Overview

                    +------------------+
                    |     Grafana      |  Dashboards, Explore (logs)
                    |   (port 3000)    |
                    +--------+---------+
                             |
              +--------------+--------------+
              |                             |
     +--------v---------+        +---------v--------+
     |    Prometheus     |        |      Loki        |
     |  (metrics, 30d)  |        |  (logs, 30d)     |
     |   15Gi PVC       |        |   20Gi PVC       |
     +--------+---------+        +---------+--------+
              |                             |
    +---------+---------+          +--------+--------+
    |                   |          |                  |
+---v---+  +----------+ |   +------v------+  +-------v------+
| node- |  | kube-    | |   | Alloy       |  | Alloy        |
|exporter|  | state-   | |   | (node 1)    |  | (node 2)     |
| (x2)  |  | metrics  | |   | DaemonSet   |  | DaemonSet    |
+-------+  +----------+ |   +-------------+  +--------------+
                         |
              +----------v---------+
              |   AlertManager     |
              |  PagerDuty routing |
              |   2Gi PVC         |
              +--------------------+

Components

Component	Pods	Purpose	Storage
Prometheus	1 StatefulSet	Metrics collection & alerting	15Gi PVC (30d retention)
Grafana	1 Deployment	Dashboards & log exploration	1Gi PVC
AlertManager	1 StatefulSet	Alert routing to PagerDuty	2Gi PVC
Loki	1 StatefulSet	Log aggregation (monolithic)	20Gi PVC (30d retention)
Alloy	2 DaemonSet pods	Log collection from all nodes	None (streams to Loki)
kube-state-metrics	1 Deployment	Kubernetes object metrics	None
node-exporter	2 DaemonSet pods	Node hardware metrics	None
Prometheus Operator	1 Deployment	Manages Prometheus CRDs	None

Total: 13 pods, 38Gi storage, ~1.5Gi memory requests

Access

All services use ClusterIP. Access via kubectl port-forward:

Grafana (dashboards and logs)

bash

kubectl port-forward svc/spektrabot-monitoring-grafana 3000:80 -n monitoring

Open http://localhost:3000

Credentials: admin / admin (change on first login)

Prometheus (metrics and PromQL)

bash

kubectl port-forward svc/spektrabot-monitoring-kube-prometheus 9090:9090 -n monitoring

Open http://localhost:9090

AlertManager (alert routing and silences)

bash

kubectl port-forward svc/spektrabot-monitoring-kube-alertmanager 9093:9093 -n monitoring

Open http://localhost:9093

Loki (log queries via Grafana)

Loki is accessed through Grafana's Explore view. No separate port-forward needed.

Open Grafana
Click Explore (compass icon) in the left sidebar
Select Loki from the datasource dropdown
Use LogQL to query logs (see Log Queries below)

Dashboards

Built-in (from kube-prometheus-stack)

These are auto-provisioned and cover general Kubernetes monitoring:

Kubernetes / Compute Resources / Cluster -- cluster-wide CPU, memory, network
Kubernetes / Compute Resources / Namespace (Pods) -- per-namespace breakdown
Kubernetes / Compute Resources / Pod -- individual pod metrics
Kubernetes / Persistent Volumes -- PVC usage
Node Exporter / Nodes -- per-node hardware metrics
Prometheus / Overview -- Prometheus self-monitoring

Custom dashboards

Dashboard	ConfigMap	Key panels
Application Performance	`dashboard-app-performance`	API latency (p50/p95/p99), request rate, error rate, WebSocket connections, LLM latency, LLM tokens, pgvector search latency
Database Health	`dashboard-db-health`	Postgres connections vs max, cache hit ratio, WAL archive status, replication lag, slow queries, connection usage, DB size

Custom dashboards are provisioned via ConfigMaps with grafana_dashboard: "1" label, auto-loaded by Grafana's sidecar.

Alert Rules

Custom alerts (9 rules in 2 PrometheusRule CRDs)

Infrastructure alerts (`core-alerts`)

Alert	Severity	Threshold	For	Meaning
APIDown	critical	`up{job="spektrabot-api"} == 0`	1m	API is unreachable
PostgresDown	critical	`cnpg_pg_postmaster_start_time_seconds == 0`	1m	Postgres pod is down
PodCrashLoopBackOff	critical	`kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"} > 0`	2m	A pod is crash-looping
NodeCPUHigh	warning	`>85%` avg CPU per node	10m	Node CPU saturated
NodeMemoryHigh	warning	`>85%` memory used per node	5m	Node memory saturated
PVCUsageHigh	warning	`>85%` PVC used	5m	Storage approaching capacity
PVCFull	critical	`>95%` PVC used	1m	Storage nearly full

Application alerts (`app-alerts`)

Alert	Severity	Threshold	For	Meaning
APIHighErrorRate	critical	`>1%` HTTP 5xx rate	2m	API returning too many errors
WebSocketMassDisconnect	critical	`>50%` disconnect rate AND `>5` active connections	2m	Mass WebSocket disconnection event

PagerDuty integration

Critical alerts route to PagerDuty via Events API v2. The routing key is injected at deploy time (never stored in the repo):

bash

helm upgrade spektrabot-monitoring helm/monitoring \
  -f helm/monitoring/values-civo.yaml \
  --set 'pagerdutyRoutingKey=YOUR_KEY' \
  -n monitoring

Without the key, AlertManager runs with a null receiver (alerts visible in UI but no external routing).

Runbooks

APIDown

Impact: Users cannot access SpektraBot. All API and WebSocket functionality is offline.

Investigate:

bash

# Check pod status
kubectl get pods -n spektrabot-pilot -l app.kubernetes.io/name=spektrabot-api

# Check recent events
kubectl describe pod <api-pod> -n spektrabot-pilot

# Check logs from the previous container (if crash-restarting)
kubectl logs <api-pod> -n spektrabot-pilot --previous

# Check if deployment was recently updated
kubectl rollout history deployment/spektrabot-api -n spektrabot-pilot

Resolve:

If OOMKilled: increase memory limits in deployment
If CrashLoopBackOff: check logs for startup errors (missing env vars, DB connection failure)
If recent deployment: rollback with kubectl rollout undo deployment/spektrabot-api -n spektrabot-pilot
If node issue: check node health with kubectl get nodes

APIHighErrorRate

Impact: Users experiencing errors. Some requests failing.

Investigate:

bash

# Check API logs for errors (via Grafana Explore with Loki)
# LogQL: {namespace="spektrabot-pilot", app="spektrabot-api"} |~ "error|Error|ERROR"

# Check if specific endpoints are failing (Prometheus)
# PromQL: rate(http_request_errors_total{job="spektrabot-api"}[5m])

# Check downstream services
kubectl get pods -n spektrabot-pilot

Resolve:

If Postgres-related: check PostgresDown alert, verify DB connectivity
If Azure OpenAI API errors: check LLM call metrics, rate limits
If recent deployment: rollback
If memory pressure: check pod resource usage

PostgresDown

Impact: All database operations fail. API will return 500 errors for most endpoints.

Investigate:

bash

# Check CNPG cluster status
kubectl get clusters -n spektrabot-pilot

# Check Postgres pod
kubectl get pods -n spektrabot-pilot -l cnpg.io/cluster=spektrabot-db

# Check operator logs
kubectl logs -n spektrabot-pilot -l app.kubernetes.io/name=cloudnative-pg --tail=50

# Check PVC usage (Postgres data volume)
kubectl exec -n spektrabot-pilot <pg-pod> -- df -h /var/lib/postgresql/data

Resolve:

If OOMKilled: increase Postgres memory limits (currently 2Gi limit)
If PVC full: expand volume or clean WAL logs
If node failure: CNPG operator should auto-failover; check operator logs
If data corruption: restore from S3 backup (see backup runbook)

PodCrashLoopBackOff

Impact: Affected service is unstable. Pod repeatedly crashing and restarting.

Investigate:

bash

# Identify which pod
kubectl get pods --all-namespaces --field-selector=status.phase!=Running

# Check pod events
kubectl describe pod <pod-name> -n <namespace>

# Check logs from the crashing container
kubectl logs <pod-name> -n <namespace> --previous

Resolve:

Check for missing environment variables or ConfigMap changes
Check for resource constraints (OOMKilled shows in events)
Check for dependency failures (DB not ready, service unavailable)
If recent deployment: rollback

NodeCPUHigh

Impact: Pod scheduling may be affected. Running pods may be throttled.

Investigate:

bash

# Check which pods are consuming CPU
kubectl top pods --all-namespaces --sort-by=cpu | head -20

# Check node utilisation
kubectl top nodes

# Check for pending pods
kubectl get pods --all-namespaces --field-selector=status.phase=Pending

Resolve:

Identify CPU-heavy pods and check for runaway processes
Reduce non-essential workloads (pause crawlers if running)
If sustained: upgrade node pool to larger instance size (see docs/runbooks/node-pool-upgrade.md)

NodeMemoryHigh

Impact: OOMKill risk. Kubernetes may evict pods to free memory.

Investigate:

bash

# Check which pods are consuming memory
kubectl top pods --all-namespaces --sort-by=memory | head -20

# Check node memory
kubectl top nodes

# Check for OOMKill events
kubectl get events --all-namespaces --field-selector reason=OOMKilling

Resolve:

Identify memory-heavy pods; check for leaks (growing memory over time)
Restart leaking pods as a short-term fix
Adjust memory limits for overconsuming pods
If sustained: upgrade node pool (see docs/runbooks/node-pool-upgrade.md)

PVCUsageHigh

Impact: Storage approaching capacity. If unaddressed, will lead to PVCFull.

Investigate:

bash

# Check all PVC usage
kubectl get pvc -n monitoring
kubectl get pvc -n spektrabot-pilot

# Check specific PVC usage inside pod
kubectl exec -n monitoring prometheus-spektrabot-monitoring-kube-prometheus-0 -- df -h
kubectl exec -n monitoring loki-0 -- df -h

PVC inventory:

PVC	Size	Service	Retention
Prometheus data	15Gi	Metrics	30d / 14GB
Loki data	20Gi	Logs	30d
Grafana	1Gi	Dashboard state	N/A
AlertManager	2Gi	Silence/nflog state	N/A

Resolve:

Prometheus PVC: Reduce retention (retention: 15d) or reduce cardinality
Loki PVC: Reduce log retention or add exclusion filters in Alloy config
Either: Expand PVC size via Civo dashboard (civo-volume supports expansion)
Check for stuck compaction jobs in Loki logs

PVCFull

Impact: Service write failures. Prometheus stops ingesting, Loki stops accepting logs.

Investigate: Same as PVCUsageHigh but urgent.

Resolve (immediate):

Prometheus: Reduce retention immediately: edit values-civo.yaml, set retention: 7d, redeploy
Loki: Force compaction: kubectl exec -n monitoring loki-0 -- wget -qO- http://localhost:3100/compactor/ring
Expand PVC: Civo supports online volume expansion; increase via dashboard then patch PVC
Emergency: Delete old data manually (last resort)

WebSocketMassDisconnect

Impact: Users disconnected from real-time chat. Messages won't be delivered until reconnection.

Note: This alert only fires when there are more than 5 active connections (avoids false positives during low-traffic periods).

Investigate:

bash

# Check API pod health
kubectl get pods -n spektrabot-pilot -l app.kubernetes.io/name=spektrabot-api

# Check for recent deployments or restarts
kubectl rollout history deployment/spektrabot-api -n spektrabot-pilot

# Check WebSocket metrics in Prometheus
# PromQL: websocket_connections_active
# PromQL: increase(websocket_disconnects_total[5m])

# Check node network
kubectl get nodes -o wide

Resolve:

If API pod restarted: connections drop during restart, should auto-recover
If node issue: check node health and network connectivity
If ingress change: verify WebSocket upgrade headers in ingress config
If sustained: check API logs for WebSocket error patterns

Log Queries

Access logs via Grafana Explore with Loki datasource.

Common LogQL queries

logql

# All API logs
{namespace="spektrabot-pilot", app="spektrabot-api"}

# Error logs only
{namespace="spektrabot-pilot", app="spektrabot-api"} |~ "error|Error|ERROR"

# Logs from a specific pod
{namespace="spektrabot-pilot", pod="spektrabot-api-abc123"}

# All monitoring namespace logs
{namespace="monitoring"}

# Postgres logs
{namespace="spektrabot-pilot", app="spektrabot-db"}

# Rate of error logs per minute
rate({namespace="spektrabot-pilot"} |~ "error" [1m])

Maintenance

Deploying changes

The monitoring stack is deployed via three Helm charts:

bash

# Monitoring stack (Prometheus, Grafana, AlertManager, alerts, dashboards)
helm upgrade spektrabot-monitoring helm/monitoring \
  -f helm/monitoring/values-civo.yaml \
  --set 'pagerdutyRoutingKey=YOUR_KEY' \
  -n monitoring

# Loki (log aggregation)
helm upgrade loki helm/loki \
  -f helm/loki/values-civo.yaml \
  -n monitoring

# Alloy (log collection)
helm upgrade alloy helm/alloy \
  -f helm/alloy/values-civo.yaml \
  -n monitoring

Adding a new alert rule

Edit helm/monitoring/templates/prometheusrule-core.yaml or prometheusrule-app.yaml
Add a new rule following the existing format
Redeploy the monitoring chart
Verify in Prometheus UI (Alerts tab) that the rule appears

Adding a new dashboard

Create the dashboard JSON in Grafana UI (or manually)
Export the JSON and save to helm/monitoring/dashboards/
Create a ConfigMap template in helm/monitoring/templates/ with labels:
yaml
```
labels:
  grafana_dashboard: "1"
```
Redeploy the monitoring chart
Dashboard appears automatically in Grafana

Rotating the PagerDuty key

The PagerDuty routing key is injected at deploy time via --set. To rotate:

Generate a new Integration Key in PagerDuty (Service > Integrations > Events API v2)

Redeploy with the new key:

bash

helm upgrade spektrabot-monitoring helm/monitoring \
  -f helm/monitoring/values-civo.yaml \
  --set 'pagerdutyRoutingKey=NEW_KEY' \
  -n monitoring

Verify in AlertManager UI that the config updated

Checking cluster health

Quick health check commands:

bash

# All monitoring pods
kubectl get pods -n monitoring

# PVC usage
kubectl get pvc -n monitoring

# Active alerts
kubectl port-forward svc/spektrabot-monitoring-kube-alertmanager 9093:9093 -n monitoring
# Then visit http://localhost:9093

# Prometheus targets
kubectl port-forward svc/spektrabot-monitoring-kube-prometheus 9090:9090 -n monitoring
# Then visit http://localhost:9090/targets

Troubleshooting

Prometheus not scraping a target

Check ServiceMonitor/PodMonitor exists: kubectl get servicemonitors -n monitoring
Verify labels match: the ServiceMonitor must have release: spektrabot-monitoring
Check Prometheus targets page for errors: http://localhost:9090/targets
Verify the target pod has the metrics port exposed

Grafana dashboard not loading

Check ConfigMap exists: kubectl get cm -n monitoring -l grafana_dashboard=1
Check sidecar logs: kubectl logs <grafana-pod> -n monitoring -c grafana-sc-dashboard
Verify JSON is valid (common issue: trailing commas)

Loki not receiving logs

Check Alloy pods are running: kubectl get pods -n monitoring -l app.kubernetes.io/name=alloy
Check Alloy logs: kubectl logs <alloy-pod> -n monitoring
Verify Loki is healthy: kubectl exec -n monitoring loki-0 -- wget -qO- http://localhost:3100/ready
Check Loki gateway: kubectl logs <loki-gateway-pod> -n monitoring

AlertManager not routing to PagerDuty

Verify secret exists: kubectl get secret alertmanager-pagerduty-config -n monitoring
Check AlertManager config: visit http://localhost:9093/#/status
Verify routing key format: must be Events API v2 Integration Key (32-char alphanumeric)
Check AlertManager logs for PagerDuty API errors

Last updated: 2026-02-12

Monitoring & Observability Guide ​

Architecture Overview ​

Components ​

Access ​

Grafana (dashboards and logs) ​

Prometheus (metrics and PromQL) ​

AlertManager (alert routing and silences) ​

Loki (log queries via Grafana) ​

Dashboards ​

Built-in (from kube-prometheus-stack) ​

Custom dashboards ​

Alert Rules ​

Custom alerts (9 rules in 2 PrometheusRule CRDs) ​

Infrastructure alerts (core-alerts) ​

Application alerts (app-alerts) ​

PagerDuty integration ​

Runbooks ​

APIDown ​

APIHighErrorRate ​

PostgresDown ​

PodCrashLoopBackOff ​

NodeCPUHigh ​

NodeMemoryHigh ​

PVCUsageHigh ​

PVCFull ​

WebSocketMassDisconnect ​

Log Queries ​

Common LogQL queries ​

Maintenance ​

Deploying changes ​

Adding a new alert rule ​

Adding a new dashboard ​

Rotating the PagerDuty key ​

Checking cluster health ​

Troubleshooting ​

Prometheus not scraping a target ​

Grafana dashboard not loading ​

Loki not receiving logs ​

AlertManager not routing to PagerDuty ​

Monitoring & Observability Guide

Architecture Overview

Components

Access

Grafana (dashboards and logs)

Prometheus (metrics and PromQL)

AlertManager (alert routing and silences)

Loki (log queries via Grafana)

Dashboards

Built-in (from kube-prometheus-stack)

Custom dashboards

Alert Rules

Custom alerts (9 rules in 2 PrometheusRule CRDs)

Infrastructure alerts (`core-alerts`)

Application alerts (`app-alerts`)

PagerDuty integration

Runbooks

APIDown

APIHighErrorRate

PostgresDown

PodCrashLoopBackOff

NodeCPUHigh

NodeMemoryHigh

PVCUsageHigh

PVCFull

WebSocketMassDisconnect

Log Queries

Common LogQL queries

Maintenance

Deploying changes

Adding a new alert rule

Adding a new dashboard

Rotating the PagerDuty key

Checking cluster health

Troubleshooting

Prometheus not scraping a target

Grafana dashboard not loading

Loki not receiving logs

AlertManager not routing to PagerDuty