Kubernetes Deployment Pod Crashes or Restarts

DataHub pods (GMS, Frontend, Actions, etc.) are crashing, restarting frequently, or stuck in CrashLoopBackOff state. This commonly occurs due to insufficient resources, configuration errors, dependency issues, or database connectivity problems.

Common Error Messages:

• CrashLoopBackOff
• OOMKilled (out of memory)
• ImagePullBackOff
• Error: unable to connect to database

You Might Be Asking:

• Why do my DataHub pods keep restarting?
• How much memory/CPU does DataHub need?
• How do I troubleshoot pod crashes?

1. Check pod status and events:

# Get pod status
kubectl get pods -n datahub

# Describe pod to see events
kubectl describe pod datahub-gms-xxx -n datahub

# Check recent logs
kubectl logs datahub-gms-xxx -n datahub --tail=100

# Check previous container logs if restarting
kubectl logs datahub-gms-xxx -n datahub --previous

2. Increase resource limits for OOMKilled pods:

# In values.yaml for Helm chart
datahub-gms:
  resources:
    requests:
      memory: "2Gi"
      cpu: "1000m"
    limits:
      memory: "4Gi"
      cpu: "2000m"

datahub-frontend:
  resources:
    requests:
      memory: "1Gi"
      cpu: "500m"
    limits:
      memory: "2Gi"
      cpu: "1000m"

3. Verify database connectivity:

# Test database connection from pod
kubectl exec -it datahub-gms-xxx -- nc -zv mysql 3306
kubectl exec -it datahub-gms-xxx -- nc -zv postgres 5432

# Check database credentials in secrets
kubectl get secret datahub-db-secret -o yaml

# Verify database is ready
kubectl get pods | grep -E "mysql|postgres"

4. Check for configuration errors:

# Validate ConfigMap
kubectl get configmap datahub-gms-config -o yaml

# Check environment variables
kubectl exec datahub-gms-xxx -- env | grep -E "DATAHUB|KAFKA|ELASTIC"

# Verify required services are running
kubectl get pods | grep -E "kafka|elasticsearch|mysql"

5. Common fixes for specific components:

# GMS: Increase heap size
env:
  - name: JAVA_OPTS
    value: "-Xms2g -Xmx4g"

# Frontend: Increase timeout
env:
  - name: DATAHUB_GMS_TIMEOUT
    value: "60000"

# Elasticsearch: Increase heap
env:
  - name: ES_JAVA_OPTS
    value: "-Xms2g -Xmx2g"

Monitor pod resource usage with kubectl top pods to identify resource constraints. Use liveness and readiness probes appropriately to avoid premature restarts during startup.

• Kubernetes Deployment Guide
• Resource Requirements