Skip to main content

DataHub Cloud Instance Downtime Due to GMS Pod Liveness Probe Failures

Angelo Bresci
Updated

Area: Deployment Issues

Sub-Area: Cloud Instance Availability

Issue

DataHub Cloud instances may experience brief downtime when the GMS (GraphQL Metadata Service) pods fail liveness probe checks and are automatically restarted by Kubernetes. This manifests as the instance becoming unresponsive for several minutes, typically during the pod restart process. The issue can affect one or more GMS pods in a multi-pod deployment and may recur if underlying resource constraints are not addressed.

You Might Be Asking

  • Why is my DataHub Cloud instance suddenly inaccessible?
  • What causes GMS pods to fail liveness probes?
  • How long does downtime typically last during pod restarts?
  • Can this issue recur after resolution?

Solution

  1. Immediate Response: If you experience instance downtime, immediately contact DataHub Support through your designated support channel. The downtime is typically brief (4-6 minutes) while pods restart.
  2. Monitoring During Incident: The liveness probe mechanism is working as designed - it detects unresponsive GMS pods and automatically restarts them to restore service.
  3. Root Cause Investigation: DataHub Support will:
    • Check GMS pod health and restart history in your namespace
    • Review Kubernetes events for your cluster
    • Examine resource utilization patterns (CPU/memory)
    • Investigate potential Elasticsearch cluster issues
  4. Resolution Steps: Based on the investigation, resolution may involve:
    • Vertical scaling of pods if resource exhaustion is identified
    • Elasticsearch cluster capacity adjustments
    • Dedicated resource allocation for tenant isolation
    • Infrastructure-level optimizations
  5. Verification: After resolution, confirm instance accessibility and monitor for recurring issues over the following days.

Additional Notes

This is an infrastructure-level issue managed entirely by DataHub Cloud operations. Customers cannot directly access or troubleshoot the underlying Kubernetes pods, Elasticsearch clusters, or monitoring systems. The liveness probe restart mechanism is a protective feature that prevents prolonged outages. If the issue recurs frequently, it may indicate resource constraints that require infrastructure scaling or tenant isolation. DataHub Cloud's managed service model means all remediation is handled by the platform team without customer intervention required. Please contact DataHub Support via this portal if this occurs, and we can escalate as necessary.

Related Documentation

Tags: cloud-downtime, gms-pod-restart, liveness-probe, kubernetes-restart, instance-unavailable, elasticsearch-memory, resource-exhaustion, infrastructure-scaling, managed-service, pod-failure