MAE/MCE Consumer Pods Fail Liveness and Readiness Probes After Upgrading to DataHub v1.0.x – DataHub Customer Support Portal

Area: Deployment Issues

Sub-Area: Kubernetes Health Probe Configuration / Helm Chart Upgrade

Issue

After upgrading DataHub from a pre-v1.0.x release to v1.0.x (e.g., v1.0.3), the datahub-mae-consumer and datahub-mce-consumer pods enter a continuous restart loop. Kubernetes reports both liveness and readiness probe failures with HTTP status code 404. The root cause is a breaking change introduced in v1.0.x: the Helm chart's health probe port selection became dynamic, depending on monitoring configuration values. When Prometheus metrics are enabled with certain metricsMode settings, the probes are directed to port 4319 (the Spring Boot management/actuator port) instead of the legacy main application port (9091 for MAE, 9090 for MCE). If the application is not correctly bound to the expected port, /actuator/health returns 404, causing probe failures and frequent pod restarts.

Error Messages

Liveness probe failed: HTTP probe failed with statuscode: 404
Readiness probe failed: HTTP probe failed with statuscode: 404
Warning BackOff: Back-off restarting failed container datahub-mae-consumer
Warning BackOff: Back-off restarting failed container datahub-mce-consumer

You Might Be Asking

Why do my MAE/MCE consumer pods keep restarting after upgrading to DataHub v1.0.x?
Why is /actuator/health returning 404 on my consumer pods after a version upgrade?
Which port should Kubernetes health probes use for MAE and MCE consumers in DataHub v1.0.x?
How do I fix liveness and readiness probe failures on DataHub consumer pods after upgrading Helm charts?
What changed in the DataHub v1.0.x Helm chart that affects consumer pod health probes?

Root Cause Detail

In DataHub v1.0.x, the Helm chart _helpers.tpl introduced dynamic probe port selection logic based on two values:

global.datahub.monitoring.enablePrometheus
global.datahub.monitoring.metricsMode

The behavior is as follows:

If enablePrometheus: true AND metricsMode is jmx_and_actuator or actuator_only → probes target port 4319 (the Spring Boot management/actuator port, named "prometheus").
Otherwise → probes target port 9091 (MAE) or port 9090 (MCE), the main HTTP port (named "http").

A 404 response (as opposed to a connection refused error) indicates the probe is successfully reaching a running server on the target port, but /actuator/health is not exposed there — a classic port mismatch. Additionally, v1.0.x introduced startup probes with initialDelaySeconds: 0, meaning probing begins immediately on pod start, which can compound instability if the application is slow to initialize.

Solution

Choose one of the following approaches based on your deployment requirements. In all cases, redeploy the affected consumer pods after applying changes.

Option 1 (Recommended): Update probes to target port 4319

If you wish to keep Prometheus metrics enabled with a modern metricsMode, update your Helm values to configure the liveness and readiness probes to use port 4319:

datahub-mae-consumer:
  livenessProbe:
    httpGet:
      path: /actuator/health
      port: 4319
  readinessProbe:
    httpGet:
      path: /actuator/health
      port: 4319

datahub-mce-consumer:
  livenessProbe:
    httpGet:
      path: /actuator/health
      port: 4319
  readinessProbe:
    httpGet:
      path: /actuator/health
      port: 4319

Option 2: Override the management server port via environment variable

If you prefer to keep probes targeting the legacy main application port, set the MANAGEMENT_SERVER_PORT environment variable to redirect the Spring Boot actuator back to the original port:

datahub-mae-consumer:
  extraEnvs:
    - name: MANAGEMENT_SERVER_PORT
      value: "9091"

datahub-mce-consumer:
  extraEnvs:
    - name: MANAGEMENT_SERVER_PORT
      value: "9090"

Option 3: Revert metricsMode to legacy (quick unblock)

To immediately stabilize a test or staging environment while investigating the correct long-term configuration, switch metricsMode to legacy. This forces probes back to port 9091/9090:

global:
  datahub:
    monitoring:
      enablePrometheus: true
      metricsMode: "legacy"

Option 4: Increase startup probe tolerance (supplemental)

If pods are slow to start and the new startup probe (with initialDelaySeconds: 0) is contributing to restarts, increase the failure threshold to extend the startup budget:

datahub-mae-consumer:
  startupProbe:
    failureThreshold: 60   # 60 × 5s = 5 minutes of startup budget
    periodSeconds: 5

datahub-mce-consumer:
  startupProbe:
    failureThreshold: 60
    periodSeconds: 5

Identify your current Helm monitoring values for global.datahub.monitoring.enablePrometheus and global.datahub.monitoring.metricsMode.
Apply one of the options above to your Helm values file (values.yaml or equivalent override file).

Redeploy the affected consumers:

helm upgrade <release-name> <chart> -f values.yaml -n <namespace>

Verify pods stabilize and probes pass:

kubectl get pods -n <namespace> -w
kubectl describe pod <mae-consumer-pod> -n <namespace>

Confirm no further Unhealthy or BackOff events appear in the pod description output.

Additional Notes

This is a breaking change introduced in DataHub v1.0.x and affects all self-hosted (on-premise) deployments upgrading from pre-v1.0.x Helm charts. DataHub Cloud customers are not affected, as Acryl manages all probe and port configurations in that environment. When upgrading across major versions (e.g., from v0.x or an older v3.x-era release to v1.0.x), always review the official upgrade guide for breaking changes before applying the new Helm chart. The port 4319 management endpoint is the new default for health and metrics in v1.0.x and later; Option 1 (updating probes to port 4319) is the forward-compatible approach. Options 2 and 3 are appropriate for short-term stabilization or environments not yet ready to adopt the new metrics architecture.