Kafka NotEnoughReplicasException Causes Ingestion Failures During Maintenance Windows – DataHub Customer Support Portal

Area: Deployment Issues

Sub-Area: Kafka Cluster Health / Ingestion Write Failures

Issue

During or shortly after a Kafka cluster maintenance event — such as a broker restart, rolling upgrade, broker replacement, or scaling operation — DataHub ingestion jobs may fail with a NotEnoughReplicasException. This occurs because Kafka rejects write requests when the number of in-sync replicas (ISR) for a partition drops below the configured min.insync.replicas threshold. Because DataHub's GMS Kafka producer uses acks=all by default, any partition that is temporarily under-replicated will cause GMS to return an HTTP 500 error to the ingestion client, surfacing as an OperationalError in ingestion logs. Affected ingestion runs may appear to have completed but will have missing or incomplete metadata for the window during which Kafka was rejecting writes.

Error Messages

datahub.configuration.common.OperationalError: ('Unable to emit metadata to DataHub GMS: java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.NotEnoughReplicasException: Messages are rejected since there are fewer in-sync replicas than required.')
com.linkedin.restli.server.RestLiServiceException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.NotEnoughReplicasException: Messages are rejected since there are fewer in-sync replicas than required.

You Might Be Asking

Why are my ingestion jobs suddenly failing with Kafka errors when nothing changed in my ingestion recipe?
Is this error caused by a misconfiguration in my DataHub ingestion source or connector?
Do I need to change my ingestion recipe to fix this error?
Will the error resolve on its own, or do I need to take action after the maintenance window ends?
Which ingestion runs do I need to re-run after a Kafka maintenance event?

Solution

Confirm the error is Kafka-originated, not a recipe misconfiguration.
The NotEnoughReplicasException is thrown by the Kafka broker, not by DataHub GMS or the ingestion framework. Changes to your ingestion recipe will not resolve this error. No modifications to source connectors or ingestion configuration are required.
Correlate the error window with Kafka maintenance activity.
Identify the timestamp of the first and last error in your ingestion logs. Compare this window against any known Kafka cluster maintenance events, such as:
- Rolling broker restarts or upgrades
- Broker replacements or instance-type changes
- Kafka cluster scaling operations
- Broker disk, CPU, or network saturation events
If you are on DataHub Cloud, contact DataHub Support to correlate the error window with infrastructure-side maintenance activity.
Verify that the Kafka cluster has recovered.
Once the maintenance window ends and all brokers have rejoined their ISR sets, writes will succeed automatically. You can verify recovery by checking that subsequent ingestion runs complete without NotEnoughReplicasException errors. On self-hosted deployments, check the under-replicated partitions metric on your Kafka cluster for the following DataHub topics:
```
MetadataChangeProposal_v1
MetadataChangeLog_Versioned_v1
MetadataChangeLog_Timeseries_v1
FailedMetadataChangeProposal_v1
```
For each topic, confirm that the replication factor, min.insync.replicas, and current ISR count are consistent and healthy. A common risk configuration is a replication factor equal to min.insync.replicas (e.g., replication factor 2 with min.insync.replicas=2), which tolerates zero broker losses. Prefer a replication factor of 3 with min.insync.replicas=2 to tolerate single-broker failures.
Identify and re-run affected ingestion jobs.
Ingestion runs that executed during the outage window will have incomplete metadata in DataHub. Because DataHub ingestion is idempotent, re-running affected recipes is safe and will not cause duplicates. To identify affected runs:
- Navigate to Ingestion in the DataHub UI and review run history for jobs that executed during the error window.
- Look for runs that show partial failures, error counts greater than zero, or runs that do not appear in run history at all (indicating they may have been entirely dropped).
- Re-run any job whose execution window overlaps with the period during which NotEnoughReplicasException errors were logged.
For ingestion sources that run on a frequent schedule (e.g., hourly), those jobs will naturally catch up on the next scheduled execution without manual intervention.
If errors persist beyond the maintenance window, escalate.
If NotEnoughReplicasException errors continue after the maintenance event is confirmed complete, the Kafka cluster may not have fully recovered. In this case:
- On DataHub Cloud: Contact DataHub Support for infrastructure-level investigation.
- On self-hosted deployments: Inspect broker health, ISR counts, and topic configurations directly on your Kafka cluster. Repair must happen at the Kafka infrastructure level — no DataHub-side configuration change will resolve a persistently under-replicated cluster.

Additional Notes

This error originates entirely from the Kafka broker layer and is not a DataHub product regression or ingestion framework bug. The DataHub GMS Kafka producer uses acks=all by default, which is the correct durability setting and intentionally surfaces broker-side replica failures as write errors rather than silently dropping data. The error will self-resolve once the Kafka cluster returns to a healthy ISR state after the maintenance event completes. Ingestion runs that must be manually re-triggered are those that ran during the outage window and cannot rely on a frequent schedule to catch up automatically. On DataHub Cloud, the infrastructure team monitors Kafka cluster health and works to minimize the impact of maintenance events on ingestion availability; improvements to maintenance procedures are ongoing.

Issue

Error Messages

You Might Be Asking

Solution

Additional Notes

Related Documentation