Kafka Topic Lag and Message Processing Delays

Kafka topics used by DataHub (MetadataChangeProposal, MetadataChangeLog, etc.) are showing high consumer lag, causing delays in metadata updates appearing in the UI. This typically occurs during high ingestion volume, insufficient consumer resources, or Kafka configuration issues.

You Might Be Asking:

• Why is my metadata taking so long to appear?
• How do I check Kafka consumer lag?
• What causes Kafka lag in DataHub?

1. Check Kafka consumer lag:

# List consumer groups
kubectl exec -it kafka-0 -- kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 --list

# Check lag for DataHub consumers
kubectl exec -it kafka-0 -- kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 \
  --group datahub-mce-consumer \
  --describe

# Look for LAG column values

2. Increase MCE consumer parallelism:

# In values.yaml
datahub-mce-consumer:
  replicaCount: 3  # Scale up consumers

  resources:
    requests:
      memory: "1Gi"
      cpu: "500m"
    limits:
      memory: "2Gi"
      cpu: "1000m"

3. Optimize Kafka topic configuration:

# Increase partition count for better parallelism
kubectl exec -it kafka-0 -- kafka-topics.sh \
  --bootstrap-server localhost:9092 \
  --alter --topic MetadataChangeProposal_v1 \
  --partitions 6

# Increase retention if needed
kubectl exec -it kafka-0 -- kafka-configs.sh \
  --bootstrap-server localhost:9092 \
  --entity-type topics \
  --entity-name MetadataChangeProposal_v1 \
  --alter --add-config retention.ms=604800000

4. Check for consumer errors:

# Check MCE consumer logs
kubectl logs deployment/datahub-mce-consumer | grep -i "error\|exception"

# Check GMS logs for processing errors
kubectl logs deployment/datahub-gms | grep "MCP\|MCE" | grep -i error

5. Monitor Kafka performance:

# Check Kafka broker health
kubectl exec -it kafka-0 -- kafka-broker-api-versions.sh \
  --bootstrap-server localhost:9092

# Monitor topic throughput
kubectl exec -it kafka-0 -- kafka-run-class.sh kafka.tools.JmxTool \
  --object-name kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec

Consumer lag is normal during bulk ingestion operations. If lag persists during normal operations, consider scaling up consumer replicas or optimizing ingestion batch sizes. Partitions must be >= consumer count for effective parallelism.

• Kafka Setup
• Architecture Overview