Large Dataset Ingestion Lag and Eventual Consistency

After running ingestion sources with large volumes of metadata (e.g., 156K Looker assets), the ingested datasets do not appear in the DataHub UI for several hours. While the ingestion job reports completion, metadata takes up to 6 hours to become visible, causing concerns about ingestion failures or data loss.

You Might Be Asking:

• Why doesn't my metadata appear immediately after ingestion?
• Is there an SLA for how long it takes for metadata to show up?
• Does large ingestion always take this long?
• How can I speed up metadata processing?
• Is this a bug or expected behavior?

DataHub is an eventually consistent system, meaning metadata ingested through the API undergoes asynchronous processing before appearing in the UI and search results.

Understanding Asynchronous Processing:

1. Ingestion Phase: Metadata is received and acknowledged quickly
2. Processing Phase: DataHub processes metadata in the background:
   • Indexing in Elasticsearch
   • Updating relationships and lineage
   • Computing aggregations
   • Updating the graph database
3. Visibility Phase: Metadata becomes searchable and visible in the UI

Expected Latency:

• Small ingestions (< 10K entities): Minutes to tens of minutes
• Medium ingestions (10K-100K entities): 30 minutes to 2 hours
• Large ingestions (> 100K entities): 2-6 hours or more
• First-time ingestions: Longer than incremental updates

Factors Affecting Processing Time:

• Total number of entities ingested
• Complexity of lineage relationships
• DataHub instance resources (CPU, memory, Elasticsearch capacity)
• Current system load
• Whether it's an initial ingestion or an update

Monitoring Ingestion Progress:

Check system logs and Elasticsearch indexing:

# Check Elasticsearch indexing lag
curl http://localhost:9200/_cat/indices?v | grep datahub

# Monitor MAE/MCE consumer lag (Kafka-based deployments)
# Check your Kafka monitoring dashboard

Best Practices for Large Ingestions:

1. Schedule during off-peak hours when the system has more resources
2. Use incremental ingestion rather than full refreshes when possible
3. Increase DataHub resources for consistent large ingestions:
   • Scale GMS replicas
   • Increase Elasticsearch cluster size
   • Allocate more memory to indexing components
4. Split large ingestions into smaller batches if possible

What This Is NOT:

• ❌ A bug or ingestion failure
• ❌ Data loss
• ❌ A problem requiring support intervention

What This IS:

• ✅ Expected behavior for large datasets
• ✅ A consequence of DataHub's asynchronous architecture
• ✅ A trade-off for system scalability and performance

• DataHub cannot guarantee fixed SLAs for metadata reflection
• The system prioritizes availability and scalability over immediate consistency
• Metadata is durable once ingested; processing delays don't affect data integrity
• For extremely large catalogs (millions of entities), consider discussing architecture optimization with support

• DataHub Architecture
• Performance Tuning

• 4832