Skip to main content

Spark Driver OOM Issues with DataHub Lineage Coalescing at Scale

Angelo Bresci
Updated

Area: Deployment Issues

Sub-Area: Spark Integration Memory Management

Issue

When using DataHub's acryl-spark-lineage connector with coalesced job emission enabled (default behavior), Spark applications with many SQL operations can experience OutOfMemory (OOM) kills in the driver during lineage emission at shutdown. The DatahubEventEmitter accumulates all OpenLineage RunEvent objects in memory throughout the application lifetime, then serializes them all into MetadataChangeProposal (MCP) objects at onApplicationEnd, causing a memory spike that can exceed container limits for large-scale pipelines.

Error Messages

  • Container OOMKilled (exit code 137)
  • Driver pod terminated during onApplicationEnd

You Might Be Asking

  • Why do my Spark jobs process data successfully but get marked as FAILED?
  • How can I prevent OOM kills while maintaining proper lineage tracking?
  • What's the difference between coalesced and non-coalesced lineage emission?

Solution

Immediate Workaround: Disable Job Coalescing

  1. Set the following Spark configuration to disable coalesced emission: 
    spark.datahub.coalesce_jobs.enabled=false
  2. This causes MCPs to be emitted per SQL execution rather than batched at shutdown, eliminating the memory spike
  3. Note that this changes the DataHub graph model from one merged DataJob per application to multiple DataJobs per SQL execution

Alternative: Increase Driver Memory

  1. If you need to maintain coalesced emission behavior, increase the Spark driver memory allocation: 
    spark.driver.memory=g
spark.driver.maxResultSize=g
  2. Monitor memory usage patterns to determine appropriate sizing for your workload scale
  3. Consider this a temporary solution while planning for more sustainable approaches

For Development and Testing

  1. To analyze lineage payload size and memory usage patterns, emit lineage to file: 
    spark.datahub.emitter=file
spark.datahub.file.filename=/path/to/lineage-output.jsonl
  2. Enable garbage collection logging to monitor memory pressure: 
    spark.driver.extraJavaOptions=-Xlog:gc*=info:file=/path/to/gc.log:time,level,tags

Additional Notes

The root cause is in the DatahubEventEmitter.java implementation where each SQL execution appends a DatahubJob object to the _datahubJobs LinkedList. With coalescing enabled, these accumulate throughout the application lifetime and are processed all at once during shutdown. For pipelines with 100+ SQL operations, this can generate 200+ MCPs in a rapid burst, overwhelming driver memory. The _datahubJobs list grows unboundedly in both coalesced and non-coalesced modes, which may represent a memory leak in the current implementation. There are currently no configuration options in the open-source version to cap the number of accumulated events or bound emitter memory usage.

Related Documentation

Tags: spark, lineage, oom, memory, driver, coalescing, acryl-spark-lineage, scale, kubernetes, deployment

Related articles