How to Extract Logs and Query Dumps from Remote Executor for Debugging

When troubleshooting ingestion issues (particularly lineage problems), support may request debug logs from your remote executor. In most cases, logs are sufficient for debugging. Occasionally, support may also request query dump files for deeper analysis. These artifacts are stored within the remote executor container and require specific steps to extract them.

You Might Be Asking:

• How do I extract logs from my remote executor pod?
• How do I enable debug mode for ingestion runs?
• Where do I find the execution request URN?
• When are query dumps needed versus just logs?
• Where are query dump files stored in the remote executor?

Part 1: Extracting Debug Logs (Most Common)

In most troubleshooting scenarios, debug logs alone are sufficient. Follow these steps to extract logs from your remote executor.

Step 1: Enable Debug Mode and Required Environment Variables

Before extracting logs, ensure debug mode is enabled in your ingestion source configuration:

1. Navigate to your ingestion source settings in the DataHub UI
2. Click through the modals until you reach the 4th step
3. Toggle "Advanced" view
4. Enable the DEBUG log option (check the debug checkbox)
5. Save the config and trigger the recipe run

Step 2: Run Your Ingestion and Get the Execution Request URN

1. After the run completes, navigate to the ingestion run in the DataHub UI
2. Click the three dots (⋮) next to the execution run
3. Copy the Execution Request URN (it will look like: urn:li:dataHubExecutionRequest:4dd36f16-ff5c-4096-afe2-11ca5baa7da6)
4. Extract the run ID from the URN (the UUID portion after the last colon, in the above case it is: 4dd36f16-ff5c-4096-afe2-11ca5baa7da6)

Step 3: Extract Logs from Remote Executor Pod

Text logs are stored in the executor pod at /tmp/datahub/logs/4dd36f16-ff5c-4096-afe2-11ca5baa7da6/executor-logs/ingestion-logs.log:

# Copy logs from pod to local machine
kubectl cp [namespace]/[executor-pod-name]:/tmp/datahub/logs/[ingestion-run-id]/executor-logs/ingestion-logs.log ./ingestion-logs.log

# Example:
kubectl cp dataeng-datahub/datahub-executor-worker-0:/tmp/datahub/logs/4dd36f16-ff5c-4096-afe2-11ca5baa7da6/executor-logs/ingestion-logs.log ./ingestion-logs.log

Step 4: Share Logs with Support

Send the extracted log file to support for analysis. In most cases, this is all that's needed for troubleshooting.

---

Part 2: Extracting Query Dumps (When Requested by Support)

Query dumps are only needed in specific cases when logs alone don't provide sufficient information for debugging. Support will explicitly request query dumps if they're needed.

Step 1: Enable query dumps

1. Navigate to the "Advanced" view of the recipe configuration modal, as before
2. Add the following environment variables to your ingestion source configuration:

   
   {
     "DATAHUBSQLAGGQUERYLOG": "STORE_ALL",
     "TMPDIR": "/tmp"
   }
   

Step 2: Locate Query Dump Paths in the Ingestion Report

After running your ingestion with debug mode enabled and above environmental variables set, you need to find the paths to the query dump files:

1. Access the JSON structured report from your ingestion run (it is printed in the log periodically, as well as at the very end of ingestion)
2. There will be 2 different paths in the form /tmp/tmp[random-characters]/query_log.db in the report, they will be available under keys:

• sql_aggregator.query_log_path
• queries_extractor.sql_aggregator.query_log_path

Example paths from a report:

/tmp/tmpxgeymfw3/query_log.db
/tmp/tmpohhcbzo3/query_log.db

For a more complete example of how the report looks and where to find these paths, refer to this example document.

Important: You need to extract BOTH query_log.db files. While only one may be necessary, it's often confusing to determine which one, so it's best to send both to support.

Step 2: Extract Query Dump Files

Once you have identified the paths, extract both query dump files:

# Extract both query dump files (replace paths with the actual ones from your report)
kubectl cp [namespace]/[executor-pod-name]:/tmp/tmpXXXXXXXX/query_log.db ./query_log_1.db
kubectl cp [namespace]/[executor-pod-name]:/tmp/tmpYYYYYYYY/query_log.db ./query_log_2.db

# Example:
kubectl cp dataeng-datahub/datahub-executor-worker-0:/tmp/tmpxgeymfw3/query_log.db ./query_log_1.db
kubectl cp dataeng-datahub/datahub-executor-worker-0:/tmp/tmpohhcbzo3/query_log.db ./query_log_2.db

Step 3: Share Query Dumps with Support

Send both query_log.db files along with your debug logs to support for analysis.

• Logs are usually sufficient: In most troubleshooting scenarios, debug logs alone are enough. Query dumps are only needed for specific deep-dive investigations.
• Do not use UI-extracted logs: The logs visible in the DataHub UI are truncated. Always extract logs directly from the executor pod.
• Multiple query_log.db files are normal: Each ingestion run generates exactly 2 query log files in different temporary directories. Extract both of them when requested by support.
• Debug mode increases log size: Enabling debug mode will significantly increase log verbosity and size.
• Temporary files may be cleaned up: Query dump files in /tmp/ directories may be deleted after some time, so extract them promptly after an ingestion run.
• Future enhancement: Starting in v0.3.14, there may be an opt-in solution to automatically send logs to support (with customer approval), reducing the need for manual extraction.

• DataHub Remote Executor
• Debugging Ingestion