dbt Ingestion Failing with Segmentation Fault or ConnectTimeoutError Due to sqlglot Bug – DataHub Customer Support Portal

```html

Area: Ingestion Issues

Sub-Area: dbt Ingestion — sqlglot Column Resolution Crash

Issue

When running dbt CLI ingestion using acryl-datahub[dbt]==1.6.0, the ingestion process may terminate abruptly with no Python traceback. The failure can surface as a ConnectTimeoutError on GET /gms/config or as a silent process exit, making it appear to be a network connectivity problem. The root cause is a native segmentation fault (SIGSEGV) inside the compiled ([c]) build of sqlglot version 30.8.0, which is pinned by CLI version 1.6.0. The crash is triggered when DataHub's column-level lineage inference attempts to run sqlglot.optimize() on SQL containing LATERAL VIEW or UNNEST expressions over an unqualified column whose base table is absent from the schema — a condition that occurs when the dbt catalog.json is incomplete (e.g., contains Node missing from catalog warnings). Because the crash occurs inside native C code, it is uncatchable by Python and produces no stack trace, causing the process to die silently and misleadingly resemble a network timeout. Other ingestion sources (such as Databricks, PowerBI, or Looker) are unaffected because they do not invoke sqlglot SQL parsing on the same code path.

Error Messages

ConnectTimeoutError: HTTPSConnectionPool(host='<your-instance>.acryl.io', port=443): Read timed out. (read timeout=30)
OSError: [Errno 139] Segmentation fault
WARNING {sqlglot:140} - Unknown subquery scope: SELECT ...
Node missing from catalog (repeated warnings in ingestion log)
Process exits with exit code 139 and no Python traceback

You Might Be Asking

Why does my dbt ingestion fail with a ConnectTimeoutError when other ingestion sources using the same token and GMS URL succeed?
Why does my dbt ingestion process terminate silently with no error message or stack trace?
Why does my dbt ingestion only fail on certain models or folders, but not others?
Does setting include_column_lineage: false prevent this crash?
Is this a network issue between my CI/CD environment and DataHub Cloud?

Solution

Upgrade the DataHub CLI to version 1.6.0.3 or later (1.6.x line).
This is the primary and recommended fix. CLI version 1.6.0.2 introduced a guard (_statement_risks_unnest_resolver_recursion) that detects the risky SQL pattern and skips column-level lineage (CLL) for that specific statement rather than crashing. Table-level lineage is unaffected. Version 1.6.0.3 is the latest stable release with this fix.

Update your CI/CD workflow or environment to pin the fixed version:
```
pip install "acryl-datahub[dbt]==1.6.0.3"
```
If you use a requirements.txt file, update the relevant line:
```
acryl-datahub[dbt]==1.6.0.3
```
If you use a GitHub Actions workflow, update the install step:
```
- name: Install DataHub CLI
  run: pip install "acryl-datahub[dbt]==1.6.0.3"
```

(Optional) Confirm the fix is present before upgrading.

The table below summarizes which CLI versions contain the guard:

CLI Version	Fix Present	Notes
1.6.0	❌ No	Original crashing version
1.6.0.1	❌ No	Still affected
1.6.0.2	✅ Yes	First release with fix
1.6.0.3	✅ Yes	Recommended stable release
1.7.0.1rc1	❌ No	⚠️ Branched before cherry-pick; do not use

⚠️ Do not upgrade to 1.7.0.x at this time. As of the time of this writing, the 1.7.0 release candidate branch was cut before the fix was cherry-picked, so upgrading to 1.7.0 would re-introduce the crash. Stay on the 1.6.0.x line until the 1.7.x line is confirmed to include the fix.

(Fallback) If you cannot upgrade immediately, use one of these workarounds:
- Disable dbt schema inference in your recipe:
```
source:
  type: dbt
  config:
    # ... other config ...
    infer_dbt_schemas: false
```
  Note: Setting include_column_lineage: false alone is not sufficient. The crash occurs in the infer_dbt_schemas path, which runs independently of the CLL setting.
- Ensure your catalog.json is complete by running dbt docs generate before ingestion.
  The crash is triggered when a table referenced in a LATERAL VIEW or UNNEST expression is missing from the catalog schema. A complete catalog removes the trigger condition.
```
# In your CI/CD pipeline, run this before datahub ingest:
dbt docs generate

# Then run ingestion as normal:
datahub ingest run --config recipe.yml
```
Verify the fix by re-running your ingestion pipeline.
After upgrading, re-run the failing ingestion. You should see normal completion rather than a silent process termination. If Node missing from catalog warnings remain in the log, they are now handled gracefully — CLL is skipped for affected statements rather than crashing the process.

Additional Notes

Why the error looks like a network timeout: When the DataHub CLI starts any ingestion pipeline, it calls GET /gms/config as part of its initial handshake (test_connection()). If the dbt ingestion process is killed by a SIGSEGV before or during this handshake — or if the crash occurs mid-run and the controlling process reports a timeout waiting for the subprocess — the failure manifests as a ConnectTimeoutError. This is misleading and is not a true network connectivity problem. You can confirm connectivity independently with a curl test:

curl -v https://<your-instance>.acryl.io/gms/config \
  -H "Authorization: Bearer <your-datahub-token>"

If this succeeds, the issue is the sqlglot crash described in this article, not a network problem.

Why only dbt is affected: Other ingestion source types (Databricks, PowerBI, Looker, etc.) do not use sqlglot to parse and optimize SQL from dbt artifact files. The crash is specific to the dbt ingestion source's column-level lineage and schema inference pipeline.

Identifying affected models: The ingestion log will stop abruptly at a line referencing sqlglot_lineage (e.g., "Prior to column qualification sql") for a specific model, with no subsequent traceback. The last model mentioned before the process dies is the one triggering the crash. Models containing LATERAL VIEW POSEXPLODE, LATERAL VIEW EXPLODE, or similar UNNEST-style expressions in Databricks SQL or Spark SQL are most likely to trigger this condition when the catalog is incomplete.

Version context: This issue was introduced when CLI version 1.6.0 pinned sqlglot[c]==30.8.0. The compiled ([c]) variant of sqlglot uses a native C extension for performance; the recursion that causes the SIGSEGV occurs in that native layer and cannot be caught by Python exception handling.

Issue

Error Messages

You Might Be Asking

Solution

Additional Notes

Related Documentation