Tableau Ingestion Hangs or Times Out: Debugging Project Pattern Regex, PAT Sharing, Virtual Connections, and Node Limits – DataHub Customer Support Portal

Area: Ingestion Issues

Sub-Area: Tableau Connector Configuration & Performance

Issue

Tableau ingestion jobs that previously completed successfully may begin hanging indefinitely, failing with server-side 30-second timeouts on databaseTablesConnection, returning 401 Unauthorized errors mid-run, or taking several hours to complete after upgrading the DataHub CLI. These failures are typically caused by a combination of interrelated configuration problems: an incorrectly escaped regex in project_pattern that silently matches no projects and forces unbounded Metadata API queries, a shared Personal Access Token (PAT) across multiple environments causing session collisions, virtual connection ingestion (enabled by default since DataHub CLI 1.6.0.4) generating thousands of additional API calls when no virtual connections are actually present, and page-size settings that cause individual Tableau Metadata API responses to exceed Tableau's 20,000-node server-side limit.

Error Messages

Query databaseTablesConnection received a retryable error with 3 retries remaining, will retry in 1 seconds: [{'message': 'Execution canceled because timeout of 30000 millis was reached', 'locations': []}]
<class 'tableauserverclient.server.endpoint.exceptions.NonXMLResponseError'>: b'{"timestamp":"...","status":401,"error":"Unauthorized","path":"/relationship-service-war/graphql"}'
Tableau Data Exceed Predefined Limit
logs appear to be stale… however the ingestion process still appears to be running

You Might Be Asking

Why does my Tableau ingestion hang forever on databaseTablesConnection with a 30-second timeout?
Why does my Tableau ingestion fail with 401 Unauthorized even though the PAT is not expired?
Why did Tableau ingestion slow down dramatically after upgrading from CLI 1.6.0 to 1.6.0.4 or later?
What does "Tableau Data Exceed Predefined Limit" mean and how do I fix it?
Why does my project_pattern filter seem to match nothing even though it looks correct?
Why does Tableau ingestion succeed when triggered manually but fail when scheduled?

Solution

Work through the following root causes in order. Each one can independently cause ingestion to hang or fail, and multiple causes are often present simultaneously.

Fix the project_pattern regex — escape literal parentheses
In Python regex, unescaped parentheses () define a capture group, not literal characters. If your Tableau project names contain parentheses (e.g., My Project (Dev)), the pattern must use backslash-escaped parentheses. Without them, the filter silently matches no projects, which forces an unbounded databaseTablesConnection query against the entire Tableau server — triggering Tableau's 30-second server-side timeout.

Incorrect (causes hang):
```
source:
  config:
    project_pattern:
      allow:
        - '.*(Dev)$'   # (Dev) is a capture group — does NOT match literal parentheses
```
Correct:
```
source:
  config:
    project_pattern:
      allow:
        - '.*$Dev$$'  # $ and $ match literal parentheses in the project name
```
Assign a unique Personal Access Token (PAT) per environment
Tableau enforces a single active session per PAT. If multiple environments (e.g., DEV, RVW, STG, PRD) share the same PAT, any environment that signs in will invalidate the active session of any other environment currently running. This manifests as scheduled runs failing within seconds with a 401, while manual re-triggers succeed (because no other environment is competing at that exact moment).

Create a dedicated PAT for each environment in Tableau Server and update each recipe to reference its own token:
```
# DEV environment recipe
source:
  type: tableau
  config:
    token_name: datahub-dev-token
    token_value: "<dev-pat-secret>"

# STG environment recipe
source:
  type: tableau
  config:
    token_name: datahub-stg-token
    token_value: "<stg-pat-secret>"

# PRD environment recipe
source:
  type: tableau
  config:
    token_name: datahub-prd-token
    token_value: "<prd-pat-secret>"
```
Note: Also ensure that previously cancelled or aborted ingestion runs have fully stopped before triggering a new run. DataHub may report that a cancelled run's process is still active in the background. A new run starting while the old process still holds a session can also produce 401 errors even with a dedicated PAT.
Disable virtual connection ingestion if no virtual connections are in use
DataHub CLI versions 1.6.0.4 and later enabled virtual connection ingestion by default (ingest_virtual_connections: true). Even when a Tableau site has no meaningful virtual connections, the connector still issues virtualConnectionsConnection queries and — more significantly — thousands of additional databaseTablesConnection paginated queries to resolve virtual connection references against database tables. In large environments this alone can account for thousands of API calls and hours of additional runtime. If your logs show 0 virtual connections processed and 0 lineage created from virtual connections, disable this feature:
```
source:
  type: tableau
  config:
    ingest_virtual_connections: false
```
This is the single highest-impact change for environments that upgraded from 1.6.0 to 1.6.0.4+ and saw dramatic slowdowns with no other recipe changes.
Tune page sizes to avoid Tableau's 30-second timeout and 20,000-node limit
Three separate page-size parameters control different Metadata API queries. Setting them too high causes Tableau to time out or truncate results; setting them too low multiplies request volume and increases total runtime. Tune them in this order based on the errors you observe:

a) database_table_page_size — fixes the 30-second databaseTablesConnection timeout

If the databaseTablesConnection timeout persists after fixing the regex and disabling virtual connections, reduce this value to split large queries into smaller chunks. Start with 5 and increase toward 10 once timeouts are resolved:
```
source:
  config:
    database_table_page_size: 5   # default is 10; reduce if 30s timeouts persist
```
b) embedded_datasource_page_size — fixes "Tableau Data Exceed Predefined Limit" warnings

The Tableau Data Exceed Predefined Limit warning originates from embeddedDatasourcesConnection queries exceeding Tableau's metadata.query.node_limit (default 20,000 nodes). Each embedded datasource query fetches fields, columns, and upstream tables inline, so even a page size of 10 can exceed the limit for complex datasources. Reduce to 5; drop to 3 if warnings persist:
```
source:
  config:
    embedded_datasource_page_size: 5   # default is 10; controls embeddedDatasourcesConnection queries
```
Note: workbook_page_size and database_table_page_size do not affect the node-limit warning on embedded datasource queries. Changing them will not resolve this warning.

c) workbook_page_size — reduces workbook request volume

The connector defaults to fetching workbooks one at a time (workbook_page_size: 1). Increasing this reduces the number of workbook requests significantly. However, workbook queries are also subject to the 20,000-node limit because fields and sheets are fetched inline. A value of 3 is a safe starting point:
```
source:
  config:
    workbook_page_size: 3   # default is 1; increase cautiously to avoid node-limit truncation
```
Verify the ingestion account has the correct Tableau role
The ingestion service account should have the Site Administrator Explorer role in Tableau Server. Accounts with the Creator role may receive permission-filtered Metadata API responses, causing some workbooks and datasources to be silently excluded from results and potentially contributing to unexpected query behavior.

Recommended complete recipe configuration incorporating all fixes:

source:
  type: tableau
  config:
    connect_uri: "https://<your-tableau-server>"
    site: "<your-site-name>"
    token_name: "<env-specific-token-name>"
    token_value: "<env-specific-token-secret>"

    project_pattern:
      allow:
        - '.*\(<YourProjectSuffix>\)$'   # Use \( and \) to match literal parentheses

    ingest_virtual_connections: false      # Disable if no virtual connections are in use

    database_table_page_size: 5            # Reduce if 30s databaseTablesConnection timeouts occur
    embedded_datasource_page_size: 5       # Reduce if "Tableau Data Exceed Predefined Limit" appears
    workbook_page_size: 3                  # Increase from default of 1 to reduce request volume

Additional Notes

Version impact: Virtual connection ingestion (ingest_virtual_connections) was introduced and enabled by default in DataHub CLI 1.6.0.4. Users upgrading from 1.6.0 to any later version who do not use Tableau virtual connections should explicitly set ingest_virtual_connections: false.
Tableau node limit: The "Tableau Data Exceed Predefined Limit" warning is enforced server-side by Tableau's metadata.query.node_limit setting (default 20,000). Tableau returns a partial result set rather than an error, so metadata can be silently incomplete when this limit is hit. Tableau administrators can raise this limit via TSM as an alternative to reducing page sizes in DataHub.
Connector reauth bug: A known bug in the Tableau connector (tracked internally) prevents re-authentication when a session expires within the first 10 minutes of a run. The connector retries on the dead session rather than signing in again. Using dedicated per-environment PATs and ensuring prior runs have fully terminated before starting a new run are the primary mitigations until a connector fix is released.
Session expiry on long runs: Tableau Server enforces its own session expiry on the Metadata API. Runs exceeding approximately 2 hours may encounter 401 errors even with correct PAT configuration. Tuning page sizes to reduce total runtime is the mitigation for this.
Permission warnings: Warnings about workbooks or datasources being permission-filtered (e.g., hundreds of results excluded) indicate the ingestion account cannot access certain Tableau projects. This is separate from the timeout and node-limit issues and requires a Tableau role or permission change on the Tableau Server side.
Testing approach: When tuning page sizes or isolating issues, run one environment at a time and confirm all other environments are not mid-run. Session collisions from concurrent runs in different environments can mimic page-size or connector bugs.

Issue

Error Messages

You Might Be Asking

Solution

Additional Notes

Related Documentation