PowerBI Ingestion: Lineage Missing When Using admin_apis_only Mode – DataHub Customer Support Portal

Area: Ingestion Issues

Sub-Area: Cross-Platform Lineage (PowerBI + Databricks/dbt)

Issue

When ingesting PowerBI metadata into DataHub with admin_apis_only: true enabled in the PowerBI recipe, cross-platform lineage between PowerBI datasets and upstream sources such as Databricks or dbt fails to resolve. This occurs because the admin API does not expose dataset parameter values that are required to parse M-query expressions and construct accurate upstream URNs. As a result, the connector cannot determine which Databricks tables a given PowerBI dataset depends on, producing high counts of M-query resolver errors and missing lineage edges. The problem is compounded when URN casing is changed mid-lifecycle (e.g., by enabling convert_urns_to_lowercase or convert_lineage_urns_to_lowercase) without ensuring consistency across all related recipes.

Error Messages

m_query_resolver_errors: 336
m_query_resolver_no_lineage: 917

You Might Be Asking

Why is lineage between PowerBI and Databricks/dbt missing after I enabled admin_apis_only: true?
Can I get cross-platform lineage from PowerBI without granting access to actual dataset data?
Why did lineage stop working after I enabled convert_urns_to_lowercase and re-ingested my sources?
What does admin_apis_only actually prevent in terms of lineage resolution?

Solution

Understand the root cause: When admin_apis_only: true is set, the PowerBI connector relies exclusively on the Power BI Admin API, which does not return dataset parameter values. These parameter values — such as server hostnames, catalog names, and table names — are embedded in M-query expressions and are essential for resolving upstream lineage to platforms like Databricks. Without them, the connector cannot construct valid upstream URNs, and lineage will be absent or broken.

Disable admin_apis_only in your PowerBI recipe: Remove or set admin_apis_only: false to allow the connector to call the regular (non-admin) workspace API solely for the purpose of fetching dataset parameter metadata. This does not expose or export actual data from your PowerBI datasets — it only retrieves the structural M-query parameter definitions needed to resolve lineage.

source:
  type: powerbi
  config:
    client_id: '${PBI_APP_ID}'
    client_secret: '${PBI_SECRET}'
    tenant_id: '${PBI_TENANT_ID}'
    platform_instance: powerbi-<your-instance>
    extract_lineage: true
    native_query_parsing: true
    enable_advance_lineage_sql_construct: true
    extract_column_level_lineage: true
    convert_lineage_urns_to_lowercase: true
    # admin_apis_only: false  # This is the default; omit or explicitly set to false
    server_to_platform_instance:
      <your-databricks-server-hostname>:
        platform_instance: <your-databricks-platform-instance>

Grant the required Power BI Admin API permissions to your Entra application if you also need full admin API coverage (e.g., for tenant-wide metadata). In the Power BI or Fabric Admin Portal, navigate to Settings → Admin Portal → Tenant Settings and enable the following options, adding your Entra application's security group under Specific security groups for each:
- Service principals can access read-only admin APIs
- Enhance admin APIs responses with detailed metadata
- Enhance admin APIs responses with DAX and mashup expressions

Ensure URN casing is consistent across all recipes. If you have enabled convert_urns_to_lowercase: true in your Databricks or dbt recipes, you must also set convert_lineage_urns_to_lowercase: true in your PowerBI recipe. Additionally, verify that the platform_instance value used in the Databricks recipe exactly matches the value used in the server_to_platform_instance mapping in the PowerBI recipe — including casing. A mismatch here will cause URN misalignment and broken lineage.

# Databricks recipe
source:
  type: databricks
  config:
    platform_instance: <your-databricks-platform-instance>  # e.g., dbx-workspace-prod
    convert_urns_to_lowercase: true

# dbt recipe
source:
  type: dbt
  config:
    target_platform_instance: <your-databricks-platform-instance>  # Must match exactly
    convert_urns_to_lowercase: true

# PowerBI recipe
source:
  type: powerbi
  config:
    convert_lineage_urns_to_lowercase: true
    server_to_platform_instance:
      <your-databricks-server-hostname>:
        platform_instance: <your-databricks-platform-instance>  # Must match exactly

Re-run ingestions in the correct order after fixing configuration: Databricks → dbt → PowerBI. If PowerBI ingestion runs before the Databricks entities exist in DataHub, lineage edges will be dangling and may not resolve correctly on subsequent runs.
Clean up stale or soft-deleted entities. If you previously soft-deleted and re-ingested assets with different URN casing, ghost entities with mismatched URNs may still exist. You can identify these using the "Show Hidden Edges" option on the lineage graph in the DataHub UI. If old checkpoints reference stale entities, you may also need to reset stateful ingestion state before re-running.

Additional Notes

Setting admin_apis_only: false (the default) enables the PowerBI connector to call the regular workspace API to retrieve M-query parameter values only — it does not read, export, or stream actual data stored in PowerBI datasets. This is a structural metadata fetch, not a data access operation. However, it does require that the Entra application be added as a member of the relevant Power BI workspaces, or that appropriate workspace-level API permissions are granted. If your organization has strict controls around workspace membership, review the permission requirements with your Power BI administrator before making this change. Note also that admin_apis_only: true carries documented caveats: Report Pages are not ingested, Power BI Parameters are not resolved during M-query processing, and dataset profiling is unavailable. Cross-platform lineage to Databricks or other parameterized sources requires parameter resolution and therefore requires admin_apis_only: false.

Issue

Error Messages

You Might Be Asking

Solution

Additional Notes

Related Documentation