Unity Catalog (Databricks) Metadata Sync

Databricks Unity Catalog metadata is not syncing to DataHub, or lineage from notebooks and workflows is incomplete. This typically occurs due to API token permissions, workspace configuration, or catalog-level access issues.

You Might Be Asking:

• How do I sync Unity Catalog to DataHub?
• What permissions are needed for Unity Catalog ingestion?
• Can DataHub extract lineage from Databricks notebooks?

1. Configure Unity Catalog source:

Example for Databricks Unity Catalog. This can be adapted for other cloud data catalog services.

source:
  type: unity-catalog
  config:
    workspace_url: "https://your-workspace.cloud.databricks.com"
    token: "${DATABRICKS_TOKEN}"

    # Specify catalogs and schemas
    catalog_pattern:
      allow: ["prod_catalog", "analytics_catalog"]

    schema_pattern:
      allow: [".*"]
      deny: [".*_temp$"]

    # Include options
    include_table_lineage: true
    include_column_lineage: true
    include_notebooks: true
    include_ownership: true

2. Create service principal with proper access:

Example for Databricks Unity Catalog. This can be adapted for other cloud data catalog services.

-- In Databricks SQL
-- Grant catalog access
GRANT USE CATALOG ON CATALOG prod_catalog TO `service-principal-id`;
GRANT USE SCHEMA ON SCHEMA prod_catalog.analytics TO `service-principal-id`;
GRANT SELECT ON SCHEMA prod_catalog.analytics TO `service-principal-id`;

-- Grant access to all tables in schema
GRANT SELECT ON ALL TABLES IN SCHEMA prod_catalog.analytics TO `service-principal-id`;

-- For lineage extraction
GRANT USE CATALOG ON CATALOG system TO `service-principal-id`;
GRANT SELECT ON system.access.table_lineage TO `service-principal-id`;

3. Extract notebook lineage:

Example for Databricks Unity Catalog. This can be adapted for other cloud data catalog services.

source:
  type: unity-catalog
  config:
    include_notebooks: true
    notebook_pattern:
      allow: ["/Production/.*"]

    # Parse notebook commands for lineage
    parse_notebook_sql: true

    # Include notebook run history
    include_usage_stats: true

4. Map external tables to DataHub platforms:

Example for Databricks Unity Catalog with external storage. This can be adapted for other cloud data catalog services.

source:
  type: unity-catalog
  config:
    # Map external locations to platforms
    external_location_map:
      "s3://prod-data/": "s3"
      "abfss://container@storage.dfs.core.windows.net/": "azure_datalake"

    platform_instance_map:
      "s3://prod-data/": "prod_s3"

5. Handle Delta Lake tables:

Example for Databricks Unity Catalog. This can be adapted for other cloud data catalog services.

source:
  type: unity-catalog
  config:
    # Include Delta table metadata
    include_delta_metadata: true

    # Extract Delta table properties
    include_table_properties: true

    # This captures:
    # - Delta version
    # - Table properties
    # - Partitioning scheme

Unity Catalog lineage extraction uses the system.access.table_lineage table which requires appropriate permissions. Notebook lineage depends on SQL parsing and may not capture all complex transformations. Consider multiple ingestion runs for different catalog levels.

• Unity Catalog Source
• Databricks Integration