Skip to main content

Advanced Sibling Management: GraphQL, Troubleshooting, and Edge Cases

Avery
Updated

While DataHub automatically creates sibling relationships between dbt models and data warehouse tables, real-world data stacks often require manual intervention. This guide covers how to manage sibling relationships programmatically using GraphQL, troubleshoot complex merging logic, and resolve "gnarly" edge cases like circular dependencies or stale references.

1. Managing Siblings via GraphQL

The most reliable way to inspect and modify sibling relationships is through the GraphQL Explorer (/api/graphiql).

Inspecting Sibling Metadata

To see exactly how DataHub views the relationship (raw vs. merged), use this query. It reveals whether the entity is Primary and fetches the raw sibling URNs.

query GetDatasetSiblings {
  dataset(urn: "urn:li:dataset:(urn:li:dataPlatform:dbt,my_project.customers,PROD)") {
    urn
    siblings {
      isPrimary
      siblings  # Raw array of linked URNs
    }
    # siblingSearch returns the actual entity details
    siblingsSearch(input: { start: 0, count: 10 }) {
      searchResults {
        entity {
          ... on Dataset {
            urn
            platform { name }
            # Check specific metadata on the sibling
            schemaMetadata {
              fields { fieldPath, nativeDataType }
            }
          }
        }
      }
    }
  }
}

Manually Creating a Relationship

If the automatic hook fails (usually due to name mismatches), you can force a link.

mutation AddSiblingRelationship {
  addSiblings(
    input: {
      # The "Leader" entity (usually Data Warehouse)
      primary: "urn:li:dataset:(urn:li:dataPlatform:bigquery,project.dataset.customers,PROD)"
      # The "Secondary" entity (usually dbt)
      siblings: [
        "urn:li:dataset:(urn:li:dataPlatform:dbt,my_project.customers,PROD)"
      ]
    }
  )
}

Changing the Primary Sibling

To swap the Primary sibling (e.g., promoting dbt to Primary), you must perform a two-step operation: remove the old relationship, then add the new one.

mutation SetNewPrimary {
  # Step 1: Break the existing link
  removeSiblings(
    input: {
      primary: "urn:li:dataset:(urn:li:dataPlatform:bigquery,project.dataset.customers,PROD)"
      siblings: ["urn:li:dataset:(urn:li:dataPlatform:dbt,my_project.customers,PROD)"]
    }
  )
  
  # Step 2: Re-create with dbt as Primary
  addSiblings(
    input: {
      primary: "urn:li:dataset:(urn:li:dataPlatform:dbt,my_project.customers,PROD)" 
      siblings: ["urn:li:dataset:(urn:li:dataPlatform:bigquery,project.dataset.customers,PROD)"]
    }
  )
}

2. Troubleshooting Decision Trees

Scenario A: Sibling Relationship is Missing

If two entities exist but are not linked:

  1. Check Existence: Do both entities exist separately in DataHub?
  2. Check Names: Does the dbt dataset name exactly match the warehouse table name? (Case sensitivity matters).
  3. Check Config: Is siblings.enabled=true in your GMS configuration?
  4. Check Logs: Look for SiblingAssociationHook errors in GMS logs.
  5. Resolution: If the names don't match (e.g., dbt_model vs DB_TABLE), use the addSiblings mutation above to link them manually.

Scenario B: Metadata is Not Merging

If the Combined View is missing expected data:

  1. Verify View Mode: Ensure ?separate_siblings=true is not in the URL.
  2. Check Source: Navigate to the individual sibling (using the URL parameter). Does the metadata exist on the source entity?
    • Example: If dbt documentation is missing, check the individual dbt entity. If it's missing there, the issue is ingestion, not siblings.
  3. Check Lineage: Remember that Lineage is NOT merged. It only shows relationships for the specific URN you navigated to.

3. Resolving Edge Cases

Circular Dependencies (The "Double Primary" Bug)

Symptoms: Infinite loops in UI, timeouts, or inconsistent metadata.
Cause: Both Entity A and Entity B are marked as isPrimary: true.
Detection: Query both entities. If both return isPrimary: true, the state is invalid.
Fix: Use removeSiblings to break the link, then addSiblings to re-establish it with a single Primary.

Stale / Deleted Siblings

Symptoms: The UI shows "1 Sibling" but clicking it results in an error or 404.
Cause: A sibling entity was hard-deleted, but the surviving entity still contains the deleted URN in its siblings aspect.
Detection:

siblings { siblings }  # Returns ["urn:li:dataset:DELETED_ENTITY"]
siblingsSearch { count }  # Returns 0

Fix: Run removeSiblings targeting the deleted URN to clean up the aspect.

Schema Mismatch (Raw vs. Transformed)

Symptoms: Combined schema shows duplicate columns or a mix of raw and transformed fields.
Cause: dbt model linked to the source table (raw) instead of the target table (transformed).
Fix:

  1. Break the relationship (removeSiblings).
  2. Re-ingest dbt with the correct target_platform configuration to match the transformed table.

3+ Siblings (The "Triplets" Scenario)

Context: dbt, Snowflake, and Looker all representing the same dataset.
Behavior: DataHub supports this. The siblings array will contain multiple URNs.
Performance Warning: Merging 3+ entities, especially with large schemas, is computationally expensive. If page loads are slow, consider enabling SHOW_SEPARATE_SIBLINGS=true globally.

4. Automation with Python

For bulk management (e.g., fixing hundreds of incorrect links), use the DatahubRestEmitter.

from datahub.emitter.rest_emitter import DatahubRestEmitter
from datahub.metadata.com.linkedin.pegasus2avro.dataset import Siblings

# Initialize emitter
emitter = DatahubRestEmitter(gms_server="http://localhost:8080")

def link_siblings(primary_urn, sibling_urns):
    """
    Manually links siblings. 
    primary_urn: The 'Leader' (e.g., BigQuery)
    sibling_urns: List of 'Followers' (e.g., dbt)
    """
    # 1. Emit aspect for the Primary
    primary_aspect = Siblings(primary=True, siblings=sibling_urns)
    emitter.emit_mcp(
        entityUrn=primary_urn,
        aspectName="siblings",
        aspect=primary_aspect
    )
    
    # 2. Emit aspect for each Secondary sibling
    for sibling in sibling_urns:
        # The sibling list for a secondary must include the primary 
        # plus any other siblings in the group
        all_others = [primary_urn] + [s for s in sibling_urns if s != sibling]
        
        secondary_aspect = Siblings(primary=False, siblings=all_others)
        emitter.emit_mcp(
            entityUrn=sibling,
            aspectName="siblings",
            aspect=secondary_aspect
        )

# Example Usage
primary = "urn:li:dataset:(urn:li:dataPlatform:bigquery,project.dataset.customers,PROD)"
secondary = "urn:li:dataset:(urn:li:dataPlatform:dbt,my_project.customers,PROD)"

link_siblings(primary, [secondary])

 

 

Related to

Related articles