Accessing Specific Datasets by Platform Instance via GraphQL

Users need to query DataHub to retrieve all datasets associated with a specific platform instance (e.g., all Snowflake datasets from a particular account or all BigQuery datasets from a specific project). The standard search queries may not efficiently filter by platform instance, requiring knowledge of the correct GraphQL query structure.

You Might Be Asking:

• How do I get all datasets for a specific platform instance?
• What GraphQL query filters by platform instance?
• Can I use the UI to discover the correct GraphQL query?
• How do I find datasets from a specific Snowflake account or BigQuery project?

Use the DataHub UI's network console to discover GraphQL queries, or use the search API with platform instance filters.

Method 1: Discover Queries from UI (Recommended)

The easiest way to find the correct GraphQL query is to use the DataHub UI and inspect network requests:

Steps: 1. Open DataHub UI in your browser 2. Open Developer Tools (F12 or Right-click → Inspect) 3. Go to the "Network" tab 4. In DataHub UI, perform a search filtered by platform instance 5. Look for GraphQL requests in the Network tab 6. Copy the query from the request payload

Example of What You'll Find:

query searchAcrossEntities($input: SearchAcrossEntitiesInput!) {
  searchAcrossEntities(input: $input) {
    start
    count
    total
    searchResults {
      entity {
        urn
        type
        ... on Dataset {
          name
          platform {
            name
          }
          properties {
            name
          }
        }
      }
    }
  }
}

Method 2: Direct GraphQL Query

Query All Datasets for a Platform Instance:

query {
  search(
    input: {
      type: DATASET,
      query: "*",
      start: 0,
      count: 100,
      filters: [
        {
          field: "platform",
          values: ["snowflake"]
        },
        {
          field: "platformInstance",
          values: ["prod-us-east-1"]
        }
      ]
    }
  ) {
    start
    count
    total
    searchResults {
      entity {
        urn
        ... on Dataset {
          name
          properties {
            name
            description
          }
          platform {
            name
          }
          platformInstance {
            instanceId
          }
        }
      }
    }
  }
}

Method 3: Using Python SDK:

from datahub.emitter.rest_emitter import DatahubRestEmitter
import requests

def get_datasets_by_platform_instance(
    platform: str,
    platform_instance: str,
    datahub_url: str,
    token: str
):
    """Get all datasets for a specific platform instance"""

    headers = {"Authorization": f"Bearer {token}"}

    query = """
    query($input: SearchInput!) {
      search(input: $input) {
        total
        searchResults {
          entity {
            urn
            ... on Dataset {
              name
              platform {
                name
              }
              platformInstance {
                instanceId
              }
            }
          }
        }
      }
    }
    """

    variables = {
        "input": {
            "type": "DATASET",
            "query": "*",
            "start": 0,
            "count": 1000,
            "filters": [
                {
                    "field": "platform",
                    "values": [platform]
                },
                {
                    "field": "platformInstance",
                    "values": [platform_instance]
                }
            ]
        }
    }

    response = requests.post(
        f"{datahub_url}/api/graphql",
        json={"query": query, "variables": variables},
        headers=headers
    )

    return response.json()

# Usage
datasets = get_datasets_by_platform_instance(
    platform="snowflake",
    platform_instance="prod.us-east-1.aws",
    datahub_url="https://your-instance.acryl.io",
    token="your-token"
)

print(f"Found {datasets['data']['search']['total']} datasets")

Method 4: Using DataHub CLI:

# Search via CLI
datahub get --urn "urn:li:dataset:(urn:li:dataPlatform:snowflake,*,PROD)" \
  --aspect platform

# Use jq to filter by platform instance
datahub search --entity dataset \
  --query "*" \
  --filter "platform=snowflake" \
  --filter "platformInstance=prod-us-east-1" \
  | jq '.[] | select(.platformInstance.instanceId == "prod-us-east-1")'

Pagination for Large Results:

def get_all_datasets_paginated(platform, platform_instance, datahub_url, token):
    """Get all datasets with pagination"""
    all_datasets = []
    start = 0
    count = 100

    while True:
        variables = {
            "input": {
                "type": "DATASET",
                "query": "*",
                "start": start,
                "count": count,
                "filters": [
                    {"field": "platform", "values": [platform]},
                    {"field": "platformInstance", "values": [platform_instance]}
                ]
            }
        }

        response = requests.post(
            f"{datahub_url}/api/graphql",
            json={"query": QUERY, "variables": variables},
            headers={"Authorization": f"Bearer {token}"}
        )

        results = response.json()['data']['search']['searchResults']
        all_datasets.extend(results)

        if len(results) 

Common Platform Instance Patterns:

• Snowflake: account-name.region.cloud (e.g., xy12345.us-east-1.aws)
• BigQuery: project-id (e.g., my-gcp-project)
• Redshift: cluster-name.region (e.g., prod-cluster.us-east-1)
• Databricks: workspace-url (e.g., company.cloud.databricks.com)

Self-Service Query Discovery:

The recommended approach is to use the UI's network console because: 1. Guarantees correct query structure 2. Shows actual working queries 3. Includes all necessary fields 4. Reflects current API version 5. Works for complex filters

• Platform instance is set during ingestion via the platform_instance config option
• Not all datasets have platform instances; some may only have platform
• Use wildcards in the query field to match all datasets
• GraphQL playground is available at https://your-instance.acryl.io/api/graphiql

• GraphQL API
• Search API

• 4246