Context Document Best Practices: Structure, YAML Frontmatter, and Asset Linking – DataHub Customer Support Portal

Area: Best Practices

Sub-Area: AI-Powered Features / Context Documents

Issue

Teams authoring Context Documents for DataHub's AI-powered features (such as Ask DataHub and semantic search) often have questions about the correct two-tier document structure, appropriate granularity for domain versus metric-level documents, and how to properly configure YAML frontmatter so that documents are correctly linked to assets, domains, owners, and tags. A common pitfall is embedding YAML frontmatter as plain text within the document body rather than passing it as structured fields via the API, which causes DataHub to ignore all metadata associations entirely.

Error Messages

Context document has no linked assets, no domain, no owner, and no tags (YAML frontmatter parsed as plain text body content)

You Might Be Asking

Should I create separate context documents for domain overviews and individual metrics, or combine them into one document?
Is per-metric documentation too granular for context documents?
Why are my context documents not linked to any assets, domains, or owners after import?
How do I correctly pass YAML frontmatter so DataHub recognizes asset links and metadata?
What best practices should I follow when authoring and importing context documents at scale?

Solution

1. Use a Two-Tier Document Structure

A two-tier hierarchy is the recommended approach for organizing context documents:

Domain/overview documents: Cover cross-cutting concerns such as entity disambiguation, date window semantics, query access priority, and gotchas that apply across all related metrics or datasets. Associate these with your DataHub Domain entity.
Per-asset or per-metric documents: Drill into specifics for each individual metric, dataset, or asset without repeating domain-level content. Associate these with the relevant dataset or metric entity in DataHub.

Cross-reference documents in both directions using related_docs links (domain → metrics and metric → domain). This bidirectional linking helps DataHub's AI surface the correct context when constructing answers. Test that cross-references resolve correctly after import.

2. Per-Metric Documents Are Not Too Granular

Metric-level context documents are exactly the right level of granularity. The highest-value content in any metric document is explicit disambiguation guidance. Write out the questions an AI agent should resolve before answering a query about a given metric — for example, identifying the relevant queue type, whether the calculation is overall or first-time, any applicable slice, and the time window. This type of agent-oriented disambiguation section is what produces reliable AI answers and is strongly recommended for every metric document.

Additionally, include a Metric-Specific Gotchas section for edge cases that do not fit cleanly in the domain overview. Prioritize disambiguation and gotcha sections over exhaustive column catalogs, since column-level descriptions are typically already handled by your dbt YAML definitions.

3. Fix YAML Frontmatter — Pass Metadata as Fields, Not Plain Text

The most critical issue to resolve before importing documents at scale is ensuring YAML frontmatter is passed as structured API fields rather than embedded as plain text in the document body. When frontmatter is left as plain text, DataHub does not parse it, and the document will have no linked assets, no domain, no owner, and no tags.

When creating or updating a context document via the API, pass frontmatter values as explicit fields. The following example uses the DataHub REST API to create a document with proper field mapping:

POST /openapi/v3/entity/document
Content-Type: application/json

{
  "urn": "urn:li:document:(urn:li:dataPlatform:file,,PROD)",
  "aspects": {
    "documentProperties": {
      "value": {
        "title": "",
        "description": "# Overview\n\nYour markdown content here...",
        "contentType": "MARKDOWN"
      }
    },
    "ownership": {
      "value": {
        "owners": [
          {
            "owner": "urn:li:corpuser:",
            "type": "DATAOWNER"
          }
        ]
      }
    },
    "domains": {
      "value": {
        "domains": [
          "urn:li:domain:"
        ]
      }
    },
    "globalTags": {
      "value": {
        "tags": [
          {
            "tag": "urn:li:tag:"
          }
        ]
      }
    }
  }
}

To link a context document to a specific dataset asset, use the datahubDocumentLink relationship or associate the document with the dataset entity URN directly:

# Example: Linking a context document to a dataset using the Python SDK
from datahub.emitter.mce_builder import make_dataset_urn
from datahub.emitter.rest_emitter import DatahubRestEmitter
from datahub.metadata.schema_classes import DocumentPropertiesClass

emitter = DatahubRestEmitter(gms_server="https://.datahubproject.io/api/gms")

dataset_urn = make_dataset_urn(
    platform="dbt",
    name=".",
    env="PROD"
)

# Associate the context document with the dataset entity
# Refer to the Documents API tutorial for the full relationship pattern

Refer to the Documents API tutorial for the complete pattern for creating and linking documents programmatically.

4. Standardize Your Frontmatter Schema Before Scaling

Before authoring a large number of documents, formalize a consistent frontmatter schema. A recommended schema includes the following fields:

# Recommended context document metadata schema (passed as API fields, not plain text)
type: domain_overview | metric_doc | dataset_doc
owner: 
author: 
domain: 
tags:
  - 
  - 
linked_assets:
  - 
  - 
related_docs:
  - 
last_verified: YYYY-MM-DD
verified_by:

Consider validating this schema in CI to catch malformed documents before they are imported. If you include snapshot statistics (e.g., metric baseline values) in your documents, add a snapshot_as_of field and establish a review cadence. Stale statistics in a context document can produce confidently incorrect AI answers, which is worse than omitting them.

5. Bulk Import at Scale Using the Python SDK

For bulk import of context documents, use the acryl-datahub Python SDK with a file-based ingestion recipe. If your documents map 1:1 to dbt resources, generate entity URNs programmatically from your dbt manifest.json to avoid manual mapping errors:

import json

# Load dbt manifest
with open("manifest.json") as f:
    manifest = json.load(f)

# Extract metric or model URNs for linking
for node_name, node in manifest.get("nodes", {}).items():
    schema = node.get("schema", "")
    name = node.get("name", "")
    dataset_urn = f"urn:li:dataset:(urn:li:dataPlatform:dbt,{schema}.{name},PROD)"
    print(f"Linking context doc for: {node_name} -> {dataset_urn}")

Associate domain-level documents with your DataHub Domain entity and per-metric documents with the MetricFlow metric entities or their backing dataset entities, whichever surfaces most prominently in your team's DataHub workflows.

Additional Notes

As of DataHub version 2.1, the Metrics and Semantic Models entity model and UI are available. However, full ingestion paths for semantic models are expected in version 2.2, and dbt MetricFlow ingestion support remains on the roadmap. For teams using dbt MetricFlow, context documents are currently the best path for capturing metric-level knowledge in DataHub. The metric entity model is built to the OSI shape, so content authored in context documents today will port cleanly when native MetricFlow ingestion becomes available. Always verify current feature availability against your deployed DataHub version.