Managing the DataHub Business Glossary with Git and CI/CD (GitOps Approach) – DataHub Customer Support Portal

Area: Best Practices

Sub-Area: Business Glossary Management / GitOps

Issue

Teams want to use a Git repository as the authoritative source of truth for their DataHub Business Glossary, automating ingestion into DataHub on merge via CI/CD pipelines. Multiple approaches exist (Terraform, CLI, custom SDK), and it is not immediately clear which is best suited for long-term maintainability, simplicity, and extensibility — including potential future bi-directional sync between DataHub and Git.

You Might Be Asking

What is the recommended way to manage my DataHub Business Glossary with Git?
How do I automatically sync a YAML glossary file to DataHub on pull request merge?
Should I use Terraform, the DataHub CLI, or a custom SDK to manage glossary terms?
What is the business-glossary-sync-action and when should I use it?
How do I avoid race conditions in a bi-directional glossary sync setup?
What does enable_auto_id do and how must it be configured for consistent syncing?
What happens if I rename a glossary term — does its URN change?
Which glossary fields does ingestion overwrite versus leave untouched?

Solution

Step 1: Choose the Right Approach

The recommended and most widely adopted pattern is the DataHub CLI + YAML + CI/CD approach. The datahub-business-glossary ingestion source carries DataHub's Certified support tier (production-tested, actively maintained) and is the standard mechanism for this use case.

Comparison of approaches:

CLI + YAML + GitHub Actions — Best fit. Low complexity, high extensibility. Recommended starting point.
Terraform — No native Terraform resources exist for Business Glossary terms. The DataHub Terraform provider covers infrastructure and platform configs, not glossary metadata content. Not recommended for glossary management.
Custom SDK — Appropriate only when programmatic logic is required (e.g., dynamically pulling definitions from another system). For human-authored glossaries, the YAML source is simpler and preferred.

Step 2: Define Your Glossary in YAML

Create a business_glossary.yml file in your Git repository. The format supports nested term groups, relationships, ownership, domains, custom properties, and Markdown descriptions.

version: "1"
source: DataHub
owners:
  users:
    - user@example.com
nodes:
  - name: Classification
    description: "Data classification terms"
    terms:
      - name: PII
        id: "pii"
        description: "Personally Identifiable Information"
        owners:
          users:
            - user@example.com
      - name: Confidential
        id: "confidential"
        description: "Confidential business data"
  - name: Finance
    description: "Financial metric terms"
    terms:
      - name: MRR
        id: "mrr"
        description: "Monthly Recurring Revenue"

Key design tip: For any term that will be referenced by lineage or downstream systems, set an explicit id: field (e.g., id: "mrr"). This ID is taken verbatim when computing the URN, providing a stable reference point across reorganizations.

Step 3: Create an Ingestion Recipe

Create a recipe file (e.g., glossary_recipe.dhub.yaml) that points to your YAML file:

source:
  type: datahub-business-glossary
  config:
    file: business_glossary.yml
    enable_auto_id: true   # Recommended — see notes below

sink:
  type: datahub-rest
  config:
    server: "https://<your-instance>.datahubproject.io/api/gms"
    token: "${DATAHUB_TOKEN}"

Step 4: Wire Up CI/CD with the Official GitHub Action

Use the acryldata/business-glossary-sync-action to automate sync on merge. This action abstracts CLI setup and makes the CI/CD wiring straightforward.

Example GitHub Actions workflow (.github/workflows/sync-glossary.yml):

name: Sync Business Glossary to DataHub

on:
  push:
    branches:
      - main
    paths:
      - 'business_glossary.yml'

jobs:
  sync:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Sync Glossary to DataHub
        uses: acryldata/business-glossary-sync-action@v1
        with:
          datahub_server: "https://<your-instance>.datahubproject.io/api/gms"
          datahub_token: ${{ secrets.DATAHUB_TOKEN }}
          glossary_file: business_glossary.yml
          enable_auto_id: "true"

Set DATAHUB_TOKEN as a GitHub Actions secret containing a DataHub Personal Access Token (PAT) with appropriate permissions.

Step 5: Understanding `enable_auto_id` and URN Stability

The enable_auto_id setting controls how term URNs are computed:

false (default): URNs are derived from the term's path — spaces become hyphens, special characters are stripped, and the path is joined with periods (e.g., urn:li:glossaryTerm:Classification.PII). Readable, but renaming or moving a term changes its URN.
true (recommended): Generates stable GUID-based IDs. Required for non-ASCII names and preferred for uniqueness guarantees.

Critical requirement: Whatever value you choose for enable_auto_id in the ingestion recipe must match exactly the value set on the business-glossary-sync-action. Mismatching values causes the two sides to compute different URNs and conflict on every run.

Important — renaming terms: In both enable_auto_id modes, a URN is derived from the term's name and its position in the hierarchy. Renaming a term or moving it under a different node produces a new URN. Because ingestion is an upsert (never a delete), this results in a brand-new term being created while the old one is left orphaned with any UI-curated metadata still attached. The only way to make a term survive a rename is to set an explicit id: field — that value is taken verbatim and is not affected by name or hierarchy changes.

Step 6: Understand What Ingestion Overwrites vs. Leaves Alone

Glossary ingestion is an upsert keyed on the term's URN. The following fields are overwritten on every ingestion run:

Term name
Description
Custom properties
Owners (rewritten on every run, even if not set in the YAML)
Relationships (related terms, inherits from, contains)

The following aspects live on separate metadata aspects and are never touched by glossary ingestion:

Structured properties
Deprecation flags
Applying a term onto a dataset or other asset

A term not present in the YAML file will not be deleted — ingestion is a partial upsert, not a full state replacement. Keeping the file complete still matters for a clean DataHub-to-Git diff, but omitting a term from the file will not remove it from DataHub.

Step 7 (Optional): Bi-Directional Sync and Race Condition Mitigations

The business-glossary-sync-action supports a DataHub → Git direction: it can detect changes made in the DataHub UI and open a pull request updating the YAML to match. A typical bi-directional flow looks like:

Your YAML is the source of truth → ingested into DataHub (Git → DataHub).
A user adds or edits terms in the DataHub UI.
The sync action runs on a schedule, detects the drift, and opens a PR updating the YAML.
The PR is merged, and the next ingestion run keeps Git and DataHub consistent.

Race condition risk: When both write paths are active, a silent overwrite scenario is possible:

A user edits a term in the DataHub UI at 2:00 PM.
A PR merges in Git with a different change to the same term; the merge-triggered ingestion runs at 2:01 PM and overwrites the UI edit silently — no conflict is surfaced.
The sync action runs at 2:15 PM, reads DataHub (which now reflects Git's version), sees no diff, and does nothing. The UI edit is permanently lost.

There is no built-in conflict detection in DataHub to catch this. The following mitigations are available:

Recommended — Git-only write path (closes the window entirely): Restrict glossary editing permissions using a DataHub Metadata Policy scoped to glossaryTerm and glossaryNode entity types, making the UI effectively read-only for glossary content. The DataHub → Git direction then becomes a drift-detection safety net rather than a competing write path. This is the cleanest model if Git is intended to be fully authoritative.
If UI editing must remain open — shrink the window:
- Run the sync action on a short schedule (e.g., every 5–15 minutes) so unmerged UI changes are captured into a PR quickly.
- Run the sync action as the first step of the same workflow that performs ingestion, so any pending UI change is read and committed to Git before Git writes back to DataHub.
- Communicate clearly to users that the UI is not a guaranteed durable write path for glossary terms.
These mitigations reduce the gap to minutes but do not eliminate it.

Additional Notes

The datahub-business-glossary ingestion source carries DataHub's Certified support tier, indicating it is production-tested and actively maintained across DataHub versions. Because ingestion is a per-URN upsert, re-running ingestion is idempotent — it updates existing entities rather than duplicating them. If you plan to allow non-technical users to contribute glossary definitions, consider using the business-glossary-sync-action in DataHub → Git mode to bootstrap a YAML export from your existing DataHub instance as a starting point. The YAML format supports Markdown in description fields, making it readable in pull request diffs and friendly for review by non-engineers. Treat term owners as a Git-managed field: because ingestion rewrites owners on every run, UI-assigned ownership will be overwritten unless it is also reflected in the YAML.