Managing Metadata from GitHub Using Ingestion Pipelines and CI/CD – DataHub Customer Support Portal

Area: Best Practices

Sub-Area: Metadata-as-Code / CI/CD Integration

Issue

Organizations managing DataHub metadata at scale often need a repeatable, auditable, and automated way to keep metadata definitions — such as glossary terms, domains, tags, ownership, and documentation — in sync with their data platform. Manually applying metadata changes through the UI or ad hoc CLI commands introduces drift, lacks version control, and makes it difficult to review or roll back changes. The recommended pattern is to treat metadata as code, store definitions in a GitHub repository, and trigger DataHub ingestion pipelines automatically via GitHub Actions or other CI/CD workflows whenever changes are merged.

You Might Be Asking

How do I version-control DataHub metadata and apply changes automatically?
Can I use GitHub Actions to run DataHub ingestion pipelines on pull request merge?
What is the recommended way to manage glossary terms, domains, and tags as code in DataHub?
How do I authenticate a CI/CD pipeline to push metadata to DataHub?
How can I preview or validate metadata changes before they are applied to production?

Solution

Organize your metadata definitions as YAML files in a GitHub repository.
Create a dedicated repository (or a directory within an existing monorepo) to store metadata definition files. Use the DataHub ingestion source formats that support file-based input, such as the datahub CLI ingest command with YAML recipe files, or structured YAML files consumed by sources like datahub-rest.

Example directory layout:
```
metadata/
  glossary/
    terms.yaml
  domains/
    domains.yaml
  tags/
    tags.yaml
  ownership/
    owners.yaml
recipes/
  glossary_ingestion.yaml
  domains_ingestion.yaml
```
Define ingestion recipes pointing to your metadata files.
Each recipe file tells the DataHub ingestion framework where to read metadata and where to emit it. Below is an example recipe for ingesting glossary terms from a local YAML file:
```
# recipes/glossary_ingestion.yaml
source:
  type: datahub-business-glossary
  config:
    file: metadata/glossary/terms.yaml

sink:
  type: datahub-rest
  config:
    server: "https://.datahubproject.io/api/gms"
    token: "${DATAHUB_TOKEN}"
```
Use environment variable substitution (e.g., ${DATAHUB_TOKEN}) for all secrets so that credentials are never stored in the repository.
Store credentials as GitHub Secrets.
In your GitHub repository, navigate to Settings → Secrets and variables → Actions and add the following secrets:
- DATAHUB_GMS_URL — the URL of your DataHub GMS endpoint, e.g., https://<your-instance>.datahubproject.io/api/gms
- DATAHUB_TOKEN — a DataHub Personal Access Token or a service account token with write permissions

Create a GitHub Actions workflow to run ingestion on merge.

Add a workflow file to your repository that installs the DataHub CLI and triggers ingestion whenever changes to metadata files are pushed to the main branch:

# .github/workflows/datahub-metadata-sync.yml
name: DataHub Metadata Sync

on:
  push:
    branches:
      - main
    paths:
      - "metadata/**"
      - "recipes/**"

jobs:
  ingest-metadata:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Install DataHub CLI
        run: |
          pip install --upgrade acryl-datahub

      - name: Verify DataHub CLI installation
        run: datahub version

      - name: Ingest Glossary Terms
        env:
          DATAHUB_GMS_URL: ${{ secrets.DATAHUB_GMS_URL }}
          DATAHUB_TOKEN: ${{ secrets.DATAHUB_TOKEN }}
        run: |
          datahub ingest -c recipes/glossary_ingestion.yaml

      - name: Ingest Domains
        env:
          DATAHUB_GMS_URL: ${{ secrets.DATAHUB_GMS_URL }}
          DATAHUB_TOKEN: ${{ secrets.DATAHUB_TOKEN }}
        run: |
          datahub ingest -c recipes/domains_ingestion.yaml

Add additional steps for each recipe file as needed.

Add a dry-run validation step for pull requests.

To catch errors before changes reach production, add a separate workflow that runs on pull requests and uses the --dry-run flag (or a preview sink) to validate the metadata without writing to DataHub:

# .github/workflows/datahub-metadata-validate.yml
name: DataHub Metadata Validate (PR)

on:
  pull_request:
    paths:
      - "metadata/**"
      - "recipes/**"

jobs:
  validate-metadata:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Install DataHub CLI
        run: pip install --upgrade acryl-datahub

      - name: Dry-run Glossary Ingestion
        run: |
          datahub ingest -c recipes/glossary_ingestion.yaml --dry-run

      - name: Dry-run Domains Ingestion
        run: |
          datahub ingest -c recipes/domains_ingestion.yaml --dry-run

This gives reviewers confidence that the metadata YAML is valid and parseable before approving a pull request.

Use pull requests as your change-review process.
Enforce branch protection rules on the main branch so that all metadata changes go through a pull request. This provides a built-in audit trail, allows peer review of metadata additions or modifications, and ensures the dry-run validation passes before any change is applied to DataHub.
Extend the pattern to other metadata types.
The same approach applies to any metadata that can be expressed as code and ingested via the DataHub CLI or REST API, including:
- Business glossary terms and term groups (datahub-business-glossary source)
- Domains and sub-domains
- Tags and tag associations
- Dataset documentation and schema field descriptions (via datahub-rest or the Python SDK emitter)
- Data product definitions
- Custom ownership types and assignments

Additional Notes

Ensure the DataHub Personal Access Token used in CI/CD has the minimum required permissions — typically Metadata Write — and rotate it periodically following your organization's security policies. If you are running DataHub on-premises or in a private network, ensure your GitHub Actions runners (self-hosted or otherwise) have network access to your DataHub GMS endpoint. The --dry-run flag behavior may vary slightly between DataHub CLI versions; always pin the CLI version in your CI workflow (e.g., pip install acryl-datahub==0.14.x.x) to ensure reproducible builds. When managing large volumes of metadata, consider splitting recipes into smaller, focused files to improve pipeline observability and make failures easier to diagnose. This pattern works with any CI/CD platform (GitLab CI, Jenkins, CircleCI, etc.) — the GitHub Actions examples above can be adapted straightforwardly to other systems.

Issue

You Might Be Asking

Solution

Additional Notes

Related Documentation