CI/CD Pipeline for Glossary Management Across Environments

Organizations need to manage DataHub glossary terms across multiple environments (sandbox, staging, production) using automated workflows. Teams want to validate, version-control, and synchronize glossaries using CI/CD pipelines with Python scripts, Docker containers, and build automation tools like Buildkite, Jenkins, or GitHub Actions.

You Might Be Asking:

• Is a Python-based CI/CD pipeline the recommended approach for managing glossaries?
• Are there native features for glossary synchronization between environments?
• What's the best practice for version-controlling glossary definitions?
• How should we handle glossary term application to datasets in automation?

Using Python-based CI/CD pipelines for glossary management is a recommended approach that aligns with current DataHub best practices.

Recommended Approach:

1. Version Control Glossary Definitions:

Store glossary terms as YAML or JSON files in Git:

# glossary_terms.yaml
terms:
  - name: "PII"
    description: "Personally Identifiable Information"
    term_source: "Security Team"
    related_terms:
      - "Sensitive Data"
      - "GDPR"

  - name: "Revenue"
    description: "Total income generated from sales"
    term_source: "Finance Team"
    owners:
      - "urn:li:corpuser:finance-team"

2. Validation Script:

# validate_glossary.py
import yaml
from datahub.emitter.rest_emitter import DatahubRestEmitter
from datahub.metadata.schema_classes import GlossaryTermInfoClass

def validate_glossary(file_path):
    """Validate glossary structure before upload"""
    with open(file_path) as f:
        glossary = yaml.safe_load(f)

    for term in glossary['terms']:
        # Validate required fields
        assert 'name' in term
        assert 'description' in term
        # Additional validation logic

    print("✓ Glossary validation passed")
    return True

3. Upload Script:

# upload_glossary.py
from datahub.emitter.rest_emitter import DatahubRestEmitter
from datahub.emitter.mce_builder import make_glossary_term_urn
from datahub.metadata.schema_classes import GlossaryTermInfoClass
from datahub.emitter.mcp import MetadataChangeProposalWrapper

def upload_glossary(terms, datahub_url, token):
    emitter = DatahubRestEmitter(
        gms_server=datahub_url,
        token=token
    )

    for term in terms:
        term_urn = make_glossary_term_urn(term['name'])

        term_info = GlossaryTermInfoClass(
            definition=term['description'],
            termSource=term.get('term_source', ''),
            sourceRef=term.get('source_ref', '')
        )

        mcp = MetadataChangeProposalWrapper(
            entityUrn=term_urn,
            aspect=term_info
        )

        emitter.emit_mcp(mcp)

    print(f"✓ Uploaded {len(terms)} glossary terms")

4. CI/CD Pipeline Configuration:

Example: Buildkite:

# .buildkite/pipeline.yml
steps:
  - label: "Validate Glossary"
    command: "python validate_glossary.py glossary_terms.yaml"

  - label: "Upload to Sandbox"
    command: |
      docker run --rm \
        -v $(pwd):/data \
        -e DATAHUB_URL=https://sandbox.acryl.io \
        -e DATAHUB_TOKEN=$SANDBOX_TOKEN \
        acryldata/datahub-ingestion:latest \
        python /data/upload_glossary.py
    branches: "develop"

  - label: "Upload to Production"
    command: |
      python upload_glossary.py \
        --url https://prod.acryl.io \
        --token $PROD_TOKEN \
        --terms glossary_terms.yaml
    branches: "main"

Example: GitHub Actions:

# .github/workflows/glossary-sync.yml
name: Sync Glossary
on:
  push:
    branches: [main]
    paths: ['glossary/**']

jobs:
  sync:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2

      - name: Setup Python
        uses: actions/setup-python@v2
        with:
          python-version: '3.10'

      - name: Install DataHub CLI
        run: pip install acryl-datahub

      - name: Upload Glossary
        env:
          DATAHUB_URL: ${{ secrets.DATAHUB_URL }}
          DATAHUB_TOKEN: ${{ secrets.DATAHUB_TOKEN }}
        run: python upload_glossary.py

5. Apply Terms to Datasets:

# apply_terms.py
from datahub.emitter.rest_emitter import DatahubRestEmitter
from datahub.metadata.schema_classes import GlossaryTermsClass
from datahub.emitter.mcp import MetadataChangeProposalWrapper

def apply_terms_to_datasets(mappings, datahub_url, token):
    """Apply glossary terms to datasets based on mappings"""
    emitter = DatahubRestEmitter(
        gms_server=datahub_url,
        token=token
    )

    for dataset_urn, terms in mappings.items():
        term_urns = [
            f"urn:li:glossaryTerm:{term}" 
            for term in terms
        ]

        glossary_terms = GlossaryTermsClass(
            terms=[
                {"urn": term_urn}
                for term_urn in term_urns
            ],
            auditStamp={
                "time": int(time.time() * 1000),
                "actor": "urn:li:corpuser:automation"
            }
        )

        mcp = MetadataChangeProposalWrapper(
            entityUrn=dataset_urn,
            aspect=glossary_terms
        )

        emitter.emit_mcp(mcp)

Best Practices:

1. Separate glossary definitions from code in version control
2. Validate before deployment to catch errors early
3. Use environment-specific tokens secured in CI/CD secrets
4. Implement approval workflows for production changes
5. Maintain audit logs of glossary changes
6. Test in sandbox before promoting to production

Native Synchronization Features:

Currently, DataHub does not provide built-in native glossary synchronization between environments. The Python-based CI/CD approach is the recommended method for managing glossaries across environments.

• This approach works for glossaries, tags, domains, and other metadata
• Can be extended to include term deprecation workflows
• Supports automatic term assignment based on dataset patterns
• Compatible with any CI/CD platform (Jenkins, GitLab CI, CircleCI, etc.)
• Recommended for organizations with multiple DataHub instances

• Glossary Terms
• Python SDK

• 4928