Configuring Multiple DataHub Environments with Consistent Metadata

Organizations running multiple DataHub environments (development, staging, production) need guidance on keeping certain metadata elements consistent across environments while allowing environment-specific data. Common challenges include synchronizing glossaries, domains, tags, and policies across instances while maintaining environment-specific dataset catalogs and lineage.

You Might Be Asking:

• How do I keep glossaries synchronized across environments?
• Should I have separate DataHub instances per environment?
• How do I promote metadata changes from dev to production?
• Can I share certain metadata while keeping datasets separate?
• What's the recommended architecture for multi-environment DataHub?

Implement a multi-environment strategy with infrastructure-as-code for metadata synchronization while maintaining separate DataHub instances per environment.

Recommended Architecture:

┌─────────────────────────────────────────────────────┐
│          Shared Repository (Git)                    │
│                                                      │
│  ├── glossaries/                                    │
│  │   └── terms.yaml                                 │
│  ├── domains/                                       │
│  │   └── domains.yaml                               │
│  ├── tags/                                          │
│  │   └── tags.yaml                                  │
│  └── policies/                                      │
│      └── access_policies.yaml                       │
│                                                      │
└────────────┬──────────────────┬────────────────────┘
             │                  │
             ▼                  ▼
     ┌───────────────┐  ┌───────────────┐
     │  Dev DataHub  │  │ Prod DataHub  │
     │               │  │               │
     │ • Dev Data    │  │ • Prod Data   │
     │ • Shared Tags │  │ • Shared Tags │
     │ • Shared Terms│  │ • Shared Terms│
     └───────────────┘  └───────────────┘

Strategy 1: Shared Metadata, Separate Instances

What to Share: - Glossary terms and business definitions - Domain structures - Tag taxonomies - Data classification policies - Access control policies (adapted per environment)

What to Keep Separate: - Dataset catalogs - Lineage relationships - Usage statistics - Data quality assertions - Environment-specific configurations

Implementation Using CI/CD:

# .github/workflows/sync-metadata.yml
name: Sync Metadata Across Environments

on:
  push:
    branches: [main]
    paths:
      - 'metadata/**'

jobs:
  sync-dev:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Setup Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.10'

      - name: Install DataHub CLI
        run: pip install acryl-datahub

      - name: Sync Glossary to Dev
        env:
          DATAHUB_URL: ${{ secrets.DEV_DATAHUB_URL }}
          DATAHUB_TOKEN: ${{ secrets.DEV_DATAHUB_TOKEN }}
        run: |
          python scripts/sync_glossary.py --env dev

      - name: Sync Domains to Dev
        run: |
          python scripts/sync_domains.py --env dev

  sync-prod:
    runs-on: ubuntu-latest
    needs: sync-dev
    environment: production
    steps:
      # Similar steps for production
      - name: Sync Glossary to Prod
        env:
          DATAHUB_URL: ${{ secrets.PROD_DATAHUB_URL }}
          DATAHUB_TOKEN: ${{ secrets.PROD_DATAHUB_TOKEN }}
        run: |
          python scripts/sync_glossary.py --env prod

Sync Script Example:

# scripts/sync_glossary.py
import yaml
from datahub.emitter.rest_emitter import DatahubRestEmitter
from datahub.emitter.mce_builder import make_glossary_term_urn
from datahub.metadata.schema_classes import GlossaryTermInfoClass
from datahub.emitter.mcp import MetadataChangeProposalWrapper
import os
import sys

def load_glossary(file_path):
    """Load glossary from YAML"""
    with open(file_path) as f:
        return yaml.safe_load(f)

def sync_glossary_to_environment(glossary, environment):
    """Sync glossary to specific environment"""

    # Get environment-specific config
    config = {
        'dev': {
            'url': os.environ['DEV_DATAHUB_URL'],
            'token': os.environ['DEV_DATAHUB_TOKEN']
        },
        'staging': {
            'url': os.environ['STAGING_DATAHUB_URL'],
            'token': os.environ['STAGING_DATAHUB_TOKEN']
        },
        'prod': {
            'url': os.environ['PROD_DATAHUB_URL'],
            'token': os.environ['PROD_DATAHUB_TOKEN']
        }
    }

    env_config = config[environment]
    emitter = DatahubRestEmitter(
        gms_server=env_config['url'],
        token=env_config['token']
    )

    for term in glossary['terms']:
        term_urn = make_glossary_term_urn(term['name'])

        term_info = GlossaryTermInfoClass(
            definition=term['description'],
            termSource=term.get('source', 'Internal'),
            sourceRef=term.get('source_ref', '')
        )

        mcp = MetadataChangeProposalWrapper(
            entityUrn=term_urn,
            aspect=term_info
        )

        emitter.emit_mcp(mcp)
        print(f"✓ Synced term '{term['name']}' to {environment}")

if __name__ == "__main__":
    environment = sys.argv[2] if len(sys.argv) > 2 else 'dev'

    glossary = load_glossary('metadata/glossaries/terms.yaml')
    sync_glossary_to_environment(glossary, environment)

    print(f"✓ Glossary sync to {environment} complete")

Metadata Definition Files:

# metadata/glossaries/terms.yaml
terms:
  - name: "Customer"
    description: "An individual or organization that purchases products or services"
    source: "Business Glossary"
    owners:
      - "urn:li:corpGroup:business-analysts"
    related_terms:
      - "Account"
      - "User"

  - name: "Revenue"
    description: "Total income generated from business operations"
    source: "Finance Team"
    owners:
      - "urn:li:corpGroup:finance"

---

# metadata/domains/domains.yaml
domains:
  - name: "Finance"
    description: "Financial data and analytics"
    owners:
      - "urn:li:corpGroup:finance-team"

  - name: "Marketing"
    description: "Marketing and customer acquisition data"
    owners:
      - "urn:li:corpGroup:marketing-team"

---

# metadata/tags/tags.yaml
tags:
  - name: "PII"
    description: "Contains Personally Identifiable Information"
    color: "#FF0000"

  - name: "Deprecated"
    description: "Asset scheduled for deprecation"
    color: "#FFA500"

Strategy 2: Environment-Specific Configuration:

# environments/dev/config.yaml
platform_instances:
  snowflake: "dev.us-east-1.aws"
  bigquery: "dev-project"

dataset_patterns:
  allow:
    - "dev_*"
    - "*_dev"

---

# environments/prod/config.yaml
platform_instances:
  snowflake: "prod.us-east-1.aws"
  bigquery: "prod-project"

dataset_patterns:
  allow:
    - "prod_*"
    - "*_prod"

Strategy 3: Single Instance with Platform Instances

Alternative: Use one DataHub instance with platform instances to separate environments:

# Ingestion for dev data
source:
  type: snowflake
  config:
    account_id: ${SNOWFLAKE_ACCOUNT}
    platform_instance: "dev.snowflake"
    database_pattern:
      allow:
        - "DEV_*"

---

# Ingestion for prod data
source:
  type: snowflake
  config:
    account_id: ${SNOWFLAKE_ACCOUNT}
    platform_instance: "prod.snowflake"
    database_pattern:
      allow:
        - "PROD_*"

Access Control by Environment:

# policies/environment_access.yaml
policies:
  - name: "Dev Environment Access"
    resources:
      platformInstances:
        - "dev.snowflake"
        - "dev.bigquery"
    actors:
      groups:
        - "developers"
        - "data-engineers"
    privileges:
      - VIEW_ENTITY_PAGE
      - EDIT_ENTITY

  - name: "Prod Environment Access"
    resources:
      platformInstances:
        - "prod.snowflake"
        - "prod.bigquery"
    actors:
      groups:
        - "senior-engineers"
    privileges:
      - VIEW_ENTITY_PAGE
      # No EDIT_ENTITY for production

Promotion Workflow:

# Promote metadata from dev to prod
# 1. Export from dev
datahub get --urn "urn:li:glossaryTerm:NewTerm" > new_term.json

# 2. Review and approve
# (Manual review or automated checks)

# 3. Import to staging
datahub put --urn "urn:li:glossaryTerm:NewTerm" < new_term.json \
  --server https://staging.datahub.company.com

# 4. After validation, import to prod
datahub put --urn "urn:li:glossaryTerm:NewTerm" < new_term.json \
  --server https://prod.datahub.company.com

Best Practices:

1. Use Infrastructure as Code for all shared metadata
2. Version Control all metadata definitions
3. Automated Testing validate metadata before promotion
4. Approval Workflows for production changes
5. Environment Parity keep structures similar across environments
6. Clear Ownership designate metadata owners
7. Documentation document environment strategy

• Separate instances provide better isolation but increase management overhead
• Platform instances offer lighter-weight environment separation
• Consider cost of running multiple DataHub instances
• Test metadata sync thoroughly in lower environments first
• Monitor sync jobs for failures

• Glossary Terms
• Platform Instances

• 4928, 5358