Metadata Import/Export and Backup

Organizations need to backup DataHub metadata, export for analysis, migrate between environments, or restore from backups. Understanding export/import mechanisms is critical for disaster recovery.

You Might Be Asking:

• How do I backup DataHub metadata?
• Can I export metadata to files?
• How do I migrate metadata between environments?

1. Export metadata using CLI:

# Export all entities of a type
datahub get --urn "urn:li:dataset:(urn:li:dataPlatform:snowflake,*,PROD)" \
  --aspect all \
  --to-file datasets_export.json

# Export specific entity
datahub get --urn "urn:li:dataset:(urn:li:dataPlatform:snowflake,db.schema.table,PROD)" \
  --to-file single_dataset.json

# Export all aspects for an entity
datahub get --urn "YOUR_URN" --aspect all \
  --to-file entity_full_export.json

2. Bulk export via GraphQL:

import requests
import json

def export_all_datasets(output_file):
    """
    Export all datasets with metadata
    """
    all_datasets = []
    start = 0
    count = 100

    while True:
        query = """
        query searchDatasets($start: Int!, $count: Int!) {
          search(input: {
            type: DATASET
            query: "*"
            start: $start
            count: $count
          }) {
            total
            searchResults {
              entity {
                urn
                ... on Dataset {
                  name
                  description
                  platform { name }
                  properties {
                    customProperties { key value }
                  }
                  ownership {
                    owners {
                      owner { urn }
                      type
                    }
                  }
                  tags {
                    tags {
                      tag { urn name }
                    }
                  }
                  glossaryTerms {
                    terms {
                      term { urn name }
                    }
                  }
                }
              }
            }
          }
        }
        """

        result = graphql_query(query, {"start": start, "count": count})
        results = result['data']['search']['searchResults']

        if not results:
            break

        all_datasets.extend([r['entity'] for r in results])
        start += count

        print(f"Exported {len(all_datasets)} datasets...")

    # Save to file
    with open(output_file, 'w') as f:
        json.dump(all_datasets, f, indent=2)

    print(f"Export complete: {len(all_datasets)} datasets saved to {output_file}")

# Usage
export_all_datasets("datahub_datasets_backup.json")

3. Import metadata from file:

# Import using CLI
datahub put --urn "urn:li:dataset:..." \
  --aspect datasetProperties \
  -d @dataset_properties.json

# Bulk import
datahub ingest -c import_recipe.yml

# import_recipe.yml
source:
  type: file
  config:
    filename: "./datahub_export.json"

sink:
  type: datahub-rest
  config:
    server: "http://localhost:8080"
    token: "${DATAHUB_TOKEN}"

4. Database backup:

Examples for MySQL and PostgreSQL. These can be adapted for other relational databases used as DataHub's metadata store.

# Backup MySQL database
mysqldump -u root -p datahub > datahub_backup_$(date +%Y%m%d).sql

# Backup specific tables
mysqldump -u root -p datahub \
  metadata_aspect_v2 \
  metadata_index \
  > datahub_metadata_backup.sql

# For PostgreSQL
pg_dump -U datahub -d datahub \
  -f datahub_backup_$(date +%Y%m%d).sql

# Restore MySQL
mysql -u root -p datahub < datahub_backup_20250115.sql

# Restore PostgreSQL
psql -U datahub -d datahub \
  -f datahub_backup_20250115.sql

5. Elasticsearch backup:

# Create snapshot repository
kubectl exec elasticsearch-0 -- curl -X PUT "localhost:9200/_snapshot/backup_repo" \
  -H 'Content-Type: application/json' \
  -d '{
    "type": "fs",
    "settings": {
      "location": "/usr/share/elasticsearch/backup"
    }
  }'

# Create snapshot
kubectl exec elasticsearch-0 -- curl -X PUT "localhost:9200/_snapshot/backup_repo/snapshot_$(date +%Y%m%d)" \
  -H 'Content-Type: application/json' \
  -d '{
    "indices": "datahub*",
    "include_global_state": false
  }'

# List snapshots
kubectl exec elasticsearch-0 -- curl "localhost:9200/_snapshot/backup_repo/_all?pretty"

# Restore snapshot
kubectl exec elasticsearch-0 -- curl -X POST "localhost:9200/_snapshot/backup_repo/snapshot_20250115/_restore" \
  -H 'Content-Type: application/json' \
  -d '{
    "indices": "datahub*"
  }'

6. Automated backup script:

#!/bin/bash
# datahub_backup.sh

BACKUP_DIR="/backups/datahub"
DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_PATH="${BACKUP_DIR}/${DATE}"

mkdir -p "${BACKUP_PATH}"

echo "Starting DataHub backup: ${DATE}"

# 1. Backup MySQL
echo "Backing up MySQL database..."
kubectl exec deployment/mysql -- mysqldump -u root -p${MYSQL_ROOT_PASSWORD} datahub \
  > "${BACKUP_PATH}/mysql_backup.sql"

# 2. Backup Elasticsearch
echo "Creating Elasticsearch snapshot..."
kubectl exec elasticsearch-0 -- curl -X PUT "localhost:9200/_snapshot/backup_repo/snapshot_${DATE}"

# 3. Export metadata via API
echo "Exporting metadata..."
python3 export_metadata.py --output "${BACKUP_PATH}/metadata_export.json"

# 4. Backup configuration
echo "Backing up configuration..."
kubectl get configmap -n datahub -o yaml > "${BACKUP_PATH}/configmaps.yaml"
kubectl get secret -n datahub -o yaml > "${BACKUP_PATH}/secrets.yaml"

# 5. Compress backup
echo "Compressing backup..."
tar -czf "${BACKUP_DIR}/datahub_backup_${DATE}.tar.gz" -C "${BACKUP_PATH}" .
rm -rf "${BACKUP_PATH}"

# 6. Cleanup old backups (keep last 30 days)
find "${BACKUP_DIR}" -name "datahub_backup_*.tar.gz" -mtime +30 -delete

echo "Backup complete: ${BACKUP_DIR}/datahub_backup_${DATE}.tar.gz"

7. Cross-environment migration:

# Migrate metadata from staging to production
def migrate_metadata(source_url, source_token, target_url, target_token):
    """
    Migrate metadata between DataHub instances
    """
    # Export from source
    source_client = DatahubRestEmitter(source_url, token=source_token)

    # Get all entities
    entities = export_all_entities(source_client)

    # Import to target
    target_client = DatahubRestEmitter(target_url, token=target_token)

    for entity in entities:
        # Transform URNs if needed (e.g., change platform instance)
        transformed_entity = transform_urns(entity, 
            old_instance="staging", 
            new_instance="prod"
        )

        # Emit to target
        target_client.emit(transformed_entity)

    print(f"Migrated {len(entities)} entities")

# Usage
migrate_metadata(
    source_url="https://staging-datahub.company.com",
    source_token="staging-token",
    target_url="https://prod-datahub.company.com",
    target_token="prod-token"
)

Regular backups are essential for disaster recovery. Test restore procedures regularly. Consider backing up to S3 or other cloud storage. Automate backups with cron jobs. Document restore procedures. Keep multiple backup versions.

• CLI Guide
• Backup & Restore