DataHub Upgrade and Migration Issues

Upgrading DataHub to a new version causes issues with existing metadata, incompatibilities, or deployment failures. Understanding the upgrade process and potential breaking changes is critical for smooth migrations.

Common Error Messages:

• Schema version mismatch
• Migration script failed
• Incompatible aspect version
• Database migration error

You Might Be Asking:

• How do I upgrade DataHub safely?
• Will I lose metadata during upgrade?
• How do I rollback if upgrade fails?

1. Pre-upgrade checklist:

# 1. Backup your data
# MySQL/PostgreSQL
mysqldump -u root -p datahub > datahub_backup_$(date +%Y%m%d).sql

# Elasticsearch snapshots
kubectl exec elasticsearch-0 -- curl -X PUT "localhost:9200/_snapshot/backup"

# 2. Review release notes
# Check https://github.com/datahub-project/datahub/releases

# 3. Test in non-production first
# Clone production to staging
# Perform upgrade on staging
# Validate functionality

# 4. Check compatibility
# Verify plugin versions
# Check custom code compatibility

2. Upgrade Kubernetes deployment:

# Update Helm chart version
helm repo update

# Check available versions
helm search repo datahub/datahub

# Upgrade with new version
helm upgrade datahub datahub/datahub \
  --version 0.12.0 \
  --values values.yaml \
  --namespace datahub

# Monitor rollout
kubectl rollout status deployment/datahub-gms -n datahub
kubectl rollout status deployment/datahub-frontend -n datahub

# Check logs for errors
kubectl logs -f deployment/datahub-gms -n datahub

3. Run database migrations:

# Migrations usually run automatically, but can be run manually

# Check current schema version
kubectl exec deployment/datahub-gms -- \
  curl localhost:8080/health | jq '.schemaVersion'

# Force migration (if needed)
kubectl exec deployment/datahub-gms -- \
  /datahub/datahub-gms/bin/datahub-upgrade \
  -u SystemUpdate

# Verify migration success
kubectl logs deployment/datahub-gms | grep -i migration

4. Handle breaking changes:

# Example: v0.11 to v0.12 upgrade

# Breaking changes might include:
# - API endpoint changes
# - Configuration format changes
# - Aspect model changes

# Update configurations
# Old format (v0.11):
ingestion:
  enabled: true

# New format (v0.12):
datahubUpgrade:
  enabled: true
  cleanupPolicy: "KEEP_ALL"

# Update ingestion recipes for new format
# Check recipe compatibility
datahub check plugins

5. Rollback procedure:

# Rollback Helm deployment
helm rollback datahub -n datahub

# Or specific revision
helm rollback datahub 5 -n datahub

# Restore database backup
mysql -u root -p datahub < datahub_backup_20250106.sql

# Restore Elasticsearch snapshot
kubectl exec elasticsearch-0 -- \
  curl -X POST "localhost:9200/_snapshot/backup/snapshot_1/_restore"

# Verify rollback
kubectl get pods -n datahub
helm list -n datahub

6. Post-upgrade validation:

# 1. Check service health
curl http://localhost:8080/health

# 2. Verify search functionality
# Test search in UI

# 3. Test ingestion
datahub ingest -c test-recipe.yml --dry-run

# 4. Verify API access
curl -X POST http://localhost:8080/api/graphql \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"query": "{ me { username } }"}'

# 5. Check metadata integrity
# Spot check key datasets in UI
# Verify lineage displays correctly
# Test assertions and monitoring

Always backup before upgrading. Test upgrades in non-production environments first. Read release notes carefully for breaking changes. Plan maintenance windows for production upgrades. Keep Helm chart values and custom configurations in version control.

• Upgrade Guide
• Release Notes