DataHub Performance Optimization

DataHub is experiencing slow response times, high latency, or performance degradation under load. This can affect UI responsiveness, API performance, and ingestion throughput.

You Might Be Asking:

• How do I optimize DataHub performance?
• Why is the UI slow?
• How do I scale DataHub for large catalogs?

If you are using DataHub Cloud, DataHub's Support team can assist you with making any of the following optimizations. If you have DataHub installed on-premises, you should be able to apply any of the following optimizations.

1. Optimize Elasticsearch for performance:

# values.yaml for Helm chart
elasticsearch:
  replicas: 3  # Scale horizontally

  resources:
    requests:
      memory: "8Gi"
      cpu: "2"
    limits:
      memory: "16Gi"
      cpu: "4"

  # Heap size (50% of container memory)
  env:
    - name: ES_JAVA_OPTS
      value: "-Xms8g -Xmx8g"

  # SSD storage for better I/O
  volumeClaimTemplate:
    storageClassName: "ssd"
    resources:
      requests:
        storage: "100Gi"

  # Elasticsearch settings
  esConfig:
    elasticsearch.yml: |
      indices.memory.index_buffer_size: 30%
      thread_pool.write.queue_size: 1000
      thread_pool.search.queue_size: 1000

2. Optimize GMS (backend) performance:

datahub-gms:
  replicas: 3  # Scale horizontally

  resources:
    requests:
      memory: "4Gi"
      cpu: "2"
    limits:
      memory: "8Gi"
      cpu: "4"

  # Java heap settings
  env:
    - name: JAVA_OPTS
      value: "-Xms4g -Xmx6g -XX:+UseG1GC"

    # Connection pool sizing
    - name: SPRING_DATASOURCE_HIKARI_MAXIMUM_POOL_SIZE
      value: "50"

    # Cache settings
    - name: CACHE_ENTITY_CACHE_SIZE
      value: "10000"

3. Optimize database performance:

Examples for MySQL and PostgreSQL. These can be adapted for other relational databases used as DataHub's metadata store.

-- For MySQL
-- Optimize InnoDB settings
SET GLOBAL innodb_buffer_pool_size = 8589934592;  -- 8GB
SET GLOBAL innodb_log_file_size = 536870912;      -- 512MB
SET GLOBAL max_connections = 500;

-- Add indexes for common queries
CREATE INDEX idx_urn ON metadata_aspect_v2(urn);
CREATE INDEX idx_aspect ON metadata_aspect_v2(aspect);

-- For PostgreSQL
-- Tune configuration
shared_buffers = 2GB
effective_cache_size = 6GB
maintenance_work_mem = 512MB
checkpoint_completion_target = 0.9
wal_buffers = 16MB
default_statistics_target = 100
random_page_cost = 1.1

4. Enable caching:

datahub-gms:
  env:
    # Entity cache
    - name: ENTITY_SERVICE_ENABLE_CACHE
      value: "true"
    - name: ENTITY_SERVICE_CACHE_SIZE
      value: "10000"

    # Relationship cache
    - name: RELATIONSHIP_SERVICE_ENABLE_CACHE
      value: "true"

    # Search result cache
    - name: SEARCH_SERVICE_ENABLE_CACHE
      value: "true"
    - name: SEARCH_SERVICE_CACHE_TTL_SECONDS
      value: "600"

5. Optimize frontend performance:

datahub-frontend:
  replicas: 2

  resources:
    requests:
      memory: "2Gi"
      cpu: "1"
    limits:
      memory: "4Gi"
      cpu: "2"

  env:
    # Session cache
    - name: PLAY_HTTP_SESSION_CACHE_ENABLED
      value: "true"

    # Connection timeout
    - name: DATAHUB_GMS_TIMEOUT_SECONDS
      value: "60"

6. Monitor and tune performance:

Examples for MySQL. These can be adapted for PostgreSQL or other databases.

# Monitor resource usage
kubectl top pods -n datahub

# Check response times
kubectl logs deployment/datahub-gms | grep "duration"

# Monitor Elasticsearch performance
kubectl exec elasticsearch-0 -- curl localhost:9200/_nodes/stats | jq

# Database query analysis
# MySQL
SHOW FULL PROCESSLIST;
SHOW ENGINE INNODB STATUS;

# Check slow queries
SET GLOBAL slow_query_log = 'ON';

Performance tuning depends on workload characteristics (catalog size, query patterns, ingestion frequency). Monitor metrics to identify bottlenecks. Consider using CDN for static assets in frontend. Use connection pooling appropriately.

• Performance Tuning
• Elasticsearch Tuning