Docker Compose Deployment Issues

DataHub docker-compose deployment is failing to start, services are not connecting to each other, or the deployment is unstable. This commonly occurs due to resource constraints, port conflicts, or misconfigured environment variables.

Common Error Messages:

• Container exited with code 137 (OOM killed)
• Connection refused to elasticsearch
• Error response from daemon: Ports are not available
• Failed to connect to MySQL

You Might Be Asking:

• How much resources does DataHub need for Docker?
• Why won't DataHub start with docker-compose?
• How do I troubleshoot Docker deployment issues?

1. Ensure sufficient Docker resources:

# Check Docker resource allocation
# Docker Desktop: Settings → Resources
# Recommended minimums:
# - Memory: 8GB
# - CPUs: 4
# - Disk: 20GB

# On Linux, check available resources
free -h
df -h

2. Start DataHub with quickstart:

# Clone the repository
git clone https://github.com/datahub-project/datahub.git
cd datahub

# Run quickstart
./docker/quickstart.sh

# Or with specific version
DATAHUB_VERSION=v0.12.0 ./docker/quickstart.sh

# Check all services are running
docker-compose -f docker-compose.yml -f docker-compose.override.yml ps

3. Debug service connectivity:

# Check logs for specific service
docker-compose logs -f datahub-gms
docker-compose logs -f elasticsearch
docker-compose logs -f mysql

# Test connectivity between services
docker-compose exec datahub-gms nc -zv elasticsearch 9200
docker-compose exec datahub-gms nc -zv mysql 3306
docker-compose exec datahub-gms nc -zv kafka 9092

4. Fix common port conflicts:

# In docker-compose.override.yml
services:
  datahub-frontend-react:
    ports:
      - "9002:9002"  # Change if port 9002 is in use

  elasticsearch:
    ports:
      - "9200:9200"  # Change if port 9200 is in use

  mysql:
    ports:
      - "3306:3306"  # Change if port 3306 is in use

5. Increase service memory limits:

# In docker-compose.override.yml
services:
  datahub-gms:
    environment:
      - JAVA_OPTS=-Xms2g -Xmx4g
    deploy:
      resources:
        limits:
          memory: 4g

  elasticsearch:
    environment:
      - "ES_JAVA_OPTS=-Xms2g -Xmx2g"
    deploy:
      resources:
        limits:
          memory: 4g

6. Reset and clean restart:

# Stop all services
docker-compose down

# Remove volumes (WARNING: deletes all data)
docker-compose down -v

# Clean up
docker system prune -a

# Fresh start
./docker/quickstart.sh

Docker Compose is recommended for development and testing only. For production deployments, use Kubernetes. Monitor resource usage with docker stats to identify bottlenecks.

• Docker Deployment
• Quickstart Guide