Remote Executor on Fargate: Insufficient Disk Space Causing Ingestion Failures – DataHub Customer Support Portal

Area: Deployment Issues

Sub-Area: Remote Ingestion Executor / AWS Fargate Ephemeral Storage

Issue

When running the DataHub Remote Ingestion Executor on AWS Fargate, ingestion jobs can fail due to insufficient ephemeral disk space. The executor base image has grown over recent releases and may consume nearly all of the default Fargate ephemeral storage allocation at container startup — before any ingestion work begins. This leaves no headroom for job execution, causing all ingestion sources (including Snowflake) to fail immediately. Disabling lineage, profiling, or reducing time-window granularity does not resolve the problem because the disk is exhausted by the image itself, not by job data. Earlier versions of the remote-ingestion-executor Terraform module did not expose a variable to configure the ECS ephemeral_storage property, preventing users from increasing the allocation without modifying the module directly.

Error Messages

[Errno 28] No space left on device
'disk_info': {'total': '22.01 GB', 'used': '21.06 GB', 'used_initally': '20.76 GB', 'free': '0 bytes'}
Tasks using the EC2 launch type do not support EphemeralStorage (secondary error — see notes below)

You Might Be Asking

Why does my Snowflake ingestion fail with a disk-full error even when I disable profiling and lineage?
How do I increase the ephemeral storage for my Fargate-based Remote Ingestion Executor?
Why do I get Tasks using the EC2 launch type do not support EphemeralStorage when I try to set ephemeral_storage in the Terraform module?
Can I use the slim executor image to reduce disk usage?

Solution

Upgrade the datahub-terraform-modules remote-ingestion-executor module to a version that includes the ephemeral_storage variable and the requires_compatibilities = ["FARGATE"] fix. The required changes were merged into the main branch of https://github.com/acryldata/datahub-terraform-modules. Pin your module source to a released tag that includes both fixes — confirm the minimum version with DataHub Support if you are unsure which tag to use.

Add the ephemeral_storage argument to your module call and set the value to at least 50 GiB. AWS Fargate supports up to 200 GiB of ephemeral storage. Size the value based on the scale of your catalog and the executor image size at your deployed version.

module "datahub-remote-executor" {
  source = "git::https://github.com/acryldata/datahub-terraform-modules.git//remote-ingestion-executor?ref=<latest-stable-tag>"

  cluster_name               = "<your-cluster-name>"
  create_tasks_iam_role      = true
  create_task_exec_iam_role  = true

  task_exec_secret_arns = [
    "<your-secret-arn>"
  ]

  datahub = <your-datahub-config>

  secrets = [
    {
      name      = "DATAHUB_GMS_TOKEN"
      valueFrom = "<your-secret-arn>"
    }
  ]

  desired_count = <desired-count>
  cpu           = <task-cpu>
  memory        = <task-memory>

  # Increase ephemeral storage to avoid disk exhaustion at container startup.
  # AWS Fargate minimum is 21 GiB; maximum is 200 GiB.
  # 50 GiB is a recommended starting point for most workloads.
  ephemeral_storage = {
    size_in_gib = 50
  }

  subnet_ids = <your-subnet-ids>

  security_group_rules = {
    egress_all = {
      type        = "egress"
      from_port   = 0
      to_port     = 0
      protocol    = "-1"
      cidr_blocks = ["0.0.0.0/0"]
    }
  }
}

Run terraform apply to update your ECS task definition. ECS will create a new task definition revision and restart the executor container with the increased storage allocation.
```
terraform apply
```
(Optional interim workaround) Use the slim executor image variant if you need relief before the module upgrade is possible. The slim image has a smaller on-disk footprint and may fit within the default 21 GiB allocation. Consult DataHub Support for the correct slim image tag for your deployment version.
Verify the fix by re-running the failing ingestion source after the executor task restarts. Check the executor logs for the disk_info line to confirm that free space is now available at startup.
```
# Expected output after fix (values will vary by configuration):
# 'disk_info': {'total': '53.66 GB', 'used': '20.80 GB', 'free': '32.86 GB'}
```

Additional Notes

Root cause: The executor base image grew significantly between releases (approximately doubling from ~10 GB to ~20 GB in a span of several months). At the default Fargate ephemeral storage allocation (AWS enforces a minimum of 21 GiB), the image alone consumes virtually all available disk before any ingestion task runs. Disabling lineage, profiling, or reducing time-window granularity has no effect on this problem because disk exhaustion occurs at startup, not during data processing.

Secondary Terraform error (Tasks using the EC2 launch type do not support EphemeralStorage): Older versions of the module set requires_compatibilities = ["EC2", "FARGATE"] by default. AWS does not permit the ephemeral_storage property when EC2 is listed as a compatible launch type, even if the service itself only runs on Fargate. The fix in the updated module changes this default to requires_compatibilities = ["FARGATE"]. If you encounter this error on an older module version, ensure you are pinned to a version that includes this correction.

Cost consideration: AWS Fargate bills for ephemeral storage above the 20 GiB baseline continuously for the lifetime of the task. Setting a larger default increases storage costs for all deployments. Choose the smallest value that provides sufficient headroom for your workload; 50 GiB is a reasonable starting point for most catalog sizes.

CloudFormation users: If you deploy the executor via AWS CloudFormation rather than Terraform, increase the TaskEphemeralStorageSizeInGiB parameter on your stack update (found under "Worker Task Parameters"). The AWS minimum is 21 GiB; set it to 50 or higher as needed.

Image size: If the executor image continues to grow in future releases and re-approaches the configured ephemeral storage ceiling, open a support ticket so the DataHub team can investigate image layer optimizations or provide a slim image variant.

Issue

Error Messages

You Might Be Asking

Solution

Additional Notes

Related Documentation