Skip to main content

DataHub Actions Registry Import Error on Deployment

Angelo Bresci
Updated

Area: Deployment Issues

Sub-Area: Container Deployment

Issue

DataHub Actions fails to start in containerized environments due to a threading race condition in the plugin registry initialization, resulting in "KeyError: key already in use" errors during startup. This commonly occurs in Kubernetes deployments and restricted environments like air-gapped systems where multiple action workers start concurrently.

Error Messages

  • [timestamp] ERROR {datahub_actions.entrypoints:122} - File "/home/datahub/.venv/lib64/python3.11/site-packages/datahub/ingestion/api/registry.py", line 124, in _ensure_not_lazy
  • KeyError: "key already in use"

You Might Be Asking

  • Why does DataHub Actions crash immediately after startup in my container deployment?
  • How do I resolve registry import failures in restricted environments?
  • What causes the plugin registry race condition?

Solution

This issue is caused by a threading race condition in the plugin registry that was fixed after the 1.5.0-oss release. Here are the resolution options:

Option 1: Upgrade to Fixed Version (Recommended)

  1. Upgrade to a DataHub Actions build that contains the threading fix (any build after April 2026)
  2. The fix adds proper thread locking to prevent concurrent registry access
  3. For air-gapped or restricted environments, work with your security team to approve the updated image

Option 2: Workaround for Immediate Relief

  1. Modify your deployment configuration to use a single worker thread
  2. Set the actions process configuration:
spec:
  containers:
  - name: datahub-actions
    env:
    - name: DATAHUB_ACTIONS_WORKERS
      value: "1"
  1. Or adjust any --num-workers parameter in your startup command to 1
  2. This prevents concurrent registry access that triggers the race condition

Option 3: Base Image Update (for Red Hat/UBI environments)

  1. If using Red Hat UBI base images, ensure you're using UBI 9.8 or later
  2. UBI 9.7 with install_weak_deps=0 has sqlite library compatibility issues
  3. Update your Dockerfile or container registry to use the latest UBI version

Additional Notes

This is a confirmed product regression in the 1.5.0.x release line that affects multi-threaded deployments. The race condition occurs when multiple threads attempt to register the same plugin keys (executor, slack, teams, snowflake_tag_propagation) simultaneously during startup. Single-threaded deployments are unaffected. For production environments, upgrading to the fixed version is strongly recommended over the single-worker workaround, as it maintains performance while resolving the underlying issue.

Related Documentation

Tags: actions, deployment, kubernetes, threading, race-condition, registry, container, startup, air-gapped, restricted-environment