How to Export Ingestion Recipes from the DataHub UI to YAML Files – DataHub Customer Support Portal

Area: Ingestion Issues

Sub-Area: Recipe Management & CI/CD Integration

Issue

When ingestion sources are created and managed through the DataHub UI, the underlying recipes are stored in the DataHub backend rather than as files on disk. Teams that want to version-control their recipes in a source code repository and deploy them through a CI/CD pipeline have no built-in "export all" button in the UI. This article explains how to programmatically export all UI-created ingestion recipes to YAML files and how to structure those files for clean, repeatable redeployment using the DataHub CLI.

You Might Be Asking

Is there a way to download all my DataHub ingestion recipes as YAML files?
How do I get my UI-created ingestion sources into source control?
How do I redeploy exported recipes with datahub ingest deploy without passing every argument on the command line?
What is the deployment: block in a DataHub recipe file and how does it work?

Solution

Step 1 — Query All Ingestion Sources via GraphQL

The listIngestionSources GraphQL query returns every ingestion source, including the full recipe stored under config.recipe as a JSON string. Run the following query against your DataHub instance:

query {
  listIngestionSources(input: { start: 0, count: 100, query: "*" }) {
    total
    ingestionSources {
      urn
      name
      schedule {
        interval
        timezone
      }
      config {
        executorId
        recipe
      }
    }
  }
}

The response will include both your user-created sources and DataHub's internal system sources (e.g., garbage collection, usage reporting). Filter out system sources before exporting — they are typically identifiable by their name or URN prefix.

Step 2 — Export Recipes to YAML Using a Python Script

The following Python script authenticates with your DataHub instance, calls the GraphQL API, and writes one YAML file per user-created ingestion source. It also writes a manifest file recording each source's URN and schedule for use during redeployment.

import json
import os
import yaml
import requests

DATAHUB_URL = "https://<your-instance>.datahubproject.io"
PERSONAL_ACCESS_TOKEN = "<your-personal-access-token>"

# System-managed source URN prefix to exclude
SYSTEM_URN_PREFIX = "urn:li:dataHubIngestionSource:system"

GRAPHQL_QUERY = """
query {
  listIngestionSources(input: { start: 0, count: 100, query: "*" }) {
    total
    ingestionSources {
      urn
      name
      schedule {
        interval
        timezone
      }
      config {
        executorId
        recipe
      }
    }
  }
}
"""

headers = {
    "Authorization": f"Bearer {PERSONAL_ACCESS_TOKEN}",
    "Content-Type": "application/json",
}

response = requests.post(
    f"{DATAHUB_URL}/api/graphql",
    headers=headers,
    json={"query": GRAPHQL_QUERY},
)
response.raise_for_status()
data = response.json()

sources = data["data"]["listIngestionSources"]["ingestionSources"]
os.makedirs("recipes", exist_ok=True)
manifest = []

for source in sources:
    # Skip internal system sources
    if source["urn"].startswith(SYSTEM_URN_PREFIX):
        continue

    recipe = json.loads(source["config"]["recipe"])
    schedule = source.get("schedule") or {}

    # Add deployment block so datahub ingest deploy reads config from file
    output = {
        "deployment": {
            "name": source["name"],
            "schedule": schedule.get("interval", ""),
            "time_zone": schedule.get("timezone", "UTC"),
            "executor_id": source["config"].get("executorId", "default"),
        }
    }
    output.update(recipe)

    safe_name = source["name"].replace(" ", "_").replace("/", "-")
    filename = f"recipes/{safe_name}.yml"

    with open(filename, "w") as f:
        yaml.dump(output, f, default_flow_style=False, sort_keys=False)

    manifest.append({
        "name": source["name"],
        "urn": source["urn"],
        "file": filename,
    })
    print(f"Exported: {filename}")

with open("recipes/manifest.json", "w") as f:
    json.dump(manifest, f, indent=2)

print(f"\nDone. Exported {len(manifest)} sources. Manifest written to recipes/manifest.json")

Step 3 — Understand the `deployment:` Block

Adding a deployment: block at the top of a recipe YAML file allows datahub ingest deploy to read the source name, schedule, timezone, and executor from the file itself, eliminating the need to pass those values as CLI flags. The block supports the following keys only: name, schedule, time_zone, cli_version, and executor_id. Any other keys in this block will be rejected. The deployment: block is stripped out before the recipe is stored in the backend, so it has no effect on actual ingestion behavior.

deployment:
  name: "Snowflake - PROD"
  schedule: "0 2 * * *"
  time_zone: "America/New_York"
  executor_id: "<your-executor-id>"

source:
  type: snowflake
  config:
    # ... your source config ...

Important: A recipe file containing a deployment: block is intended for use with datahub ingest deploy. Running it locally with datahub ingest run will fail because that command does not recognize the extra block. Keep separate files or remove the block for local testing.

Step 4 — Redeploy Recipes from Source Control

Once your YAML files are in a git repository, you have two options for redeployment in your CI/CD pipeline:

Option A — Update existing UI-created sources in place (preserves the original URN):

# Pass the URN recorded in the manifest to update the existing source
datahub ingest deploy --urn <existing-urn-from-manifest> -c recipes/snowflake_prod.yml

Use this option when you want to continue managing the same sources that were originally built in the UI. The URN is recorded in recipes/manifest.json by the export script.

Option B — Let the CLI derive a stable URN from the deployment name (no URN flag required):

# The CLI derives a deterministic URN from the name in the deployment block
datahub ingest deploy -c recipes/snowflake_prod.yml

Use this option for a fully file-driven GitOps workflow where no per-source URN management is needed. Running the same command repeatedly will always update the same source because the URN is derived deterministically from the name. Important trade-off: This creates a new source rather than reusing the original UI-created one. Perform a one-time cutover: deploy all files, confirm the new sources run successfully, then delete or disable the original UI-created sources to avoid duplicate ingestion of the same platform.

Step 5 — Authentication Requirements

The personal access token used for both the export script and the datahub ingest deploy step must have the Manage Metadata Ingestion privilege. A token with lesser permissions will succeed on the read (export) but fail on the write (deploy).

Step 6 — Secret References

Recipes store credentials as secret references in the form ${SECRET_NAME} rather than the actual secret values. Before committing exported files to a repository, scan each file to confirm no plaintext credentials are present. If a recipe was created with a credential entered directly (rather than via a named DataHub secret), the raw value will appear in the exported YAML and must be replaced with a reference or removed before committing.

Additional Notes

The listIngestionSources query uses pagination. If your instance has more than 100 sources, increment the start offset in a loop until total is reached.
There is currently no built-in UI button or CLI command to bulk-export all recipes in one step. The GraphQL-based approach described here is the supported workaround.
The deployment: block feature is available in DataHub CLI (acryl-datahub) version 0.12.x and later. Check your installed version with datahub version before using it.
Internal system ingestion sources (e.g., DataHub's own garbage collection and usage reporting jobs) will appear in the listIngestionSources response. These should be excluded from export and should not be redeployed via the CLI, as they are managed by the platform.