Configuring dbt Artifact Ingestion from Cloud Storage (GCS, S3) into DataHub – DataHub Customer Support Portal

Area: Ingestion Issues

Sub-Area: dbt Source Configuration & Cloud Storage Integration

Issue

When setting up dbt ingestion into DataHub, users need to understand which artifact files are required, where to store them, and how to configure the ingestion recipe to read from cloud storage locations such as Amazon S3 or Google Cloud Storage (GCS). Additionally, teams running dbt on-premises or in CI/CD pipelines want to automate the upload of dbt artifacts to cloud storage so that DataHub always ingests the latest metadata without manual intervention.

You Might Be Asking

Which dbt artifact files does DataHub require, and which are optional?
Where do I put manifest.json and catalog.json so the dbt source can read them?
How do I configure the ingestion recipe to read dbt artifacts from S3 or GCS?
How do I automate uploading dbt artifacts to cloud storage after each dbt run?
Do I need to push files directly into DataHub, or just point the recipe at a storage location?

Solution

Step 1: Understand the Required and Optional Artifact Files

DataHub's dbt source reads the following artifact files produced by dbt Core. Note there is no metadata.json — the correct files are:

manifest.json — Required. Generated by dbt run, dbt compile, or dbt docs generate. Contains models, lineage, tests, and descriptions.
catalog.json — Highly recommended. Generated by dbt docs generate. Adds column schemas and table statistics. Without it, column-level metadata will not appear in DataHub.
sources.json — Optional. Generated by dbt source freshness. Adds source freshness data.
run_results.json — Optional. Adds dbt test results. Important caveat: dbt docs generate overwrites run_results.json, so copy it aside before running docs generation and restore it afterward. See the recommended sequence in the dbt test integration docs.

Step 2: Choose a Storage Location

You do not upload artifact files into DataHub itself. Instead, you store them in an accessible location and configure the recipe to point at that location. DataHub reads from it on every scheduled run. Supported locations are:

Local path — Works only when the ingestion process runs on the same filesystem as the files. Not suitable for automated or scheduled cloud ingestion.
Amazon S3 — Recommended for AWS-based environments. Use s3:// URIs.
Google Cloud Storage — Recommended for GCP-based environments. Use gs:// URIs.

For automation, upload artifacts to a fixed, stable prefix (e.g., s3://<your-bucket>/dbt/target/), overwriting the files in place on each run rather than writing to timestamped folders. Timestamped paths cause DataHub to reprocess old artifacts on every cycle.

Step 3: Configure the Ingestion Recipe for S3

source:
  type: dbt
  config:
    manifest_path: "s3://<your-bucket>/dbt/target/manifest.json"
    catalog_path: "s3://<your-bucket>/dbt/target/catalog.json"
    # Optional: add these if you want test results and source freshness
    # run_results_paths:
    #   - "s3://<your-bucket>/dbt/target/run_results.json"
    # sources_path: "s3://<your-bucket>/dbt/target/sources.json"
    target_platform: "<your-platform>"  # e.g., snowflake, bigquery, redshift
    aws_connection:
      aws_access_key_id: "${AWS_ACCESS_KEY_ID}"
      aws_secret_access_key: "${AWS_SECRET_ACCESS_KEY}"

sink:
  type: datahub-rest
  config:
    server: "https://<your-instance>.datahubproject.io/api/gms"
    token: "${DATAHUB_TOKEN}"

Step 4: Configure the Ingestion Recipe for GCS

GCS authentication for the dbt source uses HMAC keys (not a service account JSON file). You must first create an HMAC key pair in your GCS project. See Google's HMAC key documentation for setup instructions.

source:
  type: dbt
  config:
    manifest_path: "gs://<your-bucket>/dbt/target/manifest.json"
    catalog_path: "gs://<your-bucket>/dbt/target/catalog.json"
    # Optional: add these if you want test results and source freshness
    # run_results_paths:
    #   - "gs://<your-bucket>/dbt/target/run_results.json"
    # sources_path: "gs://<your-bucket>/dbt/target/sources.json"
    target_platform: "<your-platform>"  # e.g., snowflake, bigquery, redshift
    gcs_connection:
      access_hmac_key: "${GCS_HMAC_ACCESS_KEY}"
      secret_hmac_key: "${GCS_HMAC_SECRET_KEY}"

sink:
  type: datahub-rest
  config:
    server: "https://<your-instance>.datahubproject.io/api/gms"
    token: "${DATAHUB_TOKEN}"

Important: Store secrets (access keys, tokens) as DataHub secrets or environment variables rather than hardcoding them inline in the recipe.

Step 5: Automate Artifact Upload from Your dbt Pipeline

The recommended automation pattern is:

Run your dbt pipeline (e.g., dbt run and then dbt docs generate).
As the final step in your CI/CD job (GitHub Actions, Airflow, Jenkins, etc.), upload the target/ artifacts to your fixed cloud storage prefix, overwriting existing files.
Schedule DataHub ingestion to run shortly after the upload completes.

Example upload commands for the last step of your pipeline:

# For S3 (using AWS CLI)
aws s3 cp target/manifest.json s3://<your-bucket>/dbt/target/manifest.json
aws s3 cp target/catalog.json s3://<your-bucket>/dbt/target/catalog.json

# For GCS (using gsutil)
gsutil cp target/manifest.json gs://<your-bucket>/dbt/target/manifest.json
gsutil cp target/catalog.json gs://<your-bucket>/dbt/target/catalog.json

If you also want to capture test results (run_results.json), copy the file aside before running dbt docs generate, then upload it:

# Run dbt and capture test results before docs generation overwrites run_results.json
dbt test
cp target/run_results.json target/run_results_backup.json
dbt docs generate
cp target/run_results_backup.json target/run_results.json

# Upload all artifacts
gsutil cp target/manifest.json gs://<your-bucket>/dbt/target/manifest.json
gsutil cp target/catalog.json gs://<your-bucket>/dbt/target/catalog.json
gsutil cp target/run_results.json gs://<your-bucket>/dbt/target/run_results.json

Step 6: Schedule DataHub Ingestion

Once the recipe is configured, set a schedule on the ingestion source in the DataHub UI (or via CLI) so it runs automatically after each artifact upload. Without a schedule, ingestion only runs when triggered manually.

For instructions on scheduling via the UI, see the UI Ingestion documentation.

Alternative: dbt Cloud Source

If you are using dbt Cloud rather than dbt Core, there is a dedicated dbt-cloud source type that pulls the same metadata directly from the dbt Cloud API using a read-only API token. This eliminates all file handling and cloud storage configuration entirely. See the dbt Cloud module documentation for details.

Additional Notes

Stateful ingestion: If stateful ingestion is enabled on your dbt source, any dataset present in a previous run but absent from the current artifacts will be soft-deleted in DataHub. Always upload manifest.json and catalog.json together, and only from a complete, successful dbt run. Uploading a partial or incomplete artifact file will cause models missing from the artifact to disappear from DataHub on the next run.
Artifact consistency: Ensure both manifest.json and catalog.json originate from the same dbt run. A mismatch (e.g., a model present in the manifest but absent from the catalog) will result in incomplete metadata for those models.
GCS authentication: The dbt source connector for GCS requires HMAC keys, not a service account JSON key file. Ensure the HMAC key has at minimum read access to the target bucket prefix.
Required field: target_platform is a required configuration field and must match the platform your dbt models run on (e.g., snowflake, bigquery, redshift).
Glob patterns: If you use glob patterns for run_results_paths, be careful to avoid picking up artifacts from multiple runs, which can cause old test failures to be reprocessed on every ingestion cycle. See the glob patterns documentation for guidance.