Fivetran Connector Limitations with Glue-Backed Data Lake Destinations – DataHub Customer Support Portal

Area: Ingestion Issues

Sub-Area: Fivetran Connector — Destination Platform Support

Issue

When Fivetran is configured to replicate data to an S3-backed data lake destination (managed by AWS Glue), the DataHub Fivetran connector cannot be used directly. The Fivetran connector requires that Fivetran Platform Connector logs — the metadata tables that record connector info, sync history, and lineage — be written into Snowflake, BigQuery, or Databricks. In lakehouse architectures where Fivetran writes to S3 and registers metadata in Glue, and where Snowflake surfaces those assets via a Glue catalog-linked database rather than receiving data directly from Fivetran, the connector will fail. Additionally, because Glue-managed schemas are created with lowercase quoted identifiers and the DataHub Fivetran connector historically uppercased schema names before quoting them in Snowflake queries, the connector would produce case-sensitive mismatches even when the Fivetran Platform Connector logs were technically accessible through a catalog-linked database.

Error Messages

Schema '<your-database>.<fivetran_metadata_schema>' does not exist or not authorized

You Might Be Asking

Why does the Fivetran connector fail when my Fivetran Platform Connector logs appear to be accessible in Snowflake?
Does DataHub support Fivetran-managed data lake (Glue) destinations?
What is the recommended ingestion strategy when Fivetran writes to S3/Glue instead of directly to Snowflake, BigQuery, or Databricks?
Does the Glue connector provide Glue-to-Snowflake lineage, or only S3-to-Glue lineage?
Why does DataHub generate quoted, uppercase schema names that fail in Snowflake when the same schema name works unquoted?

Solution

Option 1: Upgrade to DataHub v1.6.0 / CLI v1.5.0.19 or Later (Recommended)

Native support for Fivetran-managed data lake destinations (including Glue-backed S3 destinations) was added in DataHub OSS v1.6.0 and CLI v1.5.0.19 (PR #17217). This release also includes a REST API mode for the Fivetran connector. If you are on a supported version, upgrade and reconfigure your Fivetran ingestion recipe to use the new destination type. Refer to the updated Fivetran source documentation for configuration details.

Option 2: Multi-Connector Approach (Prior Versions)

If you cannot upgrade immediately, use a combination of the AWS Glue, Snowflake, and optionally S3 connectors to achieve full catalog coverage. This is also the correct pattern for pipelines where Fivetran-specific lineage (connector metadata, sync history, source-to-destination mapping) is not required.

Configure the AWS Glue connector to ingest Glue catalog metadata, table definitions, and S3-to-Glue lineage.
```
source:
  type: glue
  config:
    aws_region: <your-aws-region>
    emit_s3_lineage: true
    glue_s3_lineage_direction: upstream  # or downstream, depending on your model
```
Note: The Glue connector produces S3-to-Glue lineage only. It does not automatically produce Glue-to-Snowflake lineage.

Configure the Snowflake connector to ingest the catalog-linked external tables that Snowflake surfaces from the Glue catalog. This adds another hop in the lineage graph but ensures those assets are discoverable in DataHub.

source:
  type: snowflake
  config:
    account_id: <your-snowflake-account>
    username: <your-service-account-username>
    password: <your-service-account-password>
    role: <your-datahub-role>
    warehouse: <your-warehouse>
    database_pattern:
      allow:
        - <your-glue-linked-database>

(Optional) Configure the S3 connector if you want file-level detail and schema extraction for Parquet or Avro files stored in S3.

source:
  type: s3
  config:
    aws_config:
      aws_region: <your-aws-region>
    path_specs:
      - include: "s3://<your-bucket>/<your-path>/**"

This multi-connector approach provides coverage across the full pipeline:

Source Systems → Fivetran → S3 → Glue Catalog → Snowflake (catalog-linked external tables)

However, it will not produce Fivetran-specific lineage (no connector metadata, sync history, or direct source-to-destination mapping from Fivetran). That capability requires the native Fivetran connector support added in v1.6.0.

Understanding the Schema Quoting Issue (Historical Context)

If you observed the error Schema '...' does not exist or not authorized even when your Snowflake role had access to the schema, the root cause was typically a case-sensitivity mismatch. Fivetran-managed Glue destinations create schemas using lowercase quoted identifiers (e.g., "fivetran_metadata_<schema_name>"). The DataHub Fivetran connector historically uppercased schema names before wrapping them in double quotes for Snowflake queries, producing "FIVETRAN_METADATA_<SCHEMA_NAME>", which does not match the lowercase schema stored in Glue-linked databases (which are read-only and cannot be renamed).

You can verify the mismatch by running these two queries in Snowflake:

-- This will succeed (lowercase, matches Glue-created schema):
USE ROLE <your-datahub-role>;
USE DATABASE "<your-glue-linked-database>";
SELECT * FROM "<fivetran_metadata_schema_lowercase>".connection LIMIT 1;

-- This will fail (uppercase, as DataHub historically sent it):
SELECT * FROM "<FIVETRAN_METADATA_SCHEMA_UPPERCASE>".connection LIMIT 1;

This issue is resolved in DataHub v1.6.0 / CLI v1.5.0.19 as part of the Managed Data Lake destination support work.

Additional Notes

Version: Native Fivetran Managed Data Lake destination support is available in DataHub OSS v1.6.0, CLI v1.5.0.19, and corresponding SaaS releases. Prior versions require the multi-connector workaround described above.
Glue connector lineage direction: The glue_s3_lineage_direction config parameter controls whether S3 datasets are treated as upstream or downstream of Glue tables. Set this based on your data flow direction. Column-level lineage can be enabled with emit_s3_lineage: true.
Snowflake catalog-linked databases: Ingesting a Glue-linked Snowflake database via the Snowflake connector adds those tables as discoverable assets in DataHub, but on its own does not create lineage back to S3 or Glue. Combine with the Glue connector for broader lineage coverage.
Fivetran Platform Connector requirement: Even with v1.6.0+ support, the Fivetran Platform Connector must be enabled in your Fivetran account for the DataHub Fivetran connector to retrieve connector metadata and lineage. Confirm this is configured before running ingestion.
QUOTED_IDENTIFIERS_IGNORE_CASE: Setting this Snowflake parameter at the user or account level does not resolve the case mismatch when the schema is surfaced via a read-only Glue-linked catalog database, because the linked objects cannot be renamed to uppercase.