AWS Lake Formation Permissions Required for Glue Connector Ingestion – DataHub Customer Support Portal

Area: Ingestion Issues

Sub-Area: AWS Glue Source — Lake Formation Access Control

Issue

When AWS Lake Formation is configured as the governance layer over an AWS Glue Data Catalog, the DataHub Glue connector may silently skip databases and tables during ingestion even though the IAM role has the standard Glue permissions (glue:GetDatabases, glue:GetTables, etc.). This occurs because Lake Formation enforces its own permission layer on top of IAM: the Glue Data Catalog APIs filter out any resource for which the calling IAM principal lacks an explicit Lake Formation DESCRIBE grant. No error is raised — the resources simply do not appear in the API response, so ingestion completes successfully while LF-governed databases and tables are never discovered.

You Might Be Asking

Why are some Glue databases missing from DataHub after ingestion even though my IAM role has full Glue permissions?
Why does my Glue ingestion succeed with no errors but not pick up Lake Formation-governed databases?
How do I grant Lake Formation permissions in bulk across many accounts and resources without using the AWS Console?
Does adding lakeformation:GetDataAccess to the IAM policy fix the missing databases problem?

Solution

Update the IAM role policy to include Lake Formation actions.
In addition to the standard Glue actions, the DataHub ingestion IAM role must include lakeformation:GetDataAccess. Without it, calls against LF-governed resources may return empty results. If you also want to ingest Lake Formation tags as DataHub tags, include lakeformation:GetResourceLFTags as well.

Minimum recommended IAM policy statement:
```
{
  "Effect": "Allow",
  "Action": [
    "glue:GetDatabases",
    "glue:GetTables",
    "glue:GetPartitions",
    "glue:GetTableVersions",
    "lakeformation:GetDataAccess"
  ],
  "Resource": "*"
}
```
If Lake Formation tag extraction is also required, add:
```
{
  "Effect": "Allow",
  "Action": [
    "lakeformation:GetResourceLFTags"
  ],
  "Resource": "*"
}
```
Grant Lake Formation DESCRIBE permissions to the IAM role.
Updating the IAM policy alone is not sufficient. A Lake Formation administrator must also grant the DataHub IAM role explicit DESCRIBE permissions on every database and table to be ingested. This can be done via the AWS Console (Lake Formation → Data Permissions → Grant) or programmatically.

DESCRIBE is sufficient for catalog metadata ingestion. SELECT is only required if column-level profiling or sample data queries are needed.
Grant permissions in bulk using the AWS CLI (recommended for large environments).
When managing permissions across many accounts or resources, use batch-grant-permissions via the AWS CLI or SDK. This requires no console access and is fully scriptable. A Lake Formation administrator can enumerate all databases and tables using list-resources and then call batch-grant-permissions iteratively.

Example CLI command to grant DESCRIBE on a single database:
```
aws lakeformation batch-grant-permissions \
  --catalog-id <ACCOUNT_ID> \
  --entries '[
    {
      "Id": "grant-db-1",
      "Principal": {
        "DataLakePrincipalIdentifier": "arn:aws:iam::<ACCOUNT_ID>:role/<DataHubIngestionRole>"
      },
      "Resource": {
        "Database": {
          "CatalogId": "<ACCOUNT_ID>",
          "Name": "<DATABASE_NAME>"
        }
      },
      "Permissions": ["DESCRIBE"],
      "PermissionsWithGrantOption": []
    }
  ]'
```
To cover all databases in an account, wrap the above in a script that iterates the output of:
```
aws glue get-databases --catalog-id <ACCOUNT_ID>
```
Verify the ignore_resource_links recipe setting.
If the Lake Formation databases are shared from another AWS account via AWS RAM, they appear in the Glue catalog as resource links. The DataHub Glue connector handles resource links by default. Confirm that ignore_resource_links is not set to true in your ingestion recipe, as this will cause all LF-shared databases to be skipped entirely.
```
source:
  type: glue
  config:
    # Ensure this is false (default) to process LF-shared resource links
    ignore_resource_links: false
```
Optionally enable Lake Formation tag ingestion.
If you want Lake Formation tags extracted and ingested as DataHub tags, enable the following config flag in your recipe. Note that only table-level and database-level LF tags are currently supported; column-level LF tags are not yet ingested.
```
source:
  type: glue
  config:
    extract_lakeformation_tags: true
```
Re-run ingestion and verify results.
After updating both the IAM policy and the Lake Formation grants, re-run the Glue ingestion. To confirm which databases were previously being silently skipped, enable debug logging. Any database that failed to enumerate its tables due to missing LF permissions will produce a warning in the ingestion report visible under the Ingestion section of the DataHub UI. A clean run with no "Failed to get tables from database" warnings confirms the grants are in place.

Additional Notes

This filtering behavior is enforced at the AWS service layer — it is not a DataHub bug or configuration issue. The Glue connector correctly calls the standard GetDatabases and GetTables APIs; AWS simply does not return LF-governed resources to principals that lack the corresponding Lake Formation grants. No DataHub-side configuration can bypass this restriction. Additionally, the lakeformation:GetDataAccess IAM action addresses certain edge cases with LF-governed resource access but does not substitute for explicit Lake Formation data permission grants — both are required. If managing Lake Formation grants across many AWS accounts is not feasible, consider evaluating the Athena source connector as an alternative; however, note that Athena also respects Lake Formation access controls by default, so environments with strict LF enforcement will encounter the same filtering behavior unless an IAM-access-based workgroup is available.

Issue

You Might Be Asking

Solution

Additional Notes

Related Documentation