Domain-Scoped Ingestion Policies and Transformer Ordering: Why Newly Created Entities Get 403 Errors – DataHub Customer Support Portal

Area: Ingestion Issues

Sub-Area: Authorization / Domain-Scoped Policies

Issue

When a DataHub metadata policy is scoped to a DOMAIN criterion, the policy engine authorizes writes by checking the domain currently stored on the entity, not the domain being proposed in the current write. For brand-new entities that do not yet exist in DataHub, this creates a bootstrapping problem: if the domains aspect is not included in the same API request as the entity-creating aspect (e.g., dashboardInfo, datasetProperties), the policy engine finds no stored domain, cannot match the criterion, and returns a 403 Forbidden error. This problem is most commonly triggered in recipe-driven ingestion (CLI, UI, or remote executors) when domain-assignment transformers such as pattern_add_dataset_domain or pattern_add_domain are used, because these transformers accumulate domain aspects and emit them in a separate, later API request — after the entity aspects have already been written and rejected.

Error Messages

403 Forbidden
Not authorized to perform action: EDIT_ENTITY_DOMAINS

You Might Be Asking

Why do my domain-scoped policies work fine for existing entities but fail with 403 for new ones?
Does aspect ordering within a request matter for domain-scoped policy evaluation?
Why does using a domain transformer break ingestion when a domain-scoped service account is used?
Can I force the domains aspect to be emitted first from a recipe ingestion run?
Is there a supported way to use domain-scoped policies with recipe-driven ingestion for net-new entities?

Solution

Understanding the Root Cause

How domain-scoped policy evaluation works: The policy engine resolves the DOMAIN criterion by looking up the domains aspect currently stored on the entity. For a brand-new URN with no persisted domains aspect, any write that arrives in a separate API request before the domains aspect has been stored will be evaluated against an empty domain set and denied.
Proposed-domain seeding (same-batch evaluation): Since DataHub Cloud 2.1.0, a mechanism called proposed-domain seeding allows the policy engine to evaluate domain authorization using the domains aspect included in the same API batch request. This means aspect ordering within a single request does not matter — if both the domains aspect and the entity aspect (e.g., dashboardInfo) are sent in the same HTTP request, the 403 will not occur regardless of which appears first in the payload.
Why transformers break this: Domain-assignment transformers (pattern_add_dataset_domain, pattern_add_domain) buffer the domains aspect and emit it at the end of the ingestion run. This means the domains MCP is always sent in a later, separate HTTP request after the entity aspects — bypassing the same-batch seeding mechanism entirely and causing 403 errors on new entities.

Recommended Workaround: Set Domains in Source Config Instead of Transformers

For connectors that support native domain configuration (such as BigQuery, Glue, and Databricks), set the domain directly in the source config rather than via a transformer. When configured this way, the connector emits the domains aspect inline with the entity aspects in the same API request, satisfying the same-batch seeding mechanism.

Remove the domain-assignment transformer from your recipe. You do not need to remove other transformers (e.g., ownership or data product transformers) — only the domain-assignment one.

Add the domain key to your source configuration in the recipe YAML:

source:
  type: bigquery  # or glue, databricks, etc.
  config:
    # ... other source config ...
    domain:
      "urn:li:domain:<your-domain-urn>":
        allow:
          - ".*"

Replace <your-domain-urn> with the URN of the target domain (e.g., urn:li:domain:finance).

Verify ingestion succeeds for new entities by running the recipe with the domain-scoped service account. Existing entities are unaffected by this change since they already have a stored domains aspect.

Alternative: SDK / REST Emitters (Programmatic Ingestion)

If you are using the DataHub Python SDK or REST emitters directly rather than recipe-driven ingestion, you can ensure the domains MCP and entity MCPs are emitted together in a single batch call:

from datahub.emitter.mce_builder import make_domain_urn, make_dataset_urn
from datahub.emitter.mcp import MetadataChangeProposalWrapper
from datahub.metadata.schema_classes import DomainsClass, DatasetPropertiesClass
from datahub.emitter.rest_emitter import DatahubRestEmitter

emitter = DatahubRestEmitter(gms_server="http://<your-datahub-host>:8080")

dataset_urn = make_dataset_urn(platform="<platform>", name="<dataset-name>")
domain_urn = make_domain_urn("<your-domain-name>")

# Emit domains aspect and entity aspect in the same batch
domains_mcp = MetadataChangeProposalWrapper(
    entityUrn=dataset_urn,
    aspect=DomainsClass(domains=[domain_urn]),
)
properties_mcp = MetadataChangeProposalWrapper(
    entityUrn=dataset_urn,
    aspect=DatasetPropertiesClass(description="My dataset"),
)

# Both MCPs in a single emit_mcps call = same API request = seeding applies
emitter.emit_mcps([domains_mcp, properties_mcp])

Known Limitations

dataJob and dataFlow entities: The native domain source config does not cover dataJob or dataFlow entity types. These remain subject to the transformer ordering problem.
Subdomain auto-creation: A domain-scoped service account cannot automatically create missing subdomains. The domain entity type does not carry a domains aspect itself, so a DOMAIN-scoped policy cannot authorize creating new subdomain entities. This is a known product gap.
Cross-domain policy scoping: Domain restrictions are strictly enforced and do not leak to sibling or parent domains. An entity assigned to a domain the policy does not cover will correctly receive a 403.

Additional Notes

This behavior was verified on DataHub Cloud 2.1.2. The proposed-domain seeding mechanism (DomainWriteAuthorizationUtils.seedProposedDomainsFromMcps()) was introduced in DataHub Cloud 2.1.0 and applies to same-batch API requests on both the Rest.li (/aspects?action=ingestProposalBatch) and OpenAPI (/openapi/v3/entity/<type>) paths. Recipe-driven ingestion via the CLI, the DataHub UI, or remote executors does not currently guarantee that the domains MCP and entity MCPs for a given URN land in the same HTTP request when a domain transformer is used. A product enhancement request has been filed to ensure domain-scoped policies work correctly across all ingestion paths, including recipe ingestion, without requiring source-level configuration as a workaround. This article will be updated when a platform-level fix is available.