BigQuery Assertions Fail After DataHub Upgrade Due to Conflicting start_time Configuration – DataHub Customer Support Portal

Area: Observability Issues

Sub-Area: BigQuery Assertion Connection Errors After Version Upgrade

Issue

After upgrading DataHub from version 2.0.3 to 2.1.3, all BigQuery assertions fail with a source connection error. The root cause is a breaking change introduced in 2.1.3 that affects how BigQuery ingestion recipes are parsed during assertion evaluation. Specifically, if a BigQuery recipe contains start_time defined at both the top level and nested under usage (e.g., usage.start_time), the recipe parser now raises a ValueError. The assertion execution engine's credential extraction logic swallows this error and returns None for the connection, causing every assertion tied to that BigQuery source to fail immediately. In DataHub 2.0.3, the nested usage.start_time field was silently ignored; in 2.1.3 it is treated as a hard conflict. This issue affects all managed BigQuery ingestion sources that share a recipe template with both fields populated.

Error Messages

Unable to retrieve valid connection for Data Platform with urn urn:li:dataPlatform:bigquery
Caught error of type AssertionResultErrorType.SOURCE_CONNECTION_ERROR when attempting to evaluate assertion with urn urn:li:assertion:<assertion-urn> and properties {'message': 'Unable to retrieve valid connection for Data Platform with urn urn:li:dataPlatform:bigquery', 'connection_urn': 'urn:li:dataPlatform:bigquery'}

You Might Be Asking

Why did all my BigQuery assertions suddenly start failing after a DataHub upgrade?
Is this a credential or permissions problem with my BigQuery service account?
Why does updating or re-saving my BigQuery connection in DataHub not fix the assertion failures?
Can I set a different start_time for usage statistics versus other ingestion in a BigQuery recipe?
Will tonight's scheduled BigQuery ingestion runs also fail because of this issue?

Solution

The fix is to remove the conflicting usage.start_time field from each affected BigQuery ingestion recipe. The top-level start_time should be retained. No changes to BigQuery service account credentials or permissions are required.

Identify affected BigQuery ingestion sources. Navigate to Ingestion in the DataHub UI and review all managed BigQuery ingestion sources. Any source whose recipe contains both a top-level start_time and a nested usage.start_time is affected. Example of the conflicting configuration:

# Conflicting configuration — causes ValueError in DataHub 2.1.3
source:
  type: bigquery
  config:
    start_time: '-1d'        # top-level start_time
    usage:
      start_time: '-7d'      # nested usage.start_time — CONFLICTS with above
    project_ids:
      - <your-gcp-project-id>

Edit each affected recipe to remove usage.start_time. Keep the top-level start_time and delete the usage.start_time key entirely. The corrected configuration should look like the following:

# Corrected configuration — compatible with DataHub 2.1.3
source:
  type: bigquery
  config:
    start_time: '-1d'        # top-level start_time retained
    usage: {}                # usage.start_time removed
    project_ids:
      - <your-gcp-project-id>

If the usage block has no other keys after removing start_time, you may remove the usage block entirely:

source:
  type: bigquery
  config:
    start_time: '-1d'
    project_ids:
      - <your-gcp-project-id>

Save the updated recipe in the DataHub UI for each affected ingestion source.
Wait approximately one hour for the assertion executor's connection cache to expire and refresh. Assertions should begin evaluating successfully after the cache clears. You do not need to restart any services manually.
Verify assertion recovery. Return to the affected datasets and confirm that assertions are now producing evaluation results (PASS or FAIL) rather than SOURCE_CONNECTION_ERROR failures.
Check upcoming scheduled ingestion runs. Any managed BigQuery ingestion sources that still carry the conflicting configuration will also fail their next scheduled ingestion run on DataHub 2.1.3. Ensure all recipes are updated before the next ingestion window to avoid a secondary outage.

Additional Notes

Version scope: This breaking change was introduced in DataHub 2.1.3. The behavior did not exist in 2.0.3 or earlier, where usage.start_time was silently ignored when a top-level start_time was also present.
Re-saving credentials does not help: The connection error is caused entirely by recipe parsing failure, not by invalid credentials. Updating or re-entering BigQuery service account keys in the DataHub Connections or Settings UI will not resolve this issue.
Template-based recipes at risk: Organizations that use a shared recipe template across many BigQuery ingestion sources may find that all sources are affected simultaneously. Audit all sources when troubleshooting.
Connection cache delay: Even after fixing the recipe, expect up to approximately one hour before assertions resume normal operation due to connection caching in the assertion executor.
Future behavior of usage.start_time: The DataHub team is actively evaluating adding proper support for usage.start_time as an override for the top-level start_time specifically for usage statistics ingestion. A future CLI release is expected to introduce this capability cleanly. In the meantime, remove usage.start_time from recipes and rely on the top-level start_time.
Underlying technical cause: The assertion executor's extract_connection_from_bigquery_recipe function parses the full BigQueryV2Config to extract credentials. In 2.1.3, the config validation raises a ValueError when both start_time and usage.start_time are set. This exception is caught and causes the function to return None, which the evaluator surfaces as a SOURCE_CONNECTION_ERROR.
Planned product fix: A longer-term fix to make assertion connection extraction parse only credential fields (rather than the full recipe config) is planned, which will reduce the blast radius of recipe validation errors on assertion evaluation.

Issue

Error Messages

You Might Be Asking

Solution

Additional Notes

Related Documentation