Skip to main content

BigQuery Permission Errors in Data Quality Assertions

Angelo Bresci
Updated

Area: Observability Issues

Sub-Area: Assertion Execution

Issue

Data quality assertions suddenly fail with BigQuery permission errors, specifically missing the bigquery.jobs.create permission, even when service account configurations appear unchanged. This typically manifests as all assertions failing simultaneously rather than gradual degradation.

Error Messages

  • User does not have bigquery.jobs.create permission in project <project-name>
  • 403 Forbidden: bigquery.jobs.create permission denied

You Might Be Asking

  • Why did all my assertions start failing at the same time?
  • What BigQuery permissions does DataHub need for assertions?
  • How do I troubleshoot sudden assertion failures?

Solution

  1. Check GCP IAM Audit Logs 
    # Look for IAM policy changes around the time assertions started failing
# Check for:
# - IAM binding removals
# - Role changes
# - Organization policy modifications
# - Service account key rotations
  2. Verify Service Account Permissions 
    # Ensure the service account has the required role
gcloud projects add-iam-policy-binding <project-id> \
    --member="serviceAccount:<service-account-email>" \
    --role="roles/bigquery.jobUser"
  3. Check DataHub Connection Configuration
    • Navigate to DataHub Cloud → Settings → Connections
    • Verify that the BigQuery monitoring connection is active
    • Test the connection to ensure credentials are valid
    • Check if the correct service account is being used
  4. Verify Project Context
    • Ensure the service account has bigquery.jobs.create permission in the project where assertion queries are executed
    • This may be different from the service account's home project
    • Check the error message for the specific project ID being referenced
  5. Review Recent Infrastructure Changes
    • Check Terraform or other infrastructure-as-code changes
    • Verify no recent service account binding modifications
    • Review any recent GCP organization policy updates
  6. Test Assertion Execution
    • Once permissions are restored, assertions should resume automatically
    • Monitor the next scheduled assertion run or trigger manually
    • Check assertion history for successful execution

Additional Notes

The bigquery.jobs.create permission is essential for DataHub's assertion executor to submit query jobs to BigQuery when running SQL-based assertions (volume, column, custom SQL, and freshness assertions). Sudden blanket failures typically indicate infrastructure-level changes rather than gradual configuration drift. The roles/bigquery.jobUser role includes the necessary bigquery.jobs.create permission. If you suspect a DataHub Cloud platform change coincided with the issue, contact support with specific error messages and timestamps for investigation.

Related Documentation

Tags: bigquery, assertions, permissions, observability, data-quality, service-account, iam, gcp, monitoring, troubleshooting

Related articles