Accessing Ingestion Run History Programmatically via GraphQL API – DataHub Customer Support Portal

Area: API Issues

Sub-Area: Ingestion Source & Execution Request GraphQL Resolvers

Issue

Users who need to programmatically access ingestion run history — including execution status, duration, timestamps, and actor information — cannot retrieve this data through the general-purpose searchAcrossEntities API, the MCP server's search and get_entities tools, or OpenSearch-backed entity search. This is because the ingestion management UI uses dedicated GraphQL resolvers that bypass the OpenSearch index entirely. Attempts to search for INGESTION_SOURCE or EXECUTION_REQUEST entity types return zero results, and querying DATA_PROCESS_INSTANCE with timing fields causes shard errors. The correct path for automation — including service-account-driven reporting jobs — is the DataHub GraphQL API via /api/graphql.

Error Messages

search entity_type=INGESTION_SOURCE → total: 0
search entity_type=EXECUTION_REQUEST → total: 0
OpenSearch: all shards failed (DATA_PROCESS_INSTANCE sort by duration)
get_entities on dataProcessInstance urns returns only urn and url — no status or timing fields

You Might Be Asking

Why does searching for INGESTION_SOURCE or EXECUTION_REQUEST return zero results even though those indexes exist?
Is the listIngestionSources GraphQL query supported for automation and service-account tokens, or is it internal-UI-only?
Is there a single GraphQL call that returns execution history across all ingestion sources, or do I have to make one call per source (N+1)?
What privilege does a service account need to call these resolvers?
How long is ingestion run history retained before cleanup removes it?
Can I receive ingestion run completions as events instead of polling?

Solution

Understand why search-based tools do not work for ingestion data.
The ingestion management UI — and its underlying GraphQL resolvers (listIngestionSources, listExecutionRequests) — reads directly from the metadata store, not from the OpenSearch index. Entity types INGESTION_SOURCE and EXECUTION_REQUEST are not indexed for searchAcrossEntities. MCP server tools such as search and get_entities wrap searchAcrossEntities and therefore cannot surface this data. No indexing flag or configuration change will alter this behavior.
Grant the MANAGE_INGESTION platform privilege to your service account.
Both listIngestionSources and listExecutionRequests enforce a server-side authorization check for the MANAGE_INGESTION platform privilege. Without it, calls return a 403 regardless of authentication method. Service-account tokens and human personal access tokens (PATs) are treated identically by the authorization layer — there is no token-type-specific scope. Assign this privilege via a platform policy in Settings → Policies.
Use listExecutionRequests for a global, cross-source run history view.
This resolver returns execution requests across all ingestion sources in a single paginated call, sorted by requestTimeMs descending by default — mirroring the unfiltered Run History tab in the UI. It accepts start, count, query, filters, sort, and systemSources inputs. Native server-side time-window filtering via filters depends on whether requestTimeMs is indexed as a facet in your deployment; if not, paginate descending and stop client-side when the timestamp exits your window.
```
POST https://<your-instance>.acryl.io/api/graphql
Authorization: Bearer <your-service-account-token>
Content-Type: application/json

{
  "query": "query ListExecutionRequests($input: ListExecutionRequestsInput!) { listExecutionRequests(input: $input) { start count total executionRequests { urn result { status startTimeMs durationMs } input { requestedAt actor { urn } } } } }",
  "variables": {
    "input": {
      "start": 0,
      "count": 100
    }
  }
}
```

Use listIngestionSources with a nested executions field for per-source metadata and last-run summary.

Embedding executions(start: 0, count: 1) inside listIngestionSources retrieves the most recent execution for every source in a single GraphQL request — no N+1 calls required for the common last-run-summary use case.

POST https://<your-instance>.acryl.io/api/graphql
Authorization: Bearer <your-service-account-token>
Content-Type: application/json

{
  "query": "query ListIngestionSources($input: ListIngestionSourcesInput!) { listIngestionSources(input: $input) { start count total ingestionSources { urn name type schedule { interval timezone } executions(start: 0, count: 1) { start count total executionRequests { urn result { status startTimeMs durationMs } input { requestedAt actor { urn } } } } } } }",
  "variables": {
    "input": {
      "start": 0,
      "count": 100
    }
  }
}

Paginate through all ingestion sources and full execution history as needed.

For a daily report covering all sources with full run history, combine both calls: listIngestionSources (paginated with start/count) for source metadata and schedule, and listExecutionRequests (paginated, sorted descending) for the cross-source execution log. Filter client-side by timestamp for your desired reporting window.

# Pseudocode for a daily ingestion health report

sources = []
start = 0
PAGE = 100
while True:
    resp = graphql(listIngestionSources, {start: start, count: PAGE})
    sources += resp.ingestionSources
    if start + PAGE >= resp.total:
        break
    start += PAGE

executions = []
start = 0
WINDOW_MS = 86400000  # last 24 hours
now_ms = current_epoch_ms()
while True:
    resp = graphql(listExecutionRequests, {start: start, count: PAGE})
    for ex in resp.executionRequests:
        if ex.result.startTimeMs < now_ms - WINDOW_MS:
            break  # sorted descending; stop when outside window
        executions.append(ex)
    else:
        start += PAGE
        continue
    break

# Compute % success, % failure, % stale from executions[]

(Optional) Use the Actions Framework for push-based run completion events.
If you prefer not to poll, configure an Actions Framework pipeline to listen to METADATA_CHANGE_LOG_EVENT_V1 events filtered on entityType = "dataHubExecutionRequest" and aspectName = "dataHubExecutionRequestResult" with changeType = UPSERT. The result.status field will be one of: SUCCESS, FAILURE, TIMEOUT, RUNNING, or CANCELLED. This allows you to maintain your own persistent run-history store without polling.

Additional Notes

Run history retention: Retention is controlled by the datahub-gc ingestion source via DatahubExecutionRequestCleanupConfig. Default values are: keep_history_min_count = 10 per source, keep_history_max_count = 1000 per source, and keep_history_max_days = 90 days. Your maximum default trend window for reporting is therefore 90 days. To extend this, increase keep_history_max_days in your datahub-gc source configuration, or use the Actions Framework approach to persist events to an external store indefinitely.

No OpenAPI v3 route for ingestion aspects: There is no general list/scroll REST endpoint for dataHubExecutionRequest or dataHubIngestionSource aspects. The Rest.li-based BatchIngestionRunResource exists for rollback and diagnostics tooling, not for reporting. Use the GraphQL path.

No MCP server ingestion tools today: As of the time of writing, list_ingestion_sources and get_ingestion_run_history MCP tools do not exist. The supported and recommended automation path is direct GraphQL calls to /api/graphql with a service-account token.

No pre-built ingestion health analytics API: DataHub does not currently expose a dedicated analytics or platform-health surface for ingestion metrics. Build your reporting logic on top of the GraphQL resolvers described above.

Issue

Error Messages

You Might Be Asking

Solution

Additional Notes

Related Documentation