Context Generation Returns Zero Queries Due to Missing Usage Features – DataHub Customer Support Portal

Area: Observability Issues

Sub-Area: AI Context Generation / Query Corpus

Issue

When running an AI context generation job scoped to a domain, the job completes successfully but produces no documents. The run report shows queries_read=0, anchors_generated=0, and query_corpus_empty=true, even though the DataHub GraphQL API (listQueries) and the MCP return a substantial number of query entities for the same set of datasets. The discrepancy occurs because the context generation engine silently drops any query that does not yet have computed usage features (e.g., 30-day run counts and top users). Crucially, skipped queries are not counted in the run log's skip counter, so skips={} is reported and the problem is not immediately obvious from the logs alone.

Error Messages

ElasticSearchQuerySource: yielded=0, errors=0, skips={}
queries_read=0, anchors_generated=0, query_corpus_empty=true

You Might Be Asking

Why does listQueries return hundreds of queries while context generation reads zero?
Why does the run log show skips={} even though queries exist in the system?
Why did context generation work for one domain but not another on the same executor at nearly the same time?
How do usage features get attached to queries, and when are they available?
Will view definitions ever be included in context generation?

Solution

Understand the root cause.
Context generation (on versions prior to 2.4.0) only reads queries that already have computed usage features. Usage features are computed by the platform's usage ingestion job, not by the query ingestion job. The two jobs are independent. A query entity can exist in DataHub — and be returned by the API — long before the usage job has processed it. Queries without usage features are silently dropped before the skip counter is incremented, which is why skips={} appears even when no queries are read.

The listQueries API and the MCP do not require usage features, so they always return the full set of matching query entities. This creates the apparent discrepancy between the API count and the context generation read count.
Identify whether queries are missing usage features.
Use the GraphQL API to check whether the queries in scope have queryUsageStatistics populated. If queryCount and similar fields are null or absent on most queries, the usage job has not yet processed them.
```
query CheckQueryUsage {
  query(urn: "urn:li:query:<your-query-urn>") {
    queryUsageStatistics {
      queryCount
      topUsers {
        user { urn }
      }
    }
  }
}
```
Repeat for a sample of the in-scope queries. If the majority return null for queryUsageStatistics, usage features have not been computed yet.
Check the timing relationship between jobs.
Confirm the scheduled run time of the usage ingestion job versus the context generation job. A common misconfiguration is a context generation job scheduled before the daily usage job completes. For example:
- Usage job runs at 09:15 UTC.
- Queries are ingested by a source run at 20:50 UTC the previous day.
- Context generation runs at 00:00 local time (e.g., 05:00 UTC) — before the next usage job.
In this scenario, even a nightly scheduled context generation run will always see zero usage features and produce no documents.
Reschedule context generation to run after the usage job.
Adjust the context generation job's schedule so it starts after the usage ingestion job has finished. Allow a buffer for the usage job to complete before context generation begins. For example, if the usage job runs at 09:15 UTC and typically takes 30 minutes, schedule context generation no earlier than 10:00 UTC.
```
# Example cron expression for context generation scheduled after the usage job
# Run at 10:00 UTC daily
schedule:
  interval: "0 10 * * *"
  timezone: "UTC"
```
Trigger a manual re-run after usage features are available.
Once the usage job has run and usage features have been computed for the in-scope queries, manually trigger the context generation job from the DataHub UI or via the ingestion API:
```
curl -X POST \
  "https://<your-instance>.datahubproject.io/api/v2/ingestion/sources/<your-source-urn>/run" \
  -H "Authorization: Bearer <your-token>" \
  -H "Content-Type: application/json"
```
After the run, verify that queries_read is greater than zero in the run report.
Note the limitation for view definitions (pre-2.4.0).
View definitions ingested as SYSTEM-source query entities typically do not receive usage statistics through the standard usage pipeline. On versions prior to 2.4.0, these will always be excluded from context generation regardless of scheduling. See the Additional Notes section below for the fix available in 2.4.0.

Additional Notes

Version behavior: This silent-drop behavior affects integrations server versions prior to 2.4.0. Starting in version 2.4.0, context generation reads all in-scope queries regardless of whether usage features have been computed, eliminating the dependency on usage job timing for query discovery. Upgrading to 2.4.0 or later is the long-term resolution, particularly for view definitions which cannot receive usage features through the standard pipeline.

Why other domains may work: If context generation succeeds for one domain but not another on the same executor, the difference is typically that the working domain's queries already had usage features computed (e.g., from an earlier usage job run), while the failing domain's queries were ingested or updated after the last usage job completed.

Skip counter behavior: The absence of entries in skips={} does not mean all queries were read. Queries dropped due to missing usage features are excluded before the skip counter is incremented. This is a known logging gap that makes the issue difficult to diagnose from run logs alone.

API vs. context generation filter: The listQueries GraphQL API and the MCP do not apply usage feature filters. They will always return query entities that match by dataset URN and platform, regardless of usage feature availability. The count returned by these APIs is not a reliable indicator of how many queries context generation will process on versions prior to 2.4.0.

Issue

Error Messages

You Might Be Asking

Solution

Additional Notes

Related Documentation