Skip to main content

Using `startTimeMillis` and `LineageFlags` in `searchAcrossLineage` and `scrollAcrossLineage` GraphQL Queries

Angelo Bresci

Area: API

Sub-Area: Lineage API

Issue

Users often need to filter lineage queries by time range to reduce result set size and improve performance. The `startTimeMillis` parameter (and, more recently, the `LineageFlags` input) allows filtering lineage results by a specific time window, but the usage and supported parameters differ between `searchAcrossLineage` and `scrollAcrossLineage` GraphQL queries.

Common Error Messages:

  • "Field 'startTimeMillis' is not defined by type 'SearchAcrossLineageInput'"

You Might Be Asking:

  • How do I filter lineage queries by time range in GraphQL?
  • Why does startTimeMillis not work in searchAcrossLineage?
  • What is the difference between using startTimeMillis and LineageFlags?
  • How do I use time filters in scrollAcrossLineage?

Solution

To filter lineage queries by time range, use the appropriate parameter for your query type:

1. searchAcrossLineage Query

  • Current Best Practice: Use the lineageFlags input object to specify startTimeMillis and endTimeMillis.
  • Legacy Support: The direct startTimeMillis and endTimeMillis fields are deprecated and may not be available in all deployments.

Example: Filtering downstream lineage by time range using lineageFlags


```graphql
query searchAcrossLineage {
  searchAcrossLineage(
    input: {
      urn: "urn:li:dataset:(urn:li:dataPlatform:hive,SampleHiveDataset,PROD)"
      query: "*"
      start: 0
      count: 10
      direction: DOWNSTREAM
      orFilters: [
        { and: [{ field: "degree", condition: EQUAL, values: ["1"] }] }
      ]
      lineageFlags: {
        startTimeMillis: 1700000000000
        endTimeMillis: 1705000000000
      }
    }
  ) {
    searchResults {
      entity {
        urn
        type
      }
      degree
    }
  }
}
```

2. scrollAcrossLineage Query

  • Supported: You can use either the direct startTimeMillis and endTimeMillis fields or the lineageFlags object.
  • Recommendation: Prefer lineageFlags for consistency with searchAcrossLineage.

Example: Filtering with startTimeMillis directly



```graphql
query scrollAcrossLineage {
  scrollAcrossLineage(
    input: {
      urn: "urn:li:dataset:(urn:li:dataPlatform:hive,SampleHiveDataset,PROD)"
      query: "*"
      count: 10
      direction: DOWNSTREAM
      startTimeMillis: 1700000000000
      endTimeMillis: 1705000000000
      orFilters: [
        { and: [{ field: "degree", condition: EQUAL, values: ["1"] }] }
      ]
    }
  ) {
    searchResults {
      entity {
        urn
        type
      }
      degree
    }
  }
}
```

Example: Filtering with lineageFlags (recommended)



```graphql
query scrollAcrossLineage {
  scrollAcrossLineage(
    input: {
      urn: "urn:li:dataset:(urn:li:dataPlatform:hive,SampleHiveDataset,PROD)"
      query: "*"
      count: 10
      direction: DOWNSTREAM
      orFilters: [
        { and: [{ field: "degree", condition: EQUAL, values: ["1"] }] }
      ]
      lineageFlags: {
        startTimeMillis: 1700000000000
        endTimeMillis: 1705000000000
      }
    }
  ) {
    searchResults {
      entity {
        urn
        type
      }
      degree
    }
  }
}
```

Why the Difference?

The lineageFlags object was introduced to standardize time and other advanced filters across lineage queries. Direct use of startTimeMillis is deprecated in searchAcrossLineage but still supported in scrollAcrossLineage for backward compatibility. Always check your DataHub version and GraphQL schema for supported fields.

Additional Notes

  • If you see errors about missing startTimeMillis, switch to using lineageFlags.
  • Filtering by time can significantly reduce query load and error rates, especially in large environments.
  • For the latest schema and parameter support, refer to your instance's GraphQL introspection or the official documentation.
  • Some UI tools (like GraphiQL) may not show deprecated fields; always use the latest recommended approach.

Related Tickets

  • 5534, 5974

Tags: api, lineage, graphql, searchAcrossLineage, scrollAcrossLineage, startTimeMillis, lineageFlags, time-filter, query-optimization, error-messages

Related to

Related articles