Timeline API Slowness Due to Datasets with Deeply Nested Schemas or High Schema Version Counts – DataHub Customer Support Portal

Area: API Issues

Sub-Area: Timeline API Performance

Issue

Calls to the DataHub Timeline API (/openapi/v2/timeline/v1/{urn}) intermittently time out or return extremely slow responses for specific datasets. Investigation reveals that the root cause is data-specific rather than infrastructure-wide: certain datasets accumulate an unusually high number of schemaMetadata versions in the backing SQL database, and/or have schemas that are deeply nested (e.g., 10 or more levels of nested children arrays). When the Timeline API is invoked with an unbounded time range (startTime=0), it must scan and deserialize all historical schema versions for the asset, causing response times that regularly exceed 30-second client timeouts. Other API endpoints such as /gms/aspects and /api/v2/graphql are typically unaffected and perform normally.

Error Messages

ReadTimeoutError("HTTPSConnectionPool(host='<your-instance>.acryl.io', port=443): Read timed out. (read timeout=30)")
Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError(...)'

You Might Be Asking

Why does the Timeline API time out for only certain datasets while other endpoints work fine?
Why are Timeline API calls slow even when the overall platform appears healthy?
How can I tell if a dataset has too many schema versions or a schema that is too deeply nested?
Is using startTime=0 safe for all datasets on the Timeline API?

Solution

Identify the problematic dataset(s).
Run a targeted test against the specific dataset URN that is causing timeouts. Compare response times across different assets to confirm the slowness is asset-specific rather than platform-wide:
```
curl -X GET \
  'https://<your-instance>.acryl.io/openapi/v2/timeline/v1/<encoded-dataset-urn>?startTime=0&endTime=0&raw=false&categories=TECHNICAL_SCHEMA&categories=TAG' \
  -H 'accept: application/json' \
  -H 'Authorization: Bearer <your-api-token>'
```
If the response is significantly slower than other dataset URNs (e.g., >10 seconds vs. milliseconds for typical assets), the dataset is likely affected by one or both of the root causes described in this article.
Check for a high number of schema versions.
A dataset that has been ingested or updated many times will accumulate a large number of schemaMetadata aspect versions. When querying with startTime=0 (unbounded full history), every version must be fetched and processed. Contact DataHub Support or your DataHub Cloud administrator to query the number of schemaMetadata rows for the affected URN in the backing SQL store. Datasets with an unusually high version count (hundreds or thousands of rows) are strong candidates for this issue.
Check for deeply nested schema structures.
Schemas with many levels of nested children arrays produce very large serialized payloads per version row. Even a moderate number of versions can become extremely slow to process when each row contains a deeply nested structure. If your dataset schema has 10 or more levels of nesting, consider whether the schema definition can be flattened upstream at the source system or during ingestion transformation.

Avoid unbounded time range queries (startTime=0).

Using startTime=0 instructs the Timeline API to scan the full change history of the asset with no lower time bound. For assets with many versions, this is the primary driver of timeout failures. Where possible, scope your queries to a relevant time window:

# Example: Query only the last 7 days of timeline changes
import time

end_time = int(time.time() * 1000)          # current time in milliseconds
start_time = end_time - (7 * 24 * 3600 * 1000)  # 7 days ago

url = (
    "https://<your-instance>.acryl.io/openapi/v2/timeline/v1/"
    "<encoded-dataset-urn>"
    f"?startTime={start_time}&endTime={end_time}"
    "&categories=TECHNICAL_SCHEMA&categories=TAG&raw=false"
)
print(url)

Implement retry logic with exponential backoff.

As a short-term mitigation while the root cause is being addressed, add retry logic with exponential backoff to your Timeline API client code to handle intermittent timeouts gracefully:

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

session = requests.Session()

retry_strategy = Retry(
    total=3,
    backoff_factor=2,           # waits 2s, 4s, 8s between retries
    status_forcelist=[429, 500, 502, 503, 504],
    allowed_methods=["GET"]
)

adapter = HTTPAdapter(max_retries=retry_strategy)
session.mount("https://", adapter)

response = session.get(
    "https://<your-instance>.acryl.io/openapi/v2/timeline/v1/<encoded-dataset-urn>",
    params={
        "startTime": <start_time_ms>,
        "endTime": <end_time_ms>,
        "categories": ["TECHNICAL_SCHEMA", "TAG"],
        "raw": False
    },
    headers={"Authorization": "Bearer <your-api-token>"},
    timeout=60   # increase client timeout if needed
)

Request schema version cleanup (DataHub Cloud managed instances).
If you are on DataHub Cloud and a specific dataset has accumulated an excessive number of historical schemaMetadata versions that are no longer needed, contact DataHub Support and request a targeted cleanup or archival of old schema versions for the affected URN(s). This is a backend operation performed by the DataHub Cloud engineering team and can significantly reduce Timeline API response times for the affected asset.
Reduce ingestion frequency for high-churn datasets.
If the high version count is being driven by frequent re-ingestion of the same dataset (e.g., running ingestion pipelines multiple times per day), consider reducing ingestion frequency or enabling schema change detection in your ingestion recipe so that a new schemaMetadata version is only written when the schema actually changes:
```
# Example ingestion recipe excerpt — enable stateful ingestion
# to avoid writing redundant schemaMetadata versions
source:
  type: <your-source-type>
  config:
    # ... your source config ...
    stateful_ingestion:
      enabled: true
      remove_stale_metadata: true
```

Additional Notes

This issue is asset-specific and does not indicate a platform-wide outage. Other API endpoints (e.g., /gms/aspects, /api/v2/graphql) are not affected by this root cause and will continue to perform normally. The Timeline API's performance is inherently sensitive to both the number of historical aspect versions stored for a given URN and the payload size of each version; deeply nested schemas compound this effect significantly. Observed response times for severely affected datasets have ranged from 12 seconds at the minimum to over 45 seconds at the 99th percentile, consistently exceeding the typical 30-second client read timeout. Scoping queries to a bounded time window with a realistic startTime value is the single most impactful client-side mitigation available. If you are on DataHub Cloud, platform upgrades and cluster maintenance windows can occasionally cause brief (1–2 minute) connectivity interruptions that are separate from and unrelated to this schema-volume root cause; coordinate with your DataHub team to receive advance notifications of such events.

Issue

Error Messages

You Might Be Asking

Solution

Additional Notes

Related Documentation