Managing datasetProfile Timeseries Aspect Retention and Bulk Deletion in DataHub – DataHub Customer Support Portal

Area: Observability Issues

Sub-Area: Timeseries Aspect Storage and Lifecycle Management

Issue

When dataset profiling is enabled in DataHub, the datasetProfile timeseries aspect accumulates one snapshot per ingestion run with no automatic expiration. On daily-scheduled sources, this results in unbounded growth over time. Operators need to understand the underlying deletion behavior, whether automatic retention policies can be configured, and how to perform bulk cleanup across large estates without enumerating individual entity URNs.

You Might Be Asking

Does datahub delete by-filter --aspect datasetProfile physically remove data or just hide it from queries?
Is there a way to configure automatic TTL or retention for timeseries aspects like datasetProfile?
Do timeseries aspects accumulate indefinitely by default, or is there a built-in cleanup mechanism?
Can I bulk-delete datasetProfile records across all datasets on a platform without specifying individual URNs?
How do I check the current size of timeseries aspect indices in my DataHub instance?

Solution

1. Understanding Deletion Behavior for Timeseries Aspects

Deleting a timeseries aspect is a hard delete — records are physically removed from storage, not soft-deleted or tombstoned. The CLI command datahub delete by-filter --aspect datasetProfile internally calls hard_delete_timeseries_aspect(), which issues an Elasticsearch deleteByQuery request against the timeseries index. There is no soft-delete path for timeseries aspects.

Query visibility: Deleted records stop appearing in queries immediately.
Disk reclamation: Disk space is reclaimed asynchronously in the background as Elasticsearch performs segment merges. The data is logically gone from the index immediately, but physical disk reclamation occurs later.

This is distinct from entity-level soft deletes, which hide entities from the UI but retain them in storage. For timeseries aspects, the delete is permanent.

2. Automatic Retention Configuration

There is currently no built-in automatic retention mechanism for timeseries aspects. DataHub's retention system (EbeanRetentionService / retention.yaml) only applies to versioned aspects stored in the relational database (MySQL or PostgreSQL). Timeseries aspects such as datasetProfile live in a separate Elasticsearch index that has no TTL, no index lifecycle policy (ILM), and no scheduled cleanup. By default, all historical profile snapshots accumulate indefinitely.

The recommended workaround is to implement a scheduled purge job using the DataHub CLI.

3. Performing Bulk Deletion Across Many Entities

The datahub delete by-filter command does not require per-URN enumeration. It supports broad filter expressions that can target an entire platform, environment, or entity type in a single pass. Example filters include:

--platform bigquery — all entities on a given platform
--env PROD — all entities in a given environment
--entity-type dataset — all datasets
--start-time / --end-time — time range for the timeseries records to remove
--query <elasticsearch-query-string> — advanced free-form filtering

These flags can be combined. For example, to hard-delete all BigQuery dataset profiles ingested before a given date:

datahub delete by-filter \
  --aspect datasetProfile \
  --platform bigquery \
  --env PROD \
  --entity-type dataset \
  --start-time "2024-01-01" \
  --end-time "2024-09-01"

Note on --recursive container filtering: In DataHub CLI version 1.6.0.16 and earlier, --container is not a supported flag on delete by-filter. To target all datasets within a container recursively, use --urn <container-urn> --recursive instead. Verify the available flags for your installed version with:

datahub delete by-filter --help

4. Building a Scheduled Purge

Since there is no automatic retention, the recommended operational pattern is a scheduled purge script. A simple example that retains the last 90 days of profiles:

#!/bin/bash
# Retain the last 90 days of datasetProfile records.
# Adjust --platform, --env, and the end-time date as appropriate.

CUTOFF_DATE=$(date -d "90 days ago" +%Y-%m-%d)

datahub delete by-filter \
  --aspect datasetProfile \
  --platform <your-platform> \
  --env PROD \
  --entity-type dataset \
  --end-time "${CUTOFF_DATE}"

Schedule this script via cron, Airflow, or any job scheduler on your chosen cadence (e.g., weekly).

5. Measuring Current Index Size

To understand the current footprint of timeseries aspect indices before deciding on a retention window, use the following DataHub OpenAPI endpoint:

GET /openapi/operations/elasticSearch/getIndexSizes

This endpoint returns size information for all Elasticsearch indices managed by DataHub, including the datasetProfile timeseries index. You can call it directly against your DataHub instance:

curl -s \
  -H "Authorization: Bearer <your-token>" \
  "https://<your-instance>.datahubproject.io/openapi/operations/elasticSearch/getIndexSizes" \
  | jq .

Additional Notes

The behavior described here applies to all timeseries aspects, not only datasetProfile. Other timeseries aspects (e.g., datasetUsageStatistics, operationEvent) are subject to the same unbounded accumulation.
DataHub's retention.yaml configuration and EbeanRetentionService are explicitly scoped to versioned aspects and have no effect on Elasticsearch-backed timeseries data.
Disk reclamation after a hard delete is asynchronous and depends on Elasticsearch segment merge scheduling. Immediately after deletion, the index size reported by getIndexSizes may not yet reflect the full space savings.
Configurable timeseries TTL is a known platform gap. Check the DataHub changelog and release notes for updates on native retention support for timeseries aspects.
Always run datahub delete by-filter with --dry-run first to preview the scope of deletion before committing.

datahub delete by-filter \
  --aspect datasetProfile \
  --platform <your-platform> \
  --end-time "2024-09-01" \
  --dry-run