Iceberg Table Stuck in Soft-Deleted, Non-Discoverable State After Failed Drop Operation – DataHub Customer Support Portal

Area: Ingestion Issues

Sub-Area: Iceberg REST Catalog / Entity Lifecycle Management

Issue

An Iceberg table registered in DataHub can become stuck in a soft-deleted, non-discoverable state following a failed or interrupted drop table operation. This typically occurs when a transient error in the search index is encountered mid-operation, leaving the entity present in the metadata store but invisible to search, lineage graphs, and API queries. Users attempting to read from or write to the table via Spark, StarRocks, or other compute engines will receive "table or view not found" errors even though the URN remains resolvable in DataHub. The DataHub UI displays a prominent warning banner on the dataset page indicating the entity is not discoverable.

Error Messages

This entity is not discoverable via search or lineage graph. Contact your DataHub admin for more information.
pyspark.errors.exceptions.captured.AnalysisException: [TABLE_OR_VIEW_NOT_FOUND] The table or view `<catalog>`.`<schema>`.`<table_name>` cannot be found. Verify the spelling and correctness of the schema and catalog.

You Might Be Asking

Why does my Iceberg table show a "not discoverable" warning in the DataHub UI even though I can see it by URN?
Why are my Spark or StarRocks jobs failing with "table not found" errors for a table that still appears in DataHub?
How does a table end up in a soft-deleted state if I never explicitly deleted it in DataHub?
Is it safe to perform a hard delete on an entity that is soft-deleted and non-discoverable?
Can a drop-and-recreate table pattern in Spark cause a DataHub entity to become corrupted?

Solution

Identify the affected entity URN.
Locate the full URN of the stuck dataset. It will be visible in the DataHub UI URL or in error output from your pipeline. The format for an Iceberg dataset URN is:
```
urn:li:dataset:(urn:li:dataPlatform:iceberg,<catalog>.<table-uuid-or-name>,<ENV>)
```
Confirm the soft-deleted state via the DataHub UI.
Navigate to the entity page in the DataHub UI. If the entity is soft-deleted and non-discoverable, you will see the following warning banner at the top of the page:
```
This entity is not discoverable via search or lineage graph.
Contact your DataHub admin for more information.
```
This confirms the entity exists in the metadata store but has been flagged as removed, preventing it from appearing in search results or lineage graphs.
Attempt re-ingestion (if the table still exists in the underlying catalog).
If the table genuinely still exists in the underlying Iceberg catalog (i.e., it was not intentionally dropped), re-ingesting the table or its parent namespace may restore its discoverable state. Run your standard Iceberg ingestion recipe targeting the affected catalog or schema. However, note that if the root cause is a failed drop operation, the table may no longer exist in the catalog, making re-ingestion ineffective.

Perform a hard delete to resolve the stuck state.

When re-ingestion is not applicable or does not resolve the issue, a hard delete of the entity from DataHub is the recommended resolution. This permanently removes the entity and all its associated metadata aspects from the metadata store, clearing the corrupted state. Use the DataHub REST API to perform the hard delete:

curl -X POST 'https://<your-datahub-instance>/api/v2/entity/batchDelete' \
  -H 'Authorization: Bearer <your-api-token>' \
  -H 'Content-Type: application/json' \
  -d '{
    "urns": [
      "urn:li:dataset:(urn:li:dataPlatform:iceberg,<catalog>.<table-name>,<ENV>)"
    ]
  }'

Alternatively, you can use the DataHub Python SDK:

from datahub.ingestion.graph.client import DatahubClientConfig, DataHubGraph

graph = DataHubGraph(DatahubClientConfig(
    server="https://<your-datahub-instance>",
    token="<your-api-token>"
))

urn = "urn:li:dataset:(urn:li:dataPlatform:iceberg,<catalog>.<table-name>,<ENV>)"
graph.hard_delete_entity(urn=urn)
print(f"Hard deleted: {urn}")

Recreate the table and re-register it in DataHub.
After the hard delete, recreate the table in your Iceberg catalog using your standard workflow (e.g., via Spark or your catalog client), then trigger a DataHub ingestion run to register the new entity. Any manually applied metadata such as tags, descriptions, or documentation that was associated with the old entity will need to be reapplied after re-ingestion.
Avoid patterns that can trigger this state.
If your workflow uses a drop-then-recreate pattern to simulate CREATE OR REPLACE TABLE (which may not be supported in your Iceberg version), be aware that a transient error between the drop and recreate steps can leave DataHub in this inconsistent state. Where possible, prefer atomic operations or idempotent ingestion patterns. Using DROP TABLE IF EXISTS before recreating can also prevent Spark-side errors if the table is already absent:
```
-- Preferred pattern to avoid orphaned metadata states
DROP TABLE IF EXISTS <catalog>.<schema>.<table_name>;
CREATE TABLE <catalog>.<schema>.<table_name> (
  -- column definitions
);
```

Additional Notes

This issue arises specifically when a transient error occurs in the DataHub search index during a drop table operation, leaving the entity in a soft-deleted state without completing the full cleanup process. The entity URN remains resolvable in the metadata store, but the entity is excluded from search and lineage graph results, causing downstream compute engines to report "table not found" errors. A soft delete alone does not corrupt the underlying Iceberg catalog; the issue is isolated to the DataHub metadata layer. Hard deleting the entity from DataHub does not affect the table in the underlying Iceberg catalog. Customers using the Iceberg REST Catalog (IRC) integration should note that IRC operations themselves will not leave entities in a soft-deleted state; soft deletion in this scenario is typically triggered by non-IRC operations (e.g., direct API calls, tag operations, or Spark-driven metadata writes) occurring in conjunction with a transient infrastructure error. After performing a hard delete, all previously associated metadata (tags, glossary terms, documentation, ownership) will be permanently removed and must be reapplied manually or via automation following re-ingestion.

Issue

Error Messages

You Might Be Asking

Solution

Additional Notes

Related Documentation