Anomaly Detection Model Feedback: True Positive Signaling in DataHub Observe – DataHub Customer Support Portal

Area: Observability Issues

Sub-Area: Smart Assertions / Anomaly Detection Feedback

Issue

When a Smart Assertion (AI-powered volume or anomaly assertion) correctly fires on a real data incident — such as a legitimate large-volume spike on an append-only table — users have no explicit way to signal to the anomaly detection model that the alert was a true positive. The two available feedback options are both unsuitable for this scenario: "Mark as Normal" incorrectly includes the anomalous data point in the model's training baseline and suppresses future alerts for similar events, while "Train as New Normal" wipes the historical training window and temporarily disables monitoring for days until enough new data accumulates. Additionally, a confirmed bug affecting volume assertions on tables with row counts greater than 15 million caused anomalous data points to be incorrectly included in the training set, causing the model to stop alerting on subsequent similar anomalies. This bug was resolved in DataHub Cloud v2.0.1.

Error Messages

No explicit "true positive" feedback option available in the Smart Assertion UI for already-fired anomaly alerts
Model stops alerting after a confirmed anomaly — anomalous data point incorrectly absorbed into training baseline (affects volume assertions on tables with >15M rows, fixed in v2.0.1)

You Might Be Asking

How do I tell DataHub that a fired assertion was a real incident so it keeps alerting me for similar events in the future?
Why did the AI anomaly detection model stop alerting after a confirmed true incident?
Does clicking "Mark as Normal" on a correctly fired assertion prevent future alerts?
What is the difference between "Mark as Normal" and "Train as New Normal" for Smart Assertions?
Why is my assertion in a training/warm-up state for multiple days after I clicked "Train as New Normal"?
No incident was created when my Smart Assertion failed — why?

Solution

Understanding the Feedback Options

The two feedback buttons on a fired Smart Assertion serve distinct and specific purposes:

Mark as Normal: Sets the data point's review state to Rejected. This tells the model the event was not anomalous, includes it in the training baseline, and suppresses future alerts for similar patterns. Do not use this for true incidents.
Train as New Normal: Adds an exclusion window that discards all historical training data prior to that timestamp. The model retrains from scratch using only post-incident data. Due to the 15-day minimum lookback requirement, monitoring is effectively paused until sufficient new data accumulates. This is not a true-positive signal — it resets the baseline entirely.

Immediate Workaround: Do Nothing

If an assertion fired correctly on a real anomaly, the recommended immediate workaround is to dismiss or close the alert notification without clicking either feedback button. The model's design treats all detected anomalies as assumed true positives unless explicitly told otherwise — fired anomalies are excluded from the training data by default. Taking no feedback action preserves the model's sensitivity and it will continue to alert on future similar events.

When the assertion alert fires and you confirm it was a real incident, close or dismiss the alert UI without selecting "Mark as Normal" or "Train as New Normal".
The anomalous data point will remain excluded from the model's training baseline.
The model will continue to alert on future events of a similar pattern.

If the Model Has Already Stopped Alerting (Bug: v2.0.1 Fix)

A confirmed bug caused volume assertions on tables with row counts greater than 15 million to incorrectly include anomalous data points in the training set — even without any user feedback action — causing the model to treat those anomalies as normal and stop alerting. This was fixed in DataHub Cloud v2.0.1. The fix applies retroactively, so previously affected anomalies are excluded from the training data going forward without any manual action required.

Verify your DataHub Cloud instance is running v2.0.1 or later. Contact DataHub Support if you are unsure of your current version.
After upgrading to v2.0.1, the model will retroactively exclude previously absorbed anomalies from training data.
If the model is still not alerting after the upgrade, try adjusting the assertion sensitivity via the Tune Predictions menu on the affected assertion.
Alternatively, if the model has already trained on the incorrect data, you can mark any missed anomaly (a data point that should have fired but did not) as an anomaly using the "Mark as Anomaly" option on the historical data point. This helps the algorithm recognize its baseline error.

Tuning Sensitivity for Append-Only Tables

Navigate to the affected dataset in DataHub.
Open the Validation tab and select Assertions.
Click on the relevant Smart Assertion and open the Tune Predictions menu.
Increase the sensitivity setting so that volume spikes consistent with your known incidents are more likely to trigger alerts.

Configuring Auto-Raise Incident for Assertion Failures

Incidents are not automatically created for every assertion failure unless explicitly configured. If no incident is being recorded when your assertion fires, follow these steps:

Navigate to the affected dataset and open the Validation > Assertions tab.
Locate the relevant assertion monitor and open its Settings tab.
Enable the Auto-raise Incident toggle.
Save the configuration. Future assertion failures will now automatically create an incident record.

Integrating Assertion Failures with External Incident Management Platforms

If you use an external incident management platform (such as Jira Service Management or PagerDuty) and prefer not to use DataHub's native incident creation, you can build a custom event listener using the DataHub Actions framework:

Set up a DataHub Actions pipeline to listen for assertion failure events. Refer to the documentation links in the Related Documentation section below.
Configure your action to forward the assertion failure payload to your external incident management tool's API.

Example Actions pipeline configuration structure:

name: assertion-failure-to-incident-platform
source:
  type: datahub-cloud-event-source
  config:
    event_types:
      - AssertionRunEvent
filter:
  event_type: AssertionRunEvent
  event:
    result:
      type: FAILURE
action:
  type: <your-custom-action-type>
  config:
    # Configure your external platform integration here
    webhook_url: <your-incident-platform-webhook-url>

Deploy the Actions pipeline in your environment following the DataHub Actions setup guide.

Additional Notes

Feature Gap — Explicit True Positive Acknowledgment: As of DataHub Cloud v2.0.1, there is no dedicated UI option to explicitly confirm a fired assertion as a "true positive" while simultaneously preserving the model's alerting sensitivity. The backend data model supports a CONFIRMED anomaly review state, but this is not yet surfaced in the UI for already-fired assertions. This has been noted as a product gap and is under consideration for a future release. The recommended workaround remains taking no feedback action on correctly fired alerts.

Bug Fix Scope (v2.0.1): The anomaly metric precision bug (OBS-2028) specifically affected volume assertions on tables with row counts greater than 15 million. Tables below this threshold were not affected by this particular bug. The fix is SaaS-only and was released on 2026-07-01 in DataHub Cloud v2.0.1.

Smart Assertions Beta Status: AI-powered Smart Assertions are currently in Beta. Behavior and available feedback options may change as the feature matures toward general availability.

Training Warm-Up Period: After clicking "Train as New Normal," the assertion requires a minimum of 15 days of data before the model can reliably resume alerting. Avoid using this option unless you intentionally want to reset the entire training baseline.