Assertions Module Execution and Data Flow

Users need documentation about DataHub's assertions module to understand where assertions are executed (in the customer's network vs. the DataHub control plane), what type of metadata and data is sent back to the control plane, and the overall architecture of the assertions system for security and compliance reviews.

You Might Be Asking:

• Where are DataHub assertions executed?
• What data is sent to the DataHub control plane?
• Is customer data transmitted to Acryl's infrastructure?
• How do assertions access source databases?
• What are the security implications of using assertions?

DataHub assertions are executed in the customer's network using the Remote Executor, with only metadata and test results sent to the DataHub control plane.

Assertions Execution Architecture:

Execution Location:

1. In-Network Execution (Recommended):

   • Assertions run via DataHub Remote Executor
   • Deployed in customer's AWS/Azure/GCP environment
   • Direct access to data sources
   • No data leaves customer network

2. DataHub Cloud Execution (Limited):

   • For cloud-accessible data sources
   • Requires network connectivity from DataHub Cloud
   • Uses customer-provided credentials

Data Flow:

┌─────────────────────────────────────┐
│     Customer Network                │
│                                     │
│  ┌──────────────┐                  │
│  │  Data Source │                  │
│  │  (Snowflake, │                  │
│  │   Redshift,  │                  │
│  │   BigQuery)  │                  │
│  └───────┬──────┘                  │
│          │                          │
│          │ Query                    │
│          ▼                          │
│  ┌──────────────┐                  │
│  │   Remote     │                  │
│  │  Executor    │                  │
│  │             │                  │
│  │  • Runs SQL │                  │
│  │  • Evaluates│                  │
│  │  • Computes │                  │
│  └───────┬──────┘                  │
│          │                          │
│          │ Metadata Only            │
│          │ (Pass/Fail, Metrics)     │
└──────────┼──────────────────────────┘
           │
           ▼
   ┌──────────────────┐
   │ DataHub Control  │
   │     Plane        │
   │  • Stores results│
   │  • Sends alerts  │
   │  • Shows UI      │
   └──────────────────┘

What Data is Sent to the Control Plane:

✅ Sent to DataHub: - Assertion pass/fail status - Execution timestamp - Aggregate metrics (row counts, null counts, etc.) - Error messages (if assertion fails) - Assertion configuration - Dataset URN and metadata

❌ NOT Sent to DataHub: - Raw customer data - Individual row values - Query results - PII or sensitive data - Actual column values

Assertion Example:

-- Assertion SQL (runs in customer network)
SELECT COUNT(*) as row_count
FROM customers
WHERE created_date = CURRENT_DATE;

-- If row_count 

Configuration for Remote Executor:

# Remote Executor Configuration
executor:
  type: datahub-remote
  config:
    # Executor runs in customer VPC
    executor_id: "prod-assertions-executor"

    # Connection to data source (stays in customer network)
    connection:
      type: snowflake
      account: customer_account
      username: ${SNOWFLAKE_USER}
      password: ${SNOWFLAKE_PASSWORD}
      # Credentials never leave customer network

    # Only assertion results sent to control plane
    datahub_api:
      server: "https://customer.acryl.io"
      token: ${DATAHUB_TOKEN}

Types of Assertions:

1. Freshness Assertions:

type: FRESHNESS
schedule: "0 9 * * *"  # Daily at 9 AM
parameters:
  max_lag_minutes: 60

2. Volume Assertions:

type: VOLUME
parameters:
  min_rows: 1000
  max_rows: 100000

3. Column Assertions:

type: FIELD
parameters:
  field: email
  operator: IS_NOT_NULL
  threshold: 0.99  # 99% non-null

4. Custom SQL Assertions:

type: SQL
parameters:
  statement: |
    SELECT COUNT(*) FROM orders
    WHERE total_amount <0 0="0" equal_to="EQUAL_TO">

Security Considerations:

1. Credentials: Stored and used only within customer network
2. Network Isolation: Remote Executor never exposes data sources externally
3. Minimal Data Transfer: Only metadata crosses network boundary
4. Audit Trail: All assertion executions logged
5. Encryption: All communication uses TLS/HTTPS

Deployment Options:

AWS ECS (Fargate or EC2):

# Using Terraform module
module "remote_executor" {
  source = "github.com/acryldata/datahub-terraform-modules//remote-ingestion-executor"

  cluster_name = "datahub-assertions-executor"
  vpc_id       = var.vpc_id
  subnet_ids   = var.private_subnet_ids

  # Assertions capability
  enable_assertions = true
}

Kubernetes:

helm install datahub-executor datahub/datahub-remote-executor \
  --set executorId=assertions-executor \
  --set datahub.url=https://customer.acryl.io

Monitoring Assertion Execution:

Use DataHub UI to monitor: - Assertion pass/fail history - Execution duration - Alert notifications - Trend analysis

• Remote Executor is required for production use of assertions
• Assertions scale with executor resources
• Multiple executors can be deployed for redundancy
• Assertions support all major data warehouses and databases
• For detailed security architecture, request a security whitepaper from Acryl support

• Assertions Overview
• Remote Executor
• Data Quality

• 5336