Data Product Creation and Management

Organizations want to group related datasets, dashboards, and pipelines into data products but are unsure how to structure and manage them in DataHub. Understanding data product concepts and implementation is key for data mesh architectures.

You Might Be Asking:

• How do I create a data product in DataHub?
• What should be included in a data product?
• How do data products relate to domains?

1. Create a data product via UI:

1. Navigate to Domains section
2. Select the domain for your data product
3. Click "Create Data Product"
4. Enter details:
   - Name and description
   - Owners
   - Related assets (datasets, dashboards, etc.)
5. Save data product

2. Create data product via GraphQL:

mutation createDataProduct {
  createDataProduct(
    input: {
      id: "customer_360"
      name: "Customer 360"
      description: "Complete view of customer data across all systems"
      # Associate with domain
      domain: "urn:li:domain:customer_analytics"

      # Define key properties
      properties: {
        customProperties: {
          "sla": "99.9%",
          "update_frequency": "hourly",
          "owner_team": "customer-analytics"
        }
      }
    }
  ) {
    urn
  }
}

3. Add assets to data product:

mutation addAssetsToDataProduct {
  # Add datasets
  batchSetDataProduct(
    input: {
      dataProductUrn: "urn:li:dataProduct:customer_360"
      resourceUrns: [
        "urn:li:dataset:(urn:li:dataPlatform:snowflake,customers,PROD)",
        "urn:li:dataset:(urn:li:dataPlatform:snowflake,orders,PROD)",
        "urn:li:dashboard:(looker,customer_dashboard)"
      ]
    }
  )
}

4. Structure a complete data product:

# Data Product: Customer 360
Components:
  # Core Datasets
  - Customers Master (Snowflake)
  - Customer Orders (Snowflake)
  - Customer Interactions (BigQuery)

  # Derived Assets
  - Customer 360 View (dbt model)
  - Customer Segments (ML feature table)

  # Consumption Layer
  - Customer Dashboard (Looker)
  - Customer API (REST endpoint)

  # Documentation
  - Data Product README
  - API Documentation
  - SLA Definitions

Ownership:
  - Product Owner: Jane Doe
  - Tech Lead: John Smith
  - Domain: Customer Analytics

Quality:
  - Data Quality Score: 95%
  - SLA: 99.9% uptime
  - Freshness: < 1 hour

5. Query data product information:

query getDataProduct {
  dataProduct(urn: "urn:li:dataProduct:customer_360") {
    name
    description
    domain {
      name
    }
    ownership {
      owners {
        owner {
          ... on CorpUser {
            username
          }
        }
        type
      }
    }
    # Get all assets in the data product
    contains {
      total
      relationships {
        entity {
          ... on Dataset {
            name
            platform {
              name
            }
          }
          ... on Dashboard {
            title
          }
        }
      }
    }
  }
}

6. Implement data product SLAs:

# Add SLA assertions to data product assets
from datahub.emitter.mce_builder import make_dataset_urn

# Define SLA for data product datasets
for dataset in data_product_datasets:
    # Freshness SLA
    create_freshness_assertion(
        dataset_urn=dataset,
        freshness_requirement="1 hour"
    )

    # Quality SLA
    create_quality_assertion(
        dataset_urn=dataset,
        quality_threshold=95.0
    )

    # Availability SLA
    create_availability_assertion(
        dataset_urn=dataset,
        uptime_requirement=99.9
    )

Data Products are a key concept in Data Mesh architecture. They should be self-contained, with clear ownership, documentation, and SLAs. Use domains to organize related data products. Consider adding monitoring and observability for data product health.

• Data Products Guide
• Data Mesh Concepts