Column-Level Security and PII Detection

Identifying and tagging columns containing personally identifiable information (PII) or sensitive data manually is time-consuming and error-prone. Organizations need automated PII detection and column-level classification.

You Might Be Asking:

• Can DataHub automatically detect PII columns?
• How do I classify sensitive data columns?
• What PII detection patterns are supported?

1. Enable PII detection in profiling:

Example for Snowflake. This can be adapted for other data sources/connectors.

source:
  type: snowflake
  config:
    profiling:
      enabled: true

      # PII detection
      pii_detection:
        enabled: true

        # Detection methods
        column_name_patterns: true
        sample_data_patterns: true

        # PII types to detect
        types:
          - EMAIL
          - PHONE_NUMBER
          - SSN
          - CREDIT_CARD
          - IP_ADDRESS
          - PERSON_NAME

2. Configure custom PII patterns:

source:
  config:
    profiling:
      pii_detection:
        enabled: true

        # Custom regex patterns
        custom_patterns:
          - name: "EMPLOYEE_ID"
            regex: "^EMP[0-9]{6}$"
            tag: "urn:li:tag:EmployeeID"

          - name: "CUSTOMER_ID"
            regex: "^CUST[0-9]{8}$"
            tag: "urn:li:tag:CustomerID"

        # Column name patterns
        column_name_patterns:
          - pattern: ".*email.*"
            tag: "urn:li:tag:Email"
          - pattern: ".*ssn.*|.*social.*"
            tag: "urn:li:tag:SSN"
          - pattern: ".*phone.*|.*mobile.*"
            tag: "urn:li:tag:Phone"

3. Use transformers for column classification:

transformers:
  - type: "pattern_add_dataset_schema_tags"
    config:
      tag_pattern:
        rules:
          # Tag columns by name pattern
          ".*_email$": ["urn:li:tag:PII", "urn:li:tag:Email"]
          ".*_phone$": ["urn:li:tag:PII", "urn:li:tag:Phone"]
          ".*_ssn$": ["urn:li:tag:PII", "urn:li:tag:SSN"]
          "credit_card.*": ["urn:li:tag:PII", "urn:li:tag:CreditCard"]

      # Apply glossary terms
      term_pattern:
        rules:
          ".*_email$": ["urn:li:glossaryTerm:EmailAddress"]

4. Manually tag sensitive columns:

Example for Snowflake. This can be adapted for other data sources/connectors.

from datahub.emitter.mce_builder import make_dataset_urn, make_schema_field_urn, make_tag_urn
from datahub.metadata.schema_classes import (
    EditableSchemaMetadataClass,
    EditableSchemaFieldInfoClass,
    GlobalTagsClass,
    TagAssociationClass
)

# Tag specific column as PII
dataset_urn = make_dataset_urn("snowflake", "db.schema.customers")
editable_schema = EditableSchemaMetadataClass(
    editableSchemaFieldInfo=[
        EditableSchemaFieldInfoClass(
            fieldPath="email_address",
            description="Customer email address (PII)",
            globalTags=GlobalTagsClass(
                tags=[
                    TagAssociationClass(tag=make_tag_urn("PII")),
                    TagAssociationClass(tag=make_tag_urn("Email"))
                ]
            )
        ),
        EditableSchemaFieldInfoClass(
            fieldPath="phone_number",
            globalTags=GlobalTagsClass(
                tags=[
                    TagAssociationClass(tag=make_tag_urn("PII")),
                    TagAssociationClass(tag=make_tag_urn("Phone"))
                ]
            )
        )
    ]
)

emitter.emit_mcp(
    entity_urn=dataset_urn,
    aspect_name="editableSchemaMetadata",
    aspect=editable_schema
)

5. Create data classification policy:

# Create policy restricting PII column access
mutation createPIIPolicy {
  createPolicy(
    input: {
      type: METADATA
      name: "Restrict PII Column Access"
      description: "Only data governance team can view PII tagged columns"
      state: ACTIVE
      resources: {
        filter: {
          criteria: [
            {
              field: "TAG"
              values: ["urn:li:tag:PII"]
            },
            {
              field: "RESOURCE_TYPE"
              values: ["schemaField"]
            }
          ]
        }
      }
      privileges: ["VIEW_DATASET_PROFILE"]
      actors: {
        groups: ["urn:li:corpGroup:data-governance"]
        allUsers: false
      }
    }
  )
}

6. Query PII columns across catalog:

query findPIIColumns {
  search(
    input: {
      type: DATASET
      query: "*"
      filters: [
        {
          field: "fieldTags"
          values: ["urn:li:tag:PII"]
        }
      ]
    }
  ) {
    searchResults {
      entity {
        ... on Dataset {
          name
          schemaMetadata {
            fields {
              fieldPath
              tags {
                tags {
                  tag {
                    name
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

PII detection using sample data requires profiling to be enabled. Automated detection is not 100% accurate - always review and validate. Consider data masking policies in addition to tagging. Document PII handling procedures.

• PII Detection
• Column-Level Tags