Snowflake View Column Definitions Not Appearing After Creation

After creating a new view in Snowflake and running DataHub ingestion, the view appears in DataHub but table and column definitions are missing or incomplete. This issue specifically affects newly created views while existing views in the same database ingest correctly with full metadata.

You Might Be Asking:

• Why is my new Snowflake view missing column definitions?
• Why do some views ingest correctly but not others?
• Do I need special configuration for views versus tables?
• How do I troubleshoot missing view metadata?

Missing view definitions are typically caused by permission issues, caching, or configuration settings related to view metadata extraction.

Troubleshooting Steps:

1. Verify Recipe Configuration:

Ensure the following flags are enabled:

source:
  type: snowflake
  config:
    account_id: ${SNOWFLAKE_ACCOUNT}
    username: ${SNOWFLAKE_USER}
    password: ${SNOWFLAKE_PASSWORD}

    # Required for view metadata
    include_views: true
    include_view_definitions: true
    include_technical_schema: true

    # Optional but recommended
    include_view_lineage: true

    database_pattern:
      allow:
        - "YOUR_DATABASE"

2. Check Snowflake Permissions:

The DataHub role must have sufficient privileges on the view:

-- Grant usage on database and schema
GRANT USAGE ON DATABASE YOUR_DATABASE TO ROLE DATAHUB_ROLE;
GRANT USAGE ON SCHEMA YOUR_DATABASE.YOUR_SCHEMA TO ROLE DATAHUB_ROLE;

-- Grant select on the specific view
GRANT SELECT ON VIEW YOUR_DATABASE.YOUR_SCHEMA.YOUR_VIEW TO ROLE DATAHUB_ROLE;

-- Or grant select on all views in the schema
GRANT SELECT ON ALL VIEWS IN SCHEMA YOUR_DATABASE.YOUR_SCHEMA TO ROLE DATAHUB_ROLE;
GRANT SELECT ON FUTURE VIEWS IN SCHEMA YOUR_DATABASE.YOUR_SCHEMA TO ROLE DATAHUB_ROLE;

-- Verify grants
SHOW GRANTS TO ROLE DATAHUB_ROLE;

3. Verify View Definition in Snowflake:

Check that the view has a valid definition:

-- Get view definition
SELECT GET_DDL('VIEW', 'YOUR_DATABASE.YOUR_SCHEMA.YOUR_VIEW');

-- Verify view columns
DESCRIBE VIEW YOUR_DATABASE.YOUR_SCHEMA.YOUR_VIEW;

-- Test view access
SELECT * FROM YOUR_DATABASE.YOUR_SCHEMA.YOUR_VIEW LIMIT 1;

4. Clear Caches and Re-run Ingestion:

# For CLI-based ingestion
datahub ingest -c your_recipe.yaml

# Force full refresh (avoid stateful ingestion cache)
datahub ingest -c your_recipe.yaml --no-default-report

5. Enable Debug Logging:

Add to your recipe to see detailed errors:

source:
  type: snowflake
  config:
    # ... other config ...

    # Enable debug output
    debug: true

sink:
  type: datahub-rest
  config:
    server: ${DATAHUB_URL}

# Add logging configuration
pipeline_name: snowflake-ingestion
debug_mode: true

Common Causes:

1. Permission Issues: DataHub role lacks SELECT privilege on the view
2. Complex Views: Views with certain SQL features may not extract correctly
3. Materialized Views: Require special handling; ensure include_views: true
4. External Tables: May need separate configuration
5. Cross-Database Views: Require permissions on all referenced databases

Workaround for Complex Views:

If automatic extraction fails, manually add metadata:

from datahub.emitter.mce_builder import make_dataset_urn
from datahub.emitter.rest_emitter import DatahubRestEmitter
from datahub.metadata.schema_classes import SchemaMetadataClass, SchemaFieldClass

# Define schema manually
fields = [
    SchemaFieldClass(
        fieldPath="column1",
        type={"type": {"com.linkedin.schema.StringType": {}}},
        nativeDataType="VARCHAR(100)",
        description="Column 1 description"
    ),
    # ... more fields
]

schema_metadata = SchemaMetadataClass(
    schemaName="your_view",
    platform="urn:li:dataPlatform:snowflake",
    version=0,
    fields=fields,
    hash="",
    platformSchema={"com.linkedin.schema.OtherSchema": {"rawSchema": ""}}
)

# Emit to DataHub
emitter = DatahubRestEmitter(gms_server="https://your-instance.acryl.io")
dataset_urn = make_dataset_urn(
    platform="snowflake",
    name="database.schema.view",
    env="PROD"
)
# ... emit logic

• Newly created views should appear after the next scheduled ingestion
• View definitions can take 1-2 ingestion cycles to fully populate
• Check ingestion logs for specific error messages about the view
• If the issue persists, contact support with view DDL and ingestion logs

• Snowflake Source Configuration
• Schema Metadata

• 4952