Data Lineage in DataHub - Presentation Outline (transcript below)

I. Use Case Introduction

• Scenario: Sarah from Business Intelligence team needs to document data sources
• Task: Understand exactly where order entry dashboard data comes from
• Context: Preparing for upcoming audit
• Challenge: Manual tracing would take hours
• Solution: DataHub automatically maps complete data lineage

II. Dashboard Overview (Level 1)

• Asset: Order Entry Dashboard
• Contains: Four key visualizations
  1. Popular Products
  2. Promotions
  3. Order Mode
  4. Orders by Day

III. First Level Lineage (Level 2)

• Flow: Dashboard data flows into Order Details
• Observation: All information feeds into single destination

IV. Table Level Lineage (Level 3)

• Destination: Single Snowflake table - Order Details table
• Composition: Data from multiple upstream sources consolidated

V. Upstream Table Structure (Level 4)

• Count: 11 upstream tables
• Platform: All tables located in Snowflake
• Relationship: Multiple sources feeding into single table

VI. Complete Data Journey (Level 5 - End-to-End)

Source to Destination Flow:

1. Source: Postgres database (source system)
2. First transformation: Spark jobs (data processing)
3. Intermediate storage: S3 (data lake)
4. Second transformation: Additional Spark jobs
5. Data warehouse: Snowflake (destination)
6. Final transformation: DBT model creates view in Snowflake

VII. Column-Level Lineage

• Granularity: Trace lineage at column level
• Method: Click on individual columns
• Result: Immediate visibility into data origins for specific fields
• Benefit: Detailed documentation capabilities

VIII. Key Benefits

• Speed: Instant visibility vs. hours of manual work
• Completeness: Automatic mapping of entire data flow
• Ease of use: Quick navigation through lineage
• Documentation: Easy to understand and document for audits
• Granularity: Asset-level and column-level traceability
• Accessibility: Business users can self-serve lineage information

IX. Summary

• Problem solved: Sarah can quickly document complete data flow for audit
• From: Source Postgres database
• Through: Multiple transformation layers (Spark, S3, DBT)
• To: Final dashboard visualizations
• Detail level: From high-level dashboard down to individual columns
• Result: Comprehensive, auditable data lineage documentation

=== TIMESTAMPED TRANSCRIPT ===

[0.00s - 9.64s]: Sarah from the Business Intelligence team needs to understand exactly where the data in our
[9.64s - 12.12s]: order entry dashboard comes from.
[12.12s - 15.56s]: She's been asked to document our data sources for an upcoming audit.
[15.56s - 19.20s]: Let me show you how a data hub can automatically map and complete the data lineage for our
[19.20s - 20.84s]: order entry dashboard.
[20.84s - 25.16s]: Instead of Sarah spending hours manually tracing through the systems, she can see everything
[25.16s - 26.16s]: instantly.
[26.16s - 32.56s]: At the first level, you can see this is a dashboard contains four key visualizations,
[32.56s - 39.28s]: popular products, promotions, order mode, orders by day.
[39.28s - 44.20s]: The next level, we can see it flows into the order details.
[44.20s - 49.48s]: Look, you're getting all the information fed into a single snowflake table, the order
[49.48s - 51.48s]: details table.
[51.48s - 56.84s]: This table is comprised of 11 upstream tables all in snowflake.
[56.84s - 61.92s]: The data movement is coming from a source-stimped postgres flowing through Spark jobs, which
[61.92s - 68.20s]: then go to S3 and again through more Spark jobs into snowflake, which then use a DBT model
[68.20s - 72.20s]: to build out our view inside of snowflake.
[72.20s - 77.20s]: This gives her a quick, easy way to find out and understand and document even at a
[77.20s - 82.60s]: columnable where each one of these assets is driving their data from.
[82.60s - 87.20s]: Quickly by clicking on the columns, she can see that information immediately.