Kafka Connect Debezium Ingestion Discovers 0 Tables When Platform Instance Is Configured – DataHub Customer Support Portal

Area: Ingestion Issues

Sub-Area: Kafka Connect / Debezium Connector Table Discovery

Issue

When running Kafka Connect ingestion with a Debezium connector that does not specify a table.include.list (relying instead on automatic table discovery via DataHub's SchemaResolver), the ingestion discovers 0 tables even though the upstream source database has already been ingested into DataHub. This occurs specifically when a platform_instance is configured in both the Kafka Connect recipe and the upstream source ingestion recipe. The SchemaResolver fails to match existing dataset URNs because the platform instance prefix is not correctly stripped during URN lookup, causing the connector to find no tables and emit no lineage.

Error Messages

Debezium connector <connector-name> has no table.include.list config - discovering all tables from database '<database-name>' using SchemaResolver
Discovered 0 tables from database '<database-name>' for platform 'postgres'
No tables discovered from database '<database-name>' for connector '<connector-name>'. Make sure you've ingested postgres datasets for database '<database-name>' into DataHub before running Kafka Connect ingestion.

You Might Be Asking

Why does DataHub report 0 discovered tables even though my upstream Postgres ingestion has already run successfully?
Why does setting platform_instance in both my Kafka Connect and Postgres recipes cause table discovery to fail for Debezium connectors?
Do I need to add table.include.list to my Debezium connector configuration to work around this issue?
Is the "Discovered 0 tables" warning a configuration error or a bug in DataHub?

Root Cause

This is a bug in the Kafka Connect ingestion source. When a platform_instance is configured, the SchemaResolver constructs dataset URNs that include the platform instance prefix (e.g., urn:li:dataset:(urn:li:dataPlatform:postgres,MY_INSTANCE.my-database.public.my_table,PROD)). However, the internal _extract_table_name_from_urn function used during schema resolution did not strip the platform instance prefix before performing table lookups, causing all lookups to fail and resulting in 0 discovered tables.

Solution

Upgrade to the fixed DataHub CLI version. This bug is resolved in DataHub CLI version 1.6.0.13 or later (fix merged to OSS master on 2026-07-09 via GitHub PR #18231). The fix centrally strips the platform_instance prefix from schema-resolver URNs inside _extract_table_name_from_urn, allowing table discovery to work correctly when a platform instance is configured.
For DataHub Cloud users: Override the CLI version in the advanced settings of your Kafka Connect ingestion recipe to 1.6.0.13 or later, then re-run the ingestion. No changes to your Debezium connector or DataHub recipe configuration are required.

Verify your recipe configuration is correct. The following recipe structure is valid and should work as expected after upgrading:

Kafka Connect recipe (source section):

source:
  type: kafka-connect
  config:
    connect_uri: 'http://<your-kafka-connect-host>:8083'
    connector_patterns:
      allow:
        - <your-debezium-connector-name>
    env: PROD
    platform_instance_map:
      kafka: <your-kafka-platform-instance>
      postgres: <your-postgres-platform-instance>
    schema_resolver_expand_patterns: true
    schema_resolver_finegrained_lineage: true
    use_schema_resolver: true

Upstream Postgres recipe (source section):

source:
  type: postgres
  config:
    host_port: '<your-postgres-host>:5432'
    database: <your-database-name>
    username: '${POSTGRES_USERNAME}'
    password: '${POSTGRES_PASSWORD}'
    platform_instance: <your-postgres-platform-instance>
    include_tables: true
    include_views: true
    stateful_ingestion:
      enabled: true

Ensure that the value used for postgres in the Kafka Connect recipe's platform_instance_map matches the platform_instance value set in the upstream Postgres ingestion recipe exactly.

Temporary workaround (if upgrading immediately is not possible): Add an explicit table.include.list to your Debezium connector configuration to enumerate the schemas or tables you want captured. For example:
```
table.include.list=public.*
```
This bypasses the SchemaResolver's automatic discovery and allows the connector to proceed. Remove this workaround once you have upgraded to the fixed CLI version.
Confirm the fix is working by re-running the Kafka Connect ingestion and verifying that:
- The log no longer shows "Discovered 0 tables".
- Lineage between Kafka topics and the upstream database tables is visible in DataHub.

Additional Notes

This bug affects Debezium connectors that omit table.include.list and rely on DataHub's SchemaResolver for automatic table discovery, but only when a platform_instance is configured in the recipe. Connectors that explicitly specify table.include.list are not affected. The upstream source database ingestion must still be run successfully before Kafka Connect ingestion, as the SchemaResolver depends on existing dataset metadata in DataHub to perform table discovery. The fix was introduced in OSS DataHub via GitHub PR #18231 and is available starting with DataHub CLI version 1.6.0.13.