Kafka Topic Metadata Ingestion Issues

Kafka topics are not being discovered during ingestion, or schema information from Schema Registry is missing or outdated. This commonly occurs due to incorrect Kafka bootstrap server configuration, Schema Registry connectivity issues, or insufficient ACL permissions.

Common Error Messages:

• Failed to connect to Kafka bootstrap servers
• Schema Registry unreachable
• Not authorized to access topics
• Timeout while fetching topic metadata

You Might Be Asking:

• How do I ingest Kafka topics into DataHub?
• Why isn't schema information showing for my topics?
• How do I handle Kafka ACLs with DataHub?

1. Configure Kafka source with schema registry:

Example for Kafka. This can be adapted for other messaging platforms.

source:
  type: kafka
  config:
    connection:
      bootstrap: "kafka-broker1:9092,kafka-broker2:9092"
      # For SASL authentication
      sasl_mechanism: "PLAIN"
      sasl_username: "${KAFKA_USER}"
      sasl_password: "${KAFKA_PASSWORD}"
      security_protocol: "SASL_SSL"

    # Schema Registry configuration
    schema_registry_url: "https://schema-registry:8081"
    schema_registry_config:
      basic.auth.user.info: "${SR_USER}:${SR_PASSWORD}"

2. For Confluent Cloud:

Example for Confluent Cloud. This can be adapted for other managed Kafka services.

source:
  type: kafka
  config:
    connection:
      bootstrap: "pkc-xxxxx.us-east-1.aws.confluent.cloud:9092"
      sasl_mechanism: "PLAIN"
      security_protocol: "SASL_SSL"
      sasl_username: "${CONFLUENT_API_KEY}"
      sasl_password: "${CONFLUENT_API_SECRET}"

    schema_registry_url: "https://psrc-xxxxx.us-east-1.aws.confluent.cloud"
    schema_registry_config:
      basic.auth.user.info: "${SR_API_KEY}:${SR_API_SECRET}"

3. Grant Kafka ACL permissions:

Example for Kafka. This can be adapted for other messaging platforms.

# Allow DataHub to describe topics
kafka-acls --bootstrap-server localhost:9092 \
  --add --allow-principal User:datahub \
  --operation Describe --topic '*'

# Allow reading consumer groups
kafka-acls --bootstrap-server localhost:9092 \
  --add --allow-principal User:datahub \
  --operation Describe --group '*'

# Allow Schema Registry access
kafka-acls --bootstrap-server localhost:9092 \
  --add --allow-principal User:datahub \
  --operation Describe --cluster

4. Filter topics by pattern:

Example for Kafka. This can be adapted for other messaging platforms.

source:
  config:
    topic_patterns:
      allow: ["^prod\..*", "^analytics\..*"]
      deny: [".*\.test$", ".*\.tmp$"]

5. Extract Kafka Connect lineage:

Example for Kafka Connect. This can be adapted for other streaming platforms.

source:
  type: kafka-connect
  config:
    connect_uri: "http://kafka-connect:8083"
    # This will extract lineage from connectors to topics
    username: "${CONNECT_USER}"
    password: "${CONNECT_PASSWORD}"

Schema Registry integration provides Avro, Protobuf, and JSON schema information for topics. For comprehensive lineage, also ingest from Kafka Connect to see source and sink connections.

• Kafka Source
• Kafka Connect Source