Kafka Topic Schema Not Showing in DataHub: Avro Parsing and Inconsistent Schema Handling – DataHub Customer Support Portal

Area: Ingestion Issues

Sub-Area: Kafka Ingestion / Schema Registry / Schema Parsing

Issue

When running Kafka ingestion in DataHub, some topics may appear without a schema, or schema ingestion may fail entirely for a subset of topics. This can occur for several reasons: the topic has no entry in the schema registry, the Avro schema contains deeply recursive type references that exceed Python's recursion limit, Protobuf schemas reference external dependencies (such as Google Well-Known Types) that are not resolvable at parse time, or the topic has no retained data and no live throughput available for sampling. Additionally, when DataHub falls back to sampling topic messages directly (for topics not registered in the schema registry), inconsistent message structures across sampled records will prevent a schema from being derived. This article explains each failure mode and provides diagnostic steps and remediation guidance.

Error Messages

Failed to decode Avro message for topic <topic-name>: RecursionError: maximum recursion depth exceeded in __instancecheck__
Exception while extracting topic <topic-name>: google/type/date.proto:-1:0 error: File not found.
Import "google/type/date.proto" was not found or had errors.
Import "google/type/decimal.proto" was not found or had errors.
".google.type.Date" is not defined.
".google.type.Decimal" is not defined.

You Might Be Asking

Why do some of my Kafka topics appear in DataHub without any schema?
Why does Avro schema ingestion fail with a RecursionError?
Why does Protobuf schema ingestion fail with "File not found" for Google types?
My topic clearly has data — why is DataHub still not showing a schema?
What does "inconsistent schema" mean in the context of Kafka topic sampling?
How does DataHub handle topics that are not registered in the schema registry?

Solution

Work through each failure mode below in order. Identify which applies to your topic(s) and follow the corresponding remediation steps.

Isolate the failing topics for targeted debugging.

Create a minimal ingestion recipe that targets only the problematic topic(s). This shortens the feedback loop and avoids noise from unrelated topics. Your existing recipe is not affected.

source:
  type: kafka
  config:
    connection:
      bootstrap: "<your-kafka-bootstrap-servers>"
      schema_registry_url: "<your-schema-registry-url>"
    consumer_config:
      security.protocol: SSL
      group.id: <your-consumer-group-id>
    topic_patterns:
      allow:
        - "<failing-topic-name>"
    profiling:
      enabled: true
    stateful_ingestion:
      enabled: true

sink:
  type: datahub-rest
  config:
    server: "http://<your-datahub-host>:8080"

Run this recipe and review the logs at debug level to identify the specific error category.

Failure Mode 1 — Avro RecursionError (deeply nested or self-referential schemas).
If your logs contain RecursionError: maximum recursion depth exceeded in __instancecheck__, the Avro schema for that topic contains circular or very deeply nested type references that exceed Python's default recursion limit during schema parsing.
- Verify the schema in your schema registry for the affected topic and check for recursive record definitions or union types that reference themselves.
- Upgrade to the latest available version of the DataHub CLI (acryl-datahub), as fixes for recursive Avro schema handling are included in recent releases.
- If upgrading does not resolve the issue, consider refactoring the Avro schema to break the recursive reference, or contact DataHub support with the schema definition for further analysis.
```
# Upgrade the DataHub CLI
pip install --upgrade acryl-datahub

# Confirm the installed version
datahub version
```
Failure Mode 2 — Protobuf schema with unresolvable Google Well-Known Type imports.
If your logs show errors such as google/type/date.proto: File not found or ".google.type.Date" is not defined, the Protobuf schema for that topic imports Google Well-Known Types or other external .proto files that are not available in the ingestion environment's protobuf include path.
- Ensure that the google-cloud-type or equivalent protobuf descriptor packages are installed in the Python environment running the DataHub CLI.
- Upgrade to the latest DataHub CLI version, which includes expanded support for Google Well-Known Types in schema registry parsing.
- If the external .proto files are custom (not Google standard types), provide them to the ingestion environment via the appropriate protoc include path configuration.
```
# Install Google API common protos which include google/type/*.proto definitions
pip install googleapis-common-protos

# Then re-run ingestion
datahub ingest -c <your-recipe-file>.yml
```
Failure Mode 3 — Topic not in schema registry; sampled data is empty or has inconsistent structure.
For topics that have no schema registry entry, DataHub falls back to sampling messages directly from the topic to infer the schema. This sampling approach has two limitations:
- No retained data / no live throughput: If the topic has no messages retained (due to retention policy) and no active producers, there is nothing to sample. DataHub will not be able to derive or display a schema. Verify whether the topic has any retained messages using your Kafka tooling:
```
# Check the topic's log-end offset to see if there are retained messages
kafka-run-class.sh kafka.tools.GetOffsetShell \
  --broker-list <your-kafka-bootstrap-servers> \
  --topic <your-topic-name> \
  --time -1
```
  If the offsets are 0 or the topic is empty, no schema can be inferred. Consider registering the schema manually in your schema registry instead.
- Inconsistent message structure: If sampled messages do not share a consistent field structure (for example, the schema has evolved across messages without registry enforcement, or different producers write different shapes), DataHub cannot derive a single canonical schema. Confirm with the topic's owning team whether the schema is intentionally variable. If a stable schema exists, register it explicitly in the schema registry so DataHub can retrieve it directly rather than relying on sampling.

Register schemas explicitly in the schema registry as a long-term fix.

The most reliable way to ensure DataHub displays accurate schemas for all topics is to register every topic's schema in the schema registry. This eliminates dependence on sampling and avoids all of the failure modes above.

# Example: Register an Avro schema using the Confluent Schema Registry REST API
curl -X POST \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data '{
    "schema": "<escaped-avro-schema-json>",
    "schemaType": "AVRO"
  }' \
  http://<your-schema-registry-host>:8081/subjects/<your-topic-name>-value/versions

Re-run ingestion and verify schema visibility in DataHub.
After applying the relevant fix, run the ingestion recipe again and navigate to the topic's profile in DataHub to confirm that the schema is now displayed correctly.
```
datahub ingest -c <your-recipe-file>.yml --report-to datahub
```

Additional Notes

Sampling behavior: When a Kafka topic is not present in the schema registry, DataHub's Kafka source samples messages from the topic (including both retained historical messages and any live throughput during the ingestion window) to attempt schema inference. The sampling looks for a consistent structure across all sampled records. Even one inconsistent record among the sample can prevent schema derivation.

Proto topics vs. Avro topics: Protobuf topics that rely on external .proto imports (including Google Well-Known Types such as google.type.Date and google.type.Decimal) require those imports to be resolvable in the ingestion environment. This is distinct from Avro parsing failures. If your topic name ends in _proto or is known to use Protobuf encoding, focus on Failure Mode 2 first.

CLI version: Several of the parsing issues described in this article (recursive Avro schemas, missing Google type support) were addressed in DataHub CLI improvements released in 2024. Always ensure you are running the latest stable version of acryl-datahub before escalating.

Debugging tip: Use a narrowly scoped recipe targeting a single failing topic (as shown in Step 1) with debug logging enabled to get the most precise error output. Running ingestion against all topics simultaneously makes it harder to correlate errors to specific topics.

Issue

Error Messages

You Might Be Asking

Solution

Additional Notes

Related Documentation