Skip to main content

AI Documentation Generation Fails for Tables Exceeding Column Limit

Angelo Bresci
Updated

Area: Product Issues

Sub-Area: AI Documentation Generation

Issue

When attempting to use the Generate with AI feature on a column description for a table that exceeds the supported column count limit, the operation fails with a generic, unhelpful error message. The error does not indicate that the column limit is the root cause, which can lead users to incorrectly assume a configuration or permissions problem. This behavior affects both individual column-level AI generation and full schema documentation generation, because the system loads the complete schema context for all columns regardless of whether a single column or the entire table is targeted.

Error Messages

  • Failed to infer schema documentation. java.lang.RuntimeException: The AI service encountered an error. Please try again in a few moments. If the issue persists, contact your DataHub administrator.
  • Too many columns for this URN. Select a table with less than 3000 columns. (internal error, not currently surfaced in the UI)

You Might Be Asking

  • Why does "Generate with AI" fail on my table even though my permissions and AI documentation settings are configured correctly?
  • Does the column limit apply even when I'm only trying to generate a description for a single column?
  • Is there a way to override the column limit for AI documentation generation?

Solution

  1. Confirm the column count is the root cause.

    Check the number of columns on the affected table. If the table has more than 3,000 columns, the column limit is the cause of the error — not a misconfiguration of permissions or AI settings. You can verify the column count by navigating to the dataset's Columns tab in DataHub and reviewing the total count displayed.

  2. Understand why the limit applies to single-column generation.

    Even when clicking Generate with AI on a single column, DataHub loads the full schema context for all columns in the table to produce accurate, context-aware descriptions. There is currently no single-column-only generation mode that bypasses this full schema load. As a result, the column limit applies universally at the table level.

  3. Review the column limits by version.

    The maximum supported column count for AI documentation generation has increased across DataHub versions:

    • Prior to v0.3.12: 100 columns
    • v0.3.12: 1,000 columns
    • v0.3.13 and later: 3,000 columns

    If your table exceeds the limit for your current version, AI documentation generation will not be available for that table.

  4. Apply a workaround if possible.

    There is currently no configuration option to raise or override the column limit. Consider one of the following alternatives:

    • If the schema can be logically partitioned, break it into smaller tables, each with fewer than 3,000 columns, and generate AI documentation on those instead.
    • If a filtered view or subset of the table exists with fewer columns, use that as the target for AI documentation generation.
    • For columns on large tables, consider writing descriptions manually or using a DataHub automation that does not rely on the AI documentation generation feature.
  5. Verify other potential causes are not contributing.

    If the table is within the supported column limit but the error still occurs, consult the AI Documentation settings and permissions guide linked below to rule out configuration issues before contacting DataHub Support.

Additional Notes

The generic error message currently displayed in the UI — "The AI service encountered an error. Please try again in a few moments." — does not communicate that the column limit has been exceeded. This is a known UX gap and a product improvement to surface a more descriptive error message (e.g., "Too many columns. Select a table with fewer than 3,000 columns.") has been noted internally. Until that improvement is released, users hitting this limit will see only the generic fallback error. DataHub Cloud manages all AI infrastructure and service limits centrally; customers cannot override the column limit or modify backend behavior. If you upgrade to a newer version of DataHub, check the release notes to confirm whether the column limit has been raised.

Related Documentation

Tags: ai-documentation, generate-with-ai, column-limit, schema-documentation, large-tables, ui-error, ai-docs, product-limitation, snowflake, troubleshooting