Skip to main content

CockroachDB Multi-Database Pattern Ingestion Not Working

Angelo Bresci
Updated

Area: Ingestion Issues

Sub-Area: Database Pattern Filtering

Issue

When using the CockroachDB source to ingest metadata from multiple databases with database_pattern filtering, the pattern is ignored and only the explicitly specified database is ingested. This occurs because the database field and database_pattern are mutually exclusive in the CockroachDB connector configuration.

Error Messages

  • database "postgres" does not exist

You Might Be Asking

  • Why is my database_pattern being ignored when I have both database and database_pattern configured?
  • How do I ingest multiple databases matching a pattern in CockroachDB?
  • Why do I get a "postgres" database error when CockroachDB uses "defaultdb"?

Solution

To enable multi-database discovery for CockroachDB, you must remove the database field and use initial_database instead:

  1. Remove the database field from your configuration
  2. Add initial_database: defaultdb to your configuration
  3. Keep your database_pattern configuration as is

Here's the corrected configuration:

source:
  type: cockroachdb
  config:
    host_port: '.cockroachlabs.cloud:26257'
    username: 
    password: '${COCKROACHDB_PASSWORD}'
    initial_database: defaultdb
    platform_instance: DEV
    env: DEV
    database_pattern:
      allow:
        - '.*_dev'
    include_tables: true
    include_views: true
    profiling:
      enabled: false
      profile_table_level_only: true
    stateful_ingestion:
      enabled: true

How this works:

  • The connector connects to initial_database (defaultdb) to enumerate all available databases
  • It applies the database_pattern.allow filter to find matching databases
  • It creates separate connections and ingests each matching database (e.g., clientdb_dev, firmdb_dev, contactdb_dev)

Additional Notes

The database field takes precedence over database_pattern when both are present. When database is specified, the connector operates in single-database mode and ignores pattern matching entirely. The default value for initial_database is "postgres" (inherited from the PostgreSQL connector), which doesn't exist in CockroachDB environments where "defaultdb" is the default database.

Related Documentation

Tags: cockroachdb, ingestion, database-pattern, multi-database, initial-database, postgres-error, pattern-filtering, database-discovery, configuration, cloud-clusters