Area: Ingestion Issues
Sub-Area: Database Pattern Filtering
Issue
When using the CockroachDB source to ingest metadata from multiple databases with database_pattern filtering, the pattern is ignored and only the explicitly specified database is ingested. This occurs because the database field and database_pattern are mutually exclusive in the CockroachDB connector configuration.
Error Messages
database "postgres" does not exist
You Might Be Asking
- Why is my database_pattern being ignored when I have both database and database_pattern configured?
- How do I ingest multiple databases matching a pattern in CockroachDB?
- Why do I get a "postgres" database error when CockroachDB uses "defaultdb"?
Solution
To enable multi-database discovery for CockroachDB, you must remove the database field and use initial_database instead:
- Remove the
databasefield from your configuration - Add
initial_database: defaultdbto your configuration - Keep your
database_patternconfiguration as is
Here's the corrected configuration:
source:
type: cockroachdb
config:
host_port: '.cockroachlabs.cloud:26257'
username:
password: '${COCKROACHDB_PASSWORD}'
initial_database: defaultdb
platform_instance: DEV
env: DEV
database_pattern:
allow:
- '.*_dev'
include_tables: true
include_views: true
profiling:
enabled: false
profile_table_level_only: true
stateful_ingestion:
enabled: true
How this works:
- The connector connects to
initial_database(defaultdb) to enumerate all available databases - It applies the
database_pattern.allowfilter to find matching databases - It creates separate connections and ingests each matching database (e.g., clientdb_dev, firmdb_dev, contactdb_dev)
Additional Notes
The database field takes precedence over database_pattern when both are present. When database is specified, the connector operates in single-database mode and ignores pattern matching entirely. The default value for initial_database is "postgres" (inherited from the PostgreSQL connector), which doesn't exist in CockroachDB environments where "defaultdb" is the default database.
Related Documentation
Tags: cockroachdb, ingestion, database-pattern, multi-database, initial-database, postgres-error, pattern-filtering, database-discovery, configuration, cloud-clusters