MySQL Metadata Extraction Problems

MySQL database ingestion is failing or showing incomplete metadata, missing foreign key relationships, or not extracting table comments and column descriptions. This typically occurs due to MySQL version compatibility, permissions on information_schema, or character encoding issues.

Common Error Messages:

• Access denied for user to information_schema
• Unknown character set
• Connection timeout to MySQL server

You Might Be Asking:

• What MySQL versions does DataHub support?
• How do I extract foreign key relationships?
• Why are my table descriptions not showing up?

1. Configure MySQL source with proper settings:

Example for MySQL. This can be adapted for other relational databases.

source:
  type: mysql
  config:
    host_port: "mysql.company.com:3306"
    database: "your_database"  # Optional: omit to ingest all databases
    username: "${MYSQL_USER}"
    password: "${MYSQL_PASSWORD}"

    # Character encoding
    options:
      charset: "utf8mb4"

    # Include specific databases
    database_pattern:
      allow: ["production", "analytics"]
      deny: ["mysql", "information_schema", "performance_schema", "sys"]

2. Grant comprehensive permissions:

Example for MySQL. This can be adapted for other relational databases.

-- Create DataHub user
CREATE USER 'datahub_user'@'%' IDENTIFIED BY 'secure_password';

-- Grant read access
GRANT SELECT ON *.* TO 'datahub_user'@'%';

-- Grant access to metadata tables
GRANT SELECT ON information_schema.* TO 'datahub_user'@'%';
GRANT SELECT ON mysql.proc TO 'datahub_user'@'%';

-- For profiling
GRANT SELECT ON performance_schema.* TO 'datahub_user'@'%';

FLUSH PRIVILEGES;

3. Extract foreign key relationships:

Example for MySQL. This can be adapted for other relational databases.

source:
  config:
    # Foreign keys create lineage between tables
    include_table_lineage: true
    include_views: true

4. For AWS RDS MySQL:

Example for AWS RDS MySQL. This can be adapted for other managed database services.

source:
  type: mysql
  config:
    host_port: "database.region.rds.amazonaws.com:3306"
    username: "datahub_user"
    password: "${RDS_PASSWORD}"

    # RDS-specific settings
    options:
      ssl_ca: "/path/to/rds-ca-cert.pem"
      ssl_mode: "VERIFY_IDENTITY"

5. Handle large databases efficiently:

Example for MySQL. This can be adapted for other relational databases.

source:
  config:
    # Use stateful ingestion
    stateful_ingestion:
      enabled: true

    # Disable profiling for large tables
    profiling:
      enabled: false

    # Parallel processing
    parallelism: 4

MySQL 5.7+ is recommended for full feature support. For MariaDB, use the MySQL source but be aware of dialect differences. Table and column comments from MySQL are automatically extracted as descriptions.

• MySQL Source
• AWS RDS MySQL