Link with Selective Database Ingestion (SDI) in DCT

Selective database ingestion (SDI) allows you to ingest specific databases from a MySQL source instead of the entire instance. For more information, read Selective database ingestion.

Prerequisites

Before enabling the selective database ingestion, the following prerequisites must be completed :

  • MySQL 8.0 or higher on the source
  • Source and staging hosts meet the requirements described in Requirements for MySQL source databases and Requirements for MySQL staging and target environments.
  • For SDI, review the starter configuration in $DLPX_TOOLKIT/my.cnf on the staging host. Parameters configured for the environment or operation take precedence over values supplied by the toolkit configuration.
  • If enabling Aurora GTID replication: Aurora parameter group must have gtid_mode=ON, enforce_gtid_consistency=ON, binlog_format=ROW, and binlog_retention_hours ≥ 24. The Aurora cluster must be rebooted after parameter group changes.
  • If enabling Standard VM replication: binary logging must be enabled on the source (log_bin=ON), and the source must have a unique non-zero server-id.

Procedure

  1. Log in to DCT.

  2. Add a source config against the discovered staging or target MySQL environment . For more information, read Create a source configuration for Connectors in DCT,

  3. Go to Data>dSources tab and click +dSource.

  4. On the Select Data Source page, select the MySQL dSource.

  5. Once the MySQL dSource is selected, the following configuration settings will appear. Do the following: 

    For SDI, values configured in the environment or the current operation override corresponding values from $DLPX_TOOLKIT/my.cnf.

    • Set Ingestion Type to Selective Database Ingestion.

    • In the Database List field, enter a comma-separated list of database names to ingest (e.g., db1,db2,db3). Names are case-sensitive and must exist on the source.

      Names can include hyphens and leading or embedded spaces. Names must be 64 characters or fewer. Empty names, trailing spaces, . or .., and names containing /, \, a period, a backtick, a comma, an ASCII control character, or a supplementary Unicode character such as an emoji are rejected before provisioning starts. For the full requirements, see Selective database ingestion.

    • Configure source connection details (host, port, user, password).

    • Configure staging connection details (port, password).

      • (Optional) Enable Replication and select the source type:

        • Aurora — uses GTID-based replication (SOURCE_AUTO_POSITION=1)

        • Standard VM — uses binlog position-based replication

      • (Optional) Select the Enable HashiCorp Vault-Managed Secrets option to retrieve credentials from Vault at runtime instead of storing them in configuration.

        • Vault must be configured at the engine level prior to initiating ingestion workflows. For detailed configuration steps, refer to the Password vault support page.

        • When performing selective database ingestion, if the Enable HashiCorp Vault-Managed Secrets option is selected, the following secrets (key-value pairs) must be present in Vault:

          • Staging Initialization User

          • Staging Initialization Password

          • Source Database User

          • Source Database Password

          • Replication User (required only if “Provide Different Replication User” is enabled for SDI)

          • Replication Password (required only if “Provide Different Replication User” is enabled for SDI)

          • The following fields must be provided for each of the credential types—Staging Initialization Credentials, Source Database User Credentials, and Replication User Credentials (applicable only when Provide Different Replication User is enabled for SDI):

            • Vault ID

            • Engine Name

            • Path

            • Username key

            • Secret key

            • Type of secret to allow from vault(must be set to any)

  6. Click Next.

  7. On the dSource Configuration screen, select the dataset group where the dSource will be placed and click Next.

  8. Select the staging environment and the environment user on the Data Management screen.

    Select the Resynchronize dSource under the Snapshot Parameters section to initialize and refresh the MySQL staging database.

  9. Click Next.

  10. (Optional) On the Policies page, select the SnapSync and Retention policies if required for the dSource and click Next.

  11. On the Hooks page, add any pre-sync and/or post-sync hooks as needed, and then click Next.

  12. On the Summary screen, review the dSource configuration and click Submit.

Database user and role export

When SDI runs, it automatically exports users, roles, and grants scoped to the selected databases from the source and imports them into staging. This preserves access control on the staging database. For managed cloud sources (RDS/Aurora) where system tables are restricted, the connector proceeds with database content only and logs a warning.