Troubleshooting CockroachDB data sources

The CockroachDB connector produces three sets of logs during normal runtime. Refer to the list below to understand what they are and how you can access them.

CockroachDB Plugin logs

  1. The CockroachDB connector is built using the plugin framework. Therefore, it adheres to Delphix Continuous Data’s standard plugin logging behavior. We recommend starting here for all troubleshooting. Refer to the Troubleshooting document more information.

  2. Plugin logs for CockroachDB are found at the below locations.

    Copy
    /{toolkit_directory}/Delphix_COMMON_{appliance_id}/plugin/sc:cockroach_{plugin-id}/CRDB/log_{os_user}/delphixCRDBDebug.log
  3. Cockroach DB cluster node startup logs are now stored under the plugin logs directory and are located in the following folder:

    Copy
    /{toolkit_directory}/Delphix_COMMON_{appliance_id}/plugin/sc:cockroach_{plugin-id}/.delphix/dlpx_<cluster-name_node<node_number>.log 

    For example: 

    /tmp/Delphix_COMMON_8e4b07995438_06675af37201_host/plugin/sc:crdb_ead7fd4c-7109-4c86-82ea-eb167c4cb9d9/.delphix/dlpx_gcp-stage_node0.log 

CockroachDB Cluster logs

  1. CockroachDB can write error messages to log files. The files are named using the following format:

    cockroach.[host].[user].[start timestamp in UTC].[process ID].logCBBACKUPMGR.
  2. To make it easier to watch a log without knowing the full filename, a symlink with the short filename cockroach.log is also created. {delphix_mount_path}/node<number>/logs/cockroach.log

  3. This symlink points to the most recent log. The Cockroach log locations on staging or target host can be found at below locations:

    Copy
    {CockroachDB_install_path} version --build-tag
    For example: 
    /usr/local/bin/cockroach version --build-tag
    v21.2.2

     

  4. Start CockroachDB multi-node cluster.

    Copy
    cockroach start --certs-dir=[path-to-certs-directory] 
    --advertise-addr=<node1 address> 
    --join=<node1 address>,<node2 address>,<node3 address> 
    --cache=.25 --max-sql-memory=.25

     

  5. List CockroachDB certificate and keys.

    Copy
    cockroach cert list --certs-dir=[path-to-certs-directory]
  6. Initialize a Cluster from any CockroachDB node.

    Copy
    cockroach init --certs-dir=[path-to-certs-directory] 
    --host=<address of any node on --join list>
  7. Start interactive CockroachDB SQL shell.

    Copy
    cockroach sql --certs-dir=[path-to-certs-directory] 
    --host=<address of any cluster-node>:<cockroach_db_port>

Use the --certs-dir option with the valid path to the certificate directory of the CockroachDB cluster.

  1. Find full backup subdirectories details.

    Copy
    SHOW BACKUPS IN 's3://{bucket name}/{path}?AWS_ACCESS_KEY_ID={placeholder}&AWS_SECRET_ACCESS_KEY={placeholder}';
  1. Find full and incremental backups in a specific full backup subdirectory details.

    Copy
    SHOW BACKUP FROM LATEST IN 's3://{bucket name}?AWS_ACCESS_KEY_ID={placeholder}&AWS_SECRET_ACCESS_KEY={placeholder}';
  2. Create CockroachDB User to login into CockroachDB console to access secure clusters.

    Copy
    create user dlpxadmin WITH PASSWORD 'dlpxadmin';
  3. Grant access to CockroachDB User to access data.

    Copy
    GRANT SELECT, INSERT ON TABLE vehicles to dlpxadmin;