dSource Creation (Disk)

Linking a data source will create a dSource object on the engine and allow Delphix to ingest/manage data from this source. The dSource is an object that the Delphix Virtualization Engine uses to create and update virtual copies of your database.

Ingestion of data is done using a staging host. Delphix expects the same version of CockroachDB binaries as the source on this staging host. Delphix will read the existing backup of the CockroachDB database from backup location on Disk (Directory path/ NFS Mountpath) and restore it on the staging host. The location of CockroachDB data file on the staging host is Delphix mounted filesystem which allows Delphix to take snapshots.

Delphix leveraging the built-in cockroach backup and restore procedures. One can use the BACKUP statements to efficiently backup cockroach cluster and data to popular cloud services such as AWS S3, Google Cloud Storage, or NFS, and the RESTORE statement to efficiently restore schema and data as necessary. The user will provide the CockroachDB Backup’s Subdirectory Absolute Path where the backups are available. Delphix finds out the latest full or incremental backup end time and restores the full-cluster as of that time.

Periodic dSource snapsync restores the incremental backups if available in the backup folder provided as input to Delphix. During snapsync, Delphix looks for the backup endtime and a new backup subdirectory found using the SHOW BACKUPS CockroachDB command executed via CRDB Plugin. Delphix action will depend on the latest information pulled from SHOW BACKUPS. In case of no new backups are found, CRDB Plugin will provide an error message on Delphix UI that no new CockroachDB backups are found.

Once the linking is done and dSource snapshot is successful, Delphix does not need old backups. After a dSource has been linked, you can begin to perform dSource operations on it such as enable, disable, delete, provision vdb, and more.

Prerequisites

Install delphix engine compatible with the CRDB plugin version as per support matrix
Install CockroachDB binaries on staging host.
Execute Discovery on staging host.
Create sourceConfig on staging host.

Create Sourceconfig

Login to the Delphix Management application.
Click Manage.
Select Environments.
Select the repository
Click on + icon ( Shown in next image )
Add required details in Add database section.
CockroachDB Staging Cluster Name: Provide CockroachDB Staging Cluster Name (dSource Name)
Click Add button

Create dSource

Steps:

Login to Delphix Management application.
Click Manage > Datasets
Select Add dSource.
In the Add dSource wizard, select the dSource Type (Source Based dSource) for creating CockroachDB staging cluster on the staging host.
In the Add dSource wizard, select the CockroachDB source config you just created on the staging host.
Select the dSource type Disk (AWS_S3 is selected as default) from the drop-down and enter dSource specific parameters for your configuration.
Click Next.
Enter a dSource Name.
Select a Database Group for the dSource.
Click Next.
Verify the Staging environment and ensure Resynchronize dSource option is selected.
Select a SnapSync policy.
Click Next.
Enter any custom pre or post sync logic as Pre-Sync or Post-Sync hook operations.
Click Next.
Review the dSource Configuration and Data Management information, and then click Finish.
The Delphix Engine will initiate two jobs to create the dSource, DB_Link and DB_Sync . You can monitor these jobs by clicking Active Jobs in the top menu bar.
Or you can view the last running jobs by selecting System –> Events Viewer
Once the jobs have completed successfully, the files icon will change to a dSource icon on the Environments –> Databases screen, and the dSource will be added to the list of Datasets under its assigned group.