Ingestion architecture

The Kubernetes (K8s) driver solution introduces a unique workflow and leverages a purpose-built connector to ingest an arbitrary filesystem as a special K8s unstructured files dSource into the Delphix Continuous Data Engine. Due to the dynamic and ephemeral nature of K8s, Delphix cannot follow the traditional environment discovery of the K8s cluster using a persistent target environment.

The K8s driver ingestion architecture leverages the Staging Push ingestion model, ingesting from an environment that has been discovered using the K8s connector, with filesystem contents that are prepared and controlled by the end-user, not Delphix.

During the provision of a K8s VDB from a K8s dSource snapshot, the solution relies on an environment to serve as a "surrogate". During the provision operation of a K8s VDB, the solution will transiently provision the VDB and enable it on the surrogate, and then immediately disable it. The provision operation will conclude with the K8s VDB in a disabled state, meaning that the dataset has been created in Delphix, but is not currently mounted to any K8s worker node or pod. This surrogate replaces the need for a traditional target environment. 

The following instructions will guide you on the proper steps to set up your architecture, ingest your dataset, and prepare Delphix Continuous Data Engine to provision persistent VDB dataset(s)  into the correct pod which is orchestrated by Kubernetes. If you have not already installed the K8s Driver connector, refer to the Installation and upgradedocumentation.

It is recommended that you have a general understanding of Delphix Continuous Data’s architecture and the Staging Push ingestion strategy before beginning. For more information, refer to the Getting started documentation.

Procedure

Perform the following steps to set up your architecture

1. Identify and/or add a staging (and surrogate) environment: Add a standard Linux host (such as RHEL, CentOS, Ubuntu, etc) as an environment to Delphix Continuous Data Engine. This environment will host the dataset files that you plan to ingest. This environment will act as both the staging and surrogate environment.

2. Add the environment’s IP Address to its NFS address list: In the Delphix Continuous Data Engine UI, navigate to the Environments and add its IP Address to the “NFS address list” configuration property. The environment’s (host’s) IP Address can be identified using tools such as pingor nslookup. This is required so that Delphix Continuous Data Engine can mount the dSource for ingestion, temporarily mount an empty folder for provisioning, and manage the ephemeral nature of Kubernetes.

3. Create a dSource:

   1. Upon adding the staging environment, the Delphix Continous Data K8s connector will discover a dataset home by the name of “Empty vFile Repository” and it will automatically create a default Database (or source config)  for dSource creation.

   2. Select the default Database (or source config) and click “Add dSource”.

   3. Complete the dSource linking wizard to configure your dataset ingestion source.

4. Ingest the dataset: Our solution follows the Staging Push ingestion approach for dSource creation. Staging Push ingestion enables you to push your containerized database volume into the Delphix-provided mount point on your own. For more information about implementation, refer to the Staging Push documentation.

5. Take a snapshot of the dSource: Once you have your data files copied to your dSource mount path, navigate to the dSource in the Delphix Continuous Data Engine UI and take a new dSource snapshot. On snapshot, Delphix Continuous Data will copy the data files from the dSource mount path on the staging environment to the export path on Delphix Continuous Data.

On completion, you have prepared Delphix Continuous Data Engine for containerized virtual database provisioning. You may now follow the remaining implementation steps.