Mainframe data set connector

In addition to databases and files, the Continuous Compliance Engine can process data stored in Mainframe data sets commonly found on the IBM z/OS operating system. For more information on data sets, see this IBM knowledge center article.

Support matrix

The Continuous Compliance Engine requires that data be encoded in EBCDIC rather than something like ASCII or UTF-8. EBCDIC is the encoding traditionally used on Mainframes.

On-The-Fly masking jobs

Continuous Compliance supports On-The-Fly (OTF) masking jobs where the data is read from a source location and written to a different target location. Only certain combinations of connector types are supported for OTF jobs.

OTF jobs with connectors of the same type are supported. For example, masking data from an Oracle source database to an Oracle target database is supported if both are using the built-in Oracle connector. OTF jobs using Extended Connectors are supported if both the source and target are using the same Extended Driver (the same uploaded JDBC driver). The following data sources are supported as source connectors for OTF jobs with delimited file targets.

  • Oracle

  • Db2

  • MS SQL

  • PostgreSQL

  • MySQL / MariaDB

  • SAP ASE (Sybase)

  • Connectors created as Extended Connectors.

For masking flat files (e.g. XML, delimited, etc) in an on-the-fly masking job, it is no longer required to copy or create empty files on the target. If the file name pattern does not match any file on the source, the execution will reported as success, although no file is masked.

No other combinations of connector types are supported. For example, an Oracle source with a PostgreSQL target, or an MS SQL source with a fixed-width file target, are unsupported.

Database to Delimited File jobs

Database to Delimited File Masking OTF jobs with a relational database as a source and a delimited file as a target is supported.

The target files must be created in advance, and their names must match those of the source tables.

Masking of binary data is not supported with this type of OTF job. Whenever binary data is present in the source database and it is not null and empty, it will be converted to Hex strings with prefixes according to the configuration of the application settings. In case of NULL values as part of binary data, it will be written as an empty string to the target file. More details on settings can be found here.

File masking jobs

Understanding in-place file masking

When performing in-place masking (IP) on files, the system uses a temporary hidden file to ensure the integrity and security of the masking process. Here's how the process works:

  • Initial Read and Masking: The system first reads the original file and applies the masking algorithms to its columns based on the inventory configuration.

  • Temporary File Creation: The masked data is then written to an intermediary temporary file with the extension *.msk. This temporary file is created on the target connector location. It is hidden and serves as a secure placeholder for the modified data.

  • Final Replacement: Once the masking process is completed and the data has been safely written to the temporary file, the data in the temporary (*.msk) file is read, and its content is written over the original file. This ensures the original file is not modified until the masking process is entirely successful, preventing data corruption or loss. This also ensures that the user/group permissions stay the same.

Advantages of Using a Temporary File

Using an intermediary temporary file during the masking process offers a safeguard against failures. If any issue arises during the process, the original file remains unchanged and intact. It is also a requirement to keep the user and group permissions the same on the source file.

Understanding on-the-fly file masking

On-the-fly (OTF) file masking is used when data is read from one (source) file and the masked data is written to another (target) file. Here’s how the OTF masking process works:

  • Initial Read and Masking: The system reads the source file and applies the masking algorithms to the data. The masking is performed "on the fly" as the data is being read.

  • Writing the Masked Data: The masked data is directly written to the target file. This process eliminates the need to create an intermediary temporary file.

Advantages of OTF file masking:

OTF file masking is significantly faster than in-place masking because it does not require an intermediary temporary file.