Databricks connector overview

Use the Hyperscale Databricks connector to mask data in Azure Databricks environments. The connector uses Unity Catalog APIs for metadata processing and reads directly from the underlying storage accounts. Masking uses Delta logs to identify active Parquet files from source and uploads masked output to the target storage accounts. At the end of a masking job, Hyperscale updates the Delta log of the masked Delta table so that the changes are reflected in the Databricks metadata layer.

Prerequisites

Before you configure the connector, make sure the source and target environments, permissions, and engine version meet the requirements for Databricks masking.

  • Use Azure-hosted Databricks environments for the source and target.

  • Only Databricks managed tables created on Azure Data Lake Storage Gen2 (ADLS Gen2) are supported.

  • Make sure the tables you want to mask already exist in the target environment.

  • Grant the required Unity Catalog privileges to the source and target service principals.

  • Assign the required Azure storage roles to the source and target storage scopes.

  • Use Compliance Engine version 2026.4.0 or later if you configure storage_auth_type to use a service principal.

Limitations

Review the current limitations before you design a masking workflow for Databricks. These limitations affect supported platforms, storage, table formats, and filtering behavior.

  • For this 2026.4 release only a single Hyperscale Databricks job (with one or more tables) can run at a time. Concurrent or parallel jobs can encounter a race condition that may cause Delta log updates to fail for one of the jobs.

  • The Databricks load-service rejects legitimate Hive-style partitioned file paths whenever a partition column's value contains a space or any other character that Delta Lake percent-encodes in its file URIs.

  • AWS-hosted and GCP-hosted Databricks environments are not currently supported.

  • Only Delta tables backed by Parquet files are supported. Other Databricks table types, such as Iceberg, are not supported.

  • Managed Delta tables that use default Databricks storage are not supported for masking.

  • Kubernetes deployment is not supported.

  • Only partition filtering or time-based filtering is supported.

  • Target tables must already exist. The Databricks connector does not create tables.

  • Service principal support for storage_auth_type requires Compliance Engine version 2026.4.0 or later.

  • Only Service principals with secrets are supported for storage_auth_types. It does not yet support Service Principals with certificates.

  • Liquid clustering partitions are not supported.

  • Databricks deletion vectors are not supported.

    • Note: Masking still occurs when the target contains Databricks deletion vectors; however, the target environment does not contain deletion vectors after masking, and additional masked rows can appear in the target Databricks environment.

  • The Databricks connector does not currently support tables with the table property delta.columnMapping.mode set to name or id.

What's next