Databricks supported features

This topic lists the supported capabilities for the Hyperscale Databricks connector. Use it to verify what the connector supports before you configure or run a masking job.

Deployment and architecture

The connector is designed to reduce operational overhead and avoid staging infrastructure.

• Closed appliance deployment is available through Azure Marketplace.

• The connector uses a stage-less workflow.

• The connector does not use NFS, S3, or Azure Blob staging.

• The connector provides a simpler deployment process and improved performance because it reads directly from the underlying storage.

• Source files are never copied to the Hyperscale product.

Databricks profiler

The Profiler automatically scans your data to identify sensitive columns and recommend masking algorithms, eliminating the need for manual inventory creation.

It classifies each column against a profile set, which is a collection of rules such as ADDRESS or LAST_NAME, and builds a masking inventory: a list of sensitive columns and the recommended masking algorithm for each. This inventory is saved as a data set and can be used directly in a Hyperscale masking job.

To learn more, see Databricks profiler

Filtering

The connector supports filtering to limit the data that a masking job processes.

• Filters can use the logical Databricks table column name or the physical Parquet column name.

• Masking inventories must use the Parquet column names because masking occurs against the underlying Parquet files.

Partition filtering

Partition filtering lets you mask a subset of data based on partition values.

• Supports Hive-style partitions, for example /region=2024/.

• Supports hash-style partitions, for example /V8/.

Time-based filtering

With time-based filtering you can filter active files by date or timestamp fields.

• Supports filtering active files by date or timestamp fields.

• Requires min and max data statistics to be enabled for the Delta logs.

• If a maximum timestamp does not include a time zone, the connector defaults the time to 00:00:00 UTC for that day. That value does not include that day in the filter.

• To include that day, set the maximum timestamp filter to 23:59:59. Include a timezone, or the connector defaults to UTC if no timezone is specified in the filter. For example:

  Copy

    { 
  
              "filter_type": "TIME_BASED", 
  
              "filter_column": "claims_date", 
  
              "min_date": "2025-01-01T00:00:00+05:30", 
  
              "max_date": "2026-01-01T23:59:59+05:30"  
  
  } 

  The above filtering bracket considers starting from 1st of Jan 2025 till End of the day 1st of Jan 2026.
  
  

• If a timestamp does not include a timezone, the connector assumes UTC when it compares values.

• The connector compares like values to like values. If the filter value uses one timezone and the data uses another timezone, the connector converts both values to UTC for comparison.

• The connector follows ISO-8601 date and timestamp standards.

Time-based filtering requires statistics to be written to the Delta log for the date or timestamp columns used in the filter. This requirement applies to the source Delta table. If these statistics are not enabled or available, the filter may not be able to identify files that match the specified criteria. If statistics collection is not already enabled for these columns, run the following SQL statements before executing a Hyperscale job that uses time-based filtering.

ALTER TABLE
  catalog.schema.table
SET TBLPROPERTIES
  (
    'delta.dataSkippingStatsColumns' = 'order_date, order_timestamp, ship_date, delivery_date'
  )

The preceding query applies only to future writes. To add statistics to the Delta log for existing data, run the following statement:

ANALYZE TABLE catalog.schema.table COMPUTE DELTA STATISTICS;

Materialized views

The connector supports masking for materialized views only in a specific workflow.

• The materialized view must already exist in the target environment.

• The underlying tables that the materialized view uses must be masked through a Hyperscale Databricks job.

• After the underlying tables are masked successfully, Databricks periodic refresh processes update the target materialized views so that they contain the masked data.

• You can also trigger the refresh manually with a SQL command.

• Hyperscale does not support materialized views directly as part of a job. It supports only the scenario described in this section.

Manual refresh command

REFRESH MATERIALIZED VIEW catalog.schema.view_name;

Platform and storage support

The connector supports Azure and AWS-hosted Databricks environments and external storage configurations that store Delta table data outside default Databricks-managed storage.

• Supports Azure-hosted and AWS-hosted Databricks environments.

• Supports masking for Managed Databricks tables backed by ADLS Gen2 (Azure Data Lake Storage) or Amazon S3.

Authentication

Catalog authentication uses OAuth M2M for both Azure-hosted and AWS-hosted Databricks environments. Storage authentication varies by cloud provider.

Operations

The connector supports operational troubleshooting workflows.

Support bundle generation is supported.

DATA_TRANSFER Job type (unmasked)

DATA_TRANSFER is a Databricks job type that copies source Delta data to the target without masking. It uses the same connector, data set, and subset filters, including partition and time-based filters, as a masking job. It also preserves Delta partitioning and Delta-log handling.

The masking step is skipped, and no Compliance Engine is involved. Data is moved using a server-side cloud copy, such as Azure Copy Blob or AWS S3 copy.

Select the job type by setting source_config.job_type to DATA_TRANSFER. The default job type is MASKING. During execution, the MASKING task reports a status of SKIPPED.

Use DATA_TRANSFER when you need the same Hyperscale orchestration and subset filtering but do not need to mask the data. For example, you can use it to move or refresh a subset of tables between environments.

What's next

Review Permissions and access requirements.