Databricks supported features

This topic lists the supported capabilities for the Hyperscale Databricks connector. Use it to verify what the connector supports before you configure or run a masking job.

Deployment and architecture

The connector is designed to reduce operational overhead and avoid staging infrastructure.

  • Closed appliance deployment is available through Azure Marketplace.

  • The connector uses a stage-less workflow.

  • The connector does not use NFS, S3, or Azure Blob staging.

  • The connector provides a simpler deployment process and improved performance because it reads directly from the underlying storage.

  • Source files are never copied to the Hyperscale product.

Databricks profiler

The Profiler automatically scans your data to identify sensitive columns and recommend masking algorithms, eliminating the need for manual inventory creation.

It classifies each column against a profile set, which is a collection of rules such as ADDRESS or LAST_NAME, and builds a masking inventory: a list of sensitive columns and the recommended masking algorithm for each. This inventory is saved as a data set and can be used directly in a Hyperscale masking job.

To learn more, see Databricks profiler

Filtering

The connector supports filtering to limit the data that a masking job processes.

  • Filters can use the logical Databricks table column name or the physical Parquet column name.

  • Masking inventories must use the Parquet column names because masking occurs against the underlying Parquet files.

Partition filtering

Partition filtering lets you mask a subset of data based on partition values.

  • Supports Hive-style partitions, for example /region=2024/.

  • Supports hash-style partitions, for example /V8/.

Time-based filtering

With time-based filtering you can filter active files by date or timestamp fields.

  • Supports filtering active files by date or timestamp fields.

  • Requires min and max data statistics to be enabled for the Delta logs.

  • If a maximum timestamp does not include a time zone, the connector defaults the time to 00:00:00 UTC for that day. That value does not include that day in the filter.

  • To include that day, set the maximum timestamp filter to 23:59:59. Include a timezone, or the connector defaults to UTC if no timezone is specified in the filter. For example:

    Copy 
      { 

                "filter_type": "TIME_BASED", 

                "filter_column": "claims_date", 

                "min_date": "2025-01-01T00:00:00+05:30", 

                "max_date": "2026-01-01T23:59:59+05:30"  

    } 

    The above filtering bracket considers starting from 1st of Jan 2025 till End of the day 1st of Jan 2026.

  • If a timestamp does not include a timezone, the connector assumes UTC when it compares values.

  • The connector compares like values to like values. If the filter value uses one timezone and the data uses another timezone, the connector converts both values to UTC for comparison.

  • The connector follows ISO-8601 date and timestamp standards.

Time-based filtering requires statistics to be written to the Delta log for the date or timestamp columns used in the filter. This requirement applies to the source Delta table. If these statistics are not enabled or available, the filter may not be able to identify files that match the specified criteria. If statistics collection is not already enabled for these columns, run the following SQL statements before executing a Hyperscale job that uses time-based filtering.

Copy 
ALTER TABLE
  catalog.schema.table
SET TBLPROPERTIES
  (
    'delta.dataSkippingStatsColumns' = 'order_date, order_timestamp, ship_date, delivery_date'
  )

The preceding query applies only to future writes. To add statistics to the Delta log for existing data, run the following statement:

ANALYZE TABLE catalog.schema.table COMPUTE DELTA STATISTICS;

Materialized views

The connector supports masking for materialized views only in a specific workflow.

  • The materialized view must already exist in the target environment.

  • The underlying tables that the materialized view uses must be masked through a Hyperscale Databricks job.

  • After the underlying tables are masked successfully, Databricks periodic refresh processes update the target materialized views so that they contain the masked data.

  • You can also trigger the refresh manually with a SQL command.

  • Hyperscale does not support materialized views directly as part of a job. It supports only the scenario described in this section.

Manual refresh command

REFRESH MATERIALIZED VIEW catalog.schema.view_name;

Platform and storage support

The connector supports Azure-hosted Databricks environments and external storage configurations that store Delta table data outside default Databricks-managed storage.

  • Supports Azure-hosted Databricks environments.

  • Supports masking for Managed Databricks tables backed by the ADLS Gen2 (Azure Data Lake Storage).

Authentication

The connector supports separate authentication methods for the catalog and the storage accounts.

  • Catalog authentication: M2M OAuth

  • Storage account authentication: storage keys

  • Storage account authentication: service principals with secrets

Operations

The connector supports operational troubleshooting workflows.

Support bundle generation is supported.

What's next

Review Permissions and access requirements.