API reference and examples

This topic provides the example request payloads and field descriptions for the Hyperscale Databricks connector. Use these examples as reference content when you create connectors, data sets, and masking jobs.

Create a connector with a storage account storage key

Copy 
{
  "connectorName": "Databricks_Connector_regression",
  "source": {
    "workspace_url": "Workspace-Url",
    "catalog_auth_type": "OAUTH_M2M",
    "catalog_auth_details": {
      "client_id": "Service Principal ID",
      "client_secret": "Unity SP Secret",
      "scope": "all-apis"
    },
    "storage_auth_type": "STORAGE_KEY",
    "catalog_name": "mkk_dbs_workspace",
    "storage_auth_details": {
      "account_key": "storage key",
      "account_name": "account name"
    }
  },
  "target": {
    "workspace_url": "Workspace-URL",
    "catalog_auth_type": "OAUTH_M2M",
    "catalog_auth_details": {
      "client_id": "Service Principal ID",
      "client_secret": "Unity SP Secret",
      "scope": "all-apis"
    },
    "storage_auth_type": "STORAGE_KEY",
    "catalog_name": "catalog name",
    "storage_auth_details": {
      "account_key": "storage key",
      "account_name": "account name"
    }
  }
}

 

Create a connector with a storage account service principal

Copy 
{
  "connectorName": "Databricks_Connector_SP",
  "source": {
    "workspace_url": "workspace-url",
    "catalog_name": "main",
    "catalog_auth_type": "OAUTH_M2M",
    "catalog_auth_details": {
      "client_id": "22222222-3333-4444-5555-666666666666",
      "client_secret": "<oauth-m2m-client-secret-placeholder>",
      "scope": "all-apis"
    },
    "storage_auth_type": "SERVICE_PRINCIPAL",
    "storage_auth_details": {
      "tenant_id": "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee",
      "client_id": "ffffffff-0000-1111-2222-333333333333",
      "client_secret": "<azure-service-principal-secret-placeholder>",
      "account_name": "account name"
    }
  },
  "target": {
    "workspace_url": "https://adb-1234567890123456.7.azuredatabricks.net",
    "catalog_name": "main",
    "catalog_auth_type": "OAUTH_M2M",
    "catalog_auth_details": {
      "client_id": "22222222-3333-4444-5555-666666666666",
      "client_secret": "<oauth-m2m-client-secret-placeholder>",
      "scope": "all-apis"
    },
    "storage_auth_type": "SERVICE_PRINCIPAL",
    "storage_auth_details": {
      "tenant_id": "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee",
      "client_id": "ffffffff-0000-1111-2222-333333333333",
      "client_secret": "<azure-service-principal-secret-placeholder>",
      "account_name": "account name"
    }
  }
}

 

 

Connector field descriptions

Use the following field descriptions when you create a connector payload.

  • workspace_url: The Databricks workspace URL.

  • catalog_name: The name of the catalog in the workspace.

  • catalog_auth_type: The authentication type for catalog access. The only supported value is OAUTH_M2M.

  • catalog_auth_details: The JSON object that contains the settings for the selected catalog_auth_type.

  • client_id: The service principal ID that is registered with Databricks.

  • client_secret: The Databricks secret associated with the service principal.

  • scope: The API-level scope for the Unity Catalog APIs.

  • storage_auth_type: The authentication pattern for the underlying storage account.

  • storage_auth_details: The JSON object that contains the settings for the selected storage_auth_type.

  • STORAGE_KEY.account_name: The name of the storage account.

  • STORAGE_KEY.account_key: The storage key.

  • SERVICE_PRINCIPAL.tenant_id: The Azure tenant ID.

  • SERVICE_PRINCIPAL.client_id: The storage account service principal ID.

  • SERVICE_PRINCIPAL.account_name: The name of the storage account.

  • SERVICE_PRINCIPAL.client_secret: The secret associated with the service principal.

 

Databricks data set example

Copy 
{
  "connector_id": 1,
  "data_info": [
    {
      "source": {
        "schema_name": "src_schema",
        "table_name": "customers",
        "options": [
          {
            "filter_type": "PARTITION",
            "filter_column": "region",
            "filter_value": "USA"
          },
          {
            "filter_type": "TIME_BASED",
            "filter_column": "order_date",
            "min_date": "2025-01-01",
            "max_date": "2025-06-06"
          }
        ]
      },
      "target": {
        "schema_name": "tgt_schema",
        "table_name": "customers",
        "options": []
      },
      "masking_inventory": [
        {
          "field_name": "FIRST_NAME",
          "domain_name": "FIRST_NAME",
          "algorithm_name": "dlpx-core:FullName"
        },
        {
          "field_name": "ADDRESS",
          "domain_name": "ADDRESS",
          "algorithm_name": "dlpx-core:CM Alpha-Numeric"
        }
      ]
    }
  ]
}

 

Data set field descriptions

  • schema_name: The schema name where the table exists.

  • table_name: The table to mask.

  • options: There are no target options at this time. This field is a placeholder for future use. Source options are currently used only for filters.

  • filter_type: The type of filter to apply. The supported values are TIME_BASED and PARTITION.

  • filter_column: The column name for the filter. This value can be the Databricks table column name (logical) or the Parquet column name (physical).

  • filter_value: The partition value used in a PARTITION filter.

  • min_date: The low end of the range used in a TIME_BASED filter.

  • max_date: The upper end of the range used in a TIME_BASED filter.

 

Databricks job example

Copy 
{
  "name": "job_1",
  "masking_engine_ids": [
    1
  ],
  "data_set_id": 1,
  "app_name_prefix": "app",
  "env_name_prefix": "env",
  "consider_continuous_compliance_warning_event_as": "SUCCESS",
  "retain_execution_data": "NO",
  "masking_job_config": {
    "max_memory": 2048,
    "min_memory": 0,
    "description": "Job created by Databricks Hyperscale Masking",
    "feedback_size": 100000,
    "stream_row_limit": 10000,
    "num_input_streams": 1
  }
}

No special fields are required for Databricks as part of job creation.