Databricks profiler API examples

All examples were captured against a live environment and use real, working values. IDs will differ in your own environment.

Create a connector (Databricks)

Registers a Databricks connector (source and target), which must exist before any profiling can run. Replace the catalog, storage, and Databricks workspace details below with values from your environment. The values shown are placeholders and are not real credentials.

POST /connector-info

Copy

{
  "connectorName": "Databricks_Connector_1",
  "source": {
    "storage_auth_type": "SERVICE_PRINCIPAL",
    "catalog_name": "example_catalog",
    "storage_auth_details": {
      "account_name": "examplestorageaccount",
      "tenant_id": "00000000-0000-0000-0000-000000000000",
      "client_id": "11111111-1111-1111-1111-111111111111",
      "client_secret": "<storage-client-secret>"
    },
    "catalog_auth_details": {
      "client_id": "11111111-1111-1111-1111-111111111111",
      "client_secret": "<catalog-client-secret>",
      "scope": "all-apis"
    },
    "workspace_url": "https://adb-0000000000000000.0.azuredatabricks.net/",
    "catalog_auth_type": "OAUTH_M2M"
  },
  "target": {
    "storage_auth_type": "SERVICE_PRINCIPAL",
    "catalog_name": "example_catalog",
    "storage_auth_details": {
      "account_name": "examplestorageaccount",
      "tenant_id": "00000000-0000-0000-0000-000000000000",
      "client_id": "11111111-1111-1111-1111-111111111111",
      "client_secret": "<storage-client-secret>"
    },
    "catalog_auth_details": {
      "client_id": "11111111-1111-1111-1111-111111111111",
      "client_secret": "<catalog-client-secret>",
      "scope": "all-apis"
    },
    "workspace_url": "https://adb-0000000000000000.0.azuredatabricks.net/",
    "catalog_auth_type": "OAUTH_M2M"
  }
}

Discover available profile sets

GET /profiler/profile-sets/{engine_id}

Copy

{
  "profile_sets": [
    {"profile_set_id": 4, "name": "ASDD Standard"},
    {"profile_set_id": 5, "name": "Financial - Legacy", "description": ""},
    {"profile_set_id": 6, "name": "HIPAA - Legacy", "description": ""},
    {"profile_set_id": 7, "name": "Standard", "description": ""}
  ]
}

engine_id (path parameter): the id of the compliance (masking) engine to query for its available profile sets.
profile_set_id: identifier to pass to a profile job's profile_set_id field (as a number).
name / description: human-readable label for the rule set.

Create a profile job — single table

POST /profiler/profile-jobs

Copy

{
  "connector_info_id": 2,
  "schema": "sales_schema",
  "table": "employee",
  "compliance_engine_id": 1,
  "profile_set_id": 4,
  "active_file_coverage": 75
}

Response: {"profile_job_id": 2, "status": "CREATED"}

Create a profile job — explicit list of tables

POST /profiler/profile-jobs

Copy

{
  "connector_info_id": 1,
  "schema": "sales_schema",
  "tables": ["employee", "customer", "orders"],
  "compliance_engine_id": 1,
  "profile_set_id": 4
}

Create a profile job — whole schema

POST /profiler/profile-jobs

Copy

{
  "connector_info_id": 1,
  "schema": "sales_schema",
  "compliance_engine_id": 1,
  "profile_set_id": 4
}

Field reference for all three profile-job request shapes above:

Field reference for all three profile-job request shapes above:

connector_info_id: the id of the connector (from POST /connector-info) to read data through.
schema: the schema name, in the source connector, to profile.
table: a single table name. Mutually exclusive with tables and with the whole-schema (omit both) form.
tables: an explicit array of table names within schema to profile together as one job. Mutually exclusive with table.
(schema only, table and tables both omitted): profiles every table in the schema.
compliance_engine_id: the masking engine id supplying the profile set.
profile_set_id: the profile set (rule set), from GET /profiler/profile-sets/{engine_id}, to classify columns against. Passed as a number.
active_file_coverage (optional): the percentage of a table's active files to scan during profile execution. Accepts integer values from 1 to 100 — for example, 75 scans 75% of active files. Lower values profile faster at the cost of sampling less data; omit it to use profiler-service's default coverage.

List profile jobs

GET /profiler/profile-jobs

Response (one entry shown):

Copy

{
  "profile_job_id": 2,
  "connector_info_id": 2,
  "schema": "sales_schema",
  "table": "employee",
  "profile_set_id": 4,
  "compliance_engine_id": 1,
  "create_data_set": true,
  "active_file_coverage": 75,
  "confidence_threshold": 0.0,
  "created_at": "2026-07-15T03:18:31.091957",
  "updated_at": "2026-07-15T03:18:31.091961"
}

create_data_set: Specifies whether a successful execution of this job automatically creates or updates a data set entry containing the resulting masking inventory. Observed default: true.
active_file_coverage: Returns the percentage of active files (1-100) scanned during each execution of this job, as specified when the job was created.
confidence_threshold: A numeric threshold associated with the job's classification step. Observed default: 0.0. The meaning of this field is inferred from its name and context and is not independently documented at the API level. Treat it as an advanced or optional tuning parameter rather than a field required for basic use.

Update a profile job

PUT /profiler/profile-jobs/{id}

Partial update — send only the fields to change. For example, to change active_file_coverage on an existing job:

{"active_file_coverage": 50}

Response:

{"profile_job_id": 2, "updated": true}

Status: 200 OK. The response confirms that the update was successful but does not return the updated record. Use GET /profiler/profile-jobs/{id} to view the updated values. The updated_at timestamp is also updated

Delete a profile job

DELETE /profiler/profile-jobs/{id}

No request body. Response body is empty.

Status: 200 OK. A subsequent GET on the same id returns 404:

{"error_field": "Error in profiler profile-job", "error_message":

 ["{\"error\":\"profile job not found\",\"profile_job_id\":3}"]}

Deleting a job does not appear to cascade-delete data-sets or executions already created from it — see Limitations.

Execute a profile job

POST /profiler/profile-executions

{"profile_job_id": 2}

Response:

{"execution_id": 2, "status": "PENDING"}

profile_job_id: the id returned when the job was created. This is the only field required to start an execution.

Poll execution status

GET /profiler/profile-executions/{execution_id}/status

While running:

Copy

{
  "execution_id": 2,
  "status": "RUNNING",
  "started_at": "2026-07-15T03:18:39.764068"
}

Once terminal (SUCCEEDED):

Copy

{
  "execution_id": 3,
  "status": "SUCCEEDED",
  "started_at": "2026-07-20T14:29:44.101439",
  "completed_at": "2026-07-20T14:29:58.097392",
  "data_set_id": 1,
  "result": {
    "connector_id": 3,
    "data_info": [
      {
        "source": {
          "schema_name": "sales_schema",
          "table_name": "employee",
          "options": []
        },
        "target": {
          "schema_name": "sales_schema",
          "table_name": "employee",
          "options": []
        },
        "masking_inventory": [
          {
            "field_name": "address",
            "logical_field_name": "address", 
            "domain_name": "ADDRESS",
            "algorithm_name": "AddrLookup"
          },
          {
            "field_name": "last_name",
            "logical_field_name": "last_name",
            "domain_name": "LAST_NAME",
            "algorithm_name": "dlpx-core:LastName"
          },
          {
            "field_name": "department",
            "logical_field_name": "department",
            "domain_name": "DEPARTMENT",
            "algorithm_name": "dlpx-core:Department SL"
          },
          {
            "field_name": "first_name",
            "logical_field_name": "first_name",
            "domain_name": "FIRST_NAME",
            "algorithm_name": "dlpx-core:FirstName"
          }
        ]
      }
    ],
    "skipped": []
  }
}

status: PENDING and RUNNING indicate that the execution has not yet reached a terminal state and should continue to be polled. SUCCEEDED and FAILED are terminal states.
data_set_id: Present once the execution reaches SUCCEEDED. Use it with GET /data-sets/{dataSetId} to retrieve the same masking inventory as a persisted data set resource.
result.data_info[].masking_inventory: The profiling output. Each entry represents a classified column and includes the detected domain and the masking algorithm selected for that column.
skipped: Columns considered but not matched to any domain in the profile set.
field_name: The physical name of the column in the underlying Parquet file or files that back the Databricks table.
logical_field_name: The column's name as it currently appears in the Databricks table schema — what you see when you query the table.

field_name and logical_field_name are usually identical, but they can differ for Databricks Delta tables with column mapping enabled. This feature allows a column to be renamed, or dropped and re-added, at the table level without rewriting the underlying Parquet files.

When this occurs, the Parquet file retains the original physical column name (field_name), while logical_field_name reflects the table's current renamed current column. Use logical_field_name when referencing the column in Databricks queries or when determining what to mask. field_name is provided primarily for traceability back to the physical file.

Verify the generated data set

GET /data-sets/{dataSetId}

Copy

{
  "id": 2,
  "connector_id": 2,
  "data_info": [{
    "source": {
      "schema_name": "sales_schema",
      "table_name": "employee",
      "options": []
    },
    "target": { ... same shape as source ... },
    "masking_inventory": [
      ... same shape as the execution result above ...
    ]
  }]
}

This confirms that the data set was persisted as a separate resource, independent of the execution response that originally returned it. This can be used as a final verification step after profiling has completed.

Generate a support bundle

POST /support-bundle

Copy

{
  "async_task_id": 1,
  "operation": "SUPPORT_BUNDLE_GENERATE",
  "reference": "hyperscale-support-15072026-03-19-10",
  "status": "RUNNING",
  "start_time": "2026-07-15T03:19:10.72628Z",
  "cancellable": true
}

GET /async-tasks/{async_task_id} — poll until terminal:

Copy

{
  "async_task_id": 1,
  "operation": "SUPPORT_BUNDLE_GENERATE",
  "reference": "hyperscale-support-15072026-03-19-10.tar.gz",
  "status": "SUCCEEDED",
  "start_time": "2026-07-15T03:19:10.72628Z",
  "end_time": "2026-07-15T03:20:11.066758Z",
  "cancellable": true
}

reference: The filename of the generated archive, available once the status reaches SUCCEEDED. The archive contains diagnostics collected from controller-service, masking-service, profiler-service, load-service, and unload-service.

Debugging via logs

The profiler service emits structured JSON log entries to stdout, each containing a request_id and trace_id that can be correlated with the corresponding request in the controller service logs:

Copy

docker logs <profiler-service-container> 2>&1 | grep -E 'connector_info_id|profile-job|profile-execution'
{"connector_info_id": 2, "connector_type": "databricks", "level": "info", "message": "connector_info inserted", "request_id": "...", "trace_id": "..."}
{"connector_info_id": 2, "level": "info", "message": "received profile-job creation", "request_id": "...", "trace_id": "..."}
{"connector_info_id": 2, "level": "info", "message": "profile_job created", "profile_job_id": 2, "request_id": "...", "trace_id": "..."}
{"level": "info", "message": "received profile-execution request", "profile_job_id": 2, "request_id": "...", "trace_id": "..."}
{"execution_id": 2, "level": "info", "message": "profile-execution accepted", "profile_job_id": 2, "status": "PENDING", "request_id": "...", "trace_id": "..."}

The "connector_info inserted" log entry shows the Profiler service's internal connector_info_id for a newly created connector. If this value differs from the connector ID reported by the controller service for the same connector, the mismatch described in the Troubleshoot known issue has occurred. This log entry provides the quickest way to verify the issue.

The controller service logs show the same requests one layer up, where they appear as proxy calls to downstream services:

Copy

docker logs <controller-service-container> 2>&1 | grep -iE 'connector|profile-job|profile-execution|error|exception'
INFO  ConnectorInfoController.createConnectorInfo - in createConnectorInfo
INFO  ProfilerServiceStub.createConnectorInfo - sending /connector-info post request to profiler service
INFO  ProfilerService.createProfileJob - Proxying create profile-job to profiler-service
INFO  ProfilerServiceStub.createProfileJob - sending /profile-job post request to profiler service

If a /profiler/* call fails, first check the controller service logs to confirm that the request reached the profiler service. Then check the Profiler service logs, filtering on the relevant ID, to determine how the request was processed.