Troubleshoot Databricks

Masking issues

Use this topic to review the current known issue for the Databricks connector and the documented workaround.

Known issue: Predictive Optimizer is not re-enabled after a failed or canceled job

Re-enabling Predictive Optimizer might not occur in a specific failure path.

  • The issue can occur if an execution fails or is canceled before the load phase begins, for example if masking fails before the load service receives POST /api/load.

  • The issue can also require that the execution retain_execution_data setting in the job request is ON_ERROR or ALWAYS.

If this issue occurs, re-enable Predictive Optimizer manually and match the table's original setting.

ALTER TABLE <catalog>.<schema>.<table> ENABLE PREDICTIVE OPTIMIZATION;

 

Potential target Databricks table corruption

Databricks table corruption can occur in the following scenario:

  1. Hyperscale Databricks Job was executed successfully for a table.

  2. That job or another job referencing the same table(s) that were previously masked is executed and during that job a failure or cancellation of the masking process is observed.

  3. This could lead to one or more of the target files that were already part of the delta log to become corrupted (previously masked file that is being overwritten by a new masked file).

  4. This corruption of the file could cause Databricks queries to fail.

Resolution

Use one of the following methods to resolve the issue:

  • Recommended: Rerun the Hyperscale job for the affected table. If the job completes successfully, the corrupted file should be corrected, the corresponding Delta log entries updated, and any obsolete files cleaned up as part of the Hyperscale job.

  • Manually repair the target table and files. Possible approaches include:

    • Roll back the Delta table to a previous version.

    • Run FSCK REPAIR on the table.

    • Create a manual Delta log entry to remove references to the corrupted files.