Synthetic data discovery methods

When you create a dataset in Synthetic Data, the system automatically discovers and assigns generation strategies to columns. This discovery process uses a three-layer approach, with each layer applying increasingly sophisticated methods to classify columns and suggest appropriate generators.

Columns are processed through the layers in sequence. A column classified by an earlier layer does not proceed to later layers.

How the layers work together

The three discovery layers form a cascade model. During application sync, the system first assigns all columns a default generator based on their data type (for example, "default string" or "default integer"). These defaults have no semantic meaning — they simply generate random values.

Each discovery layer then processes only the columns that still have these default generator assignments:

  • ASDD — Replaces default generators with meaningful ones based on regex rules

  • Deep Search — Replaces default generators with meaningful ones based on semantic similarity

  • LLM-based generation — Replaces remaining default generators with LLM-suggested meaningful generators

If a column receives a meaningful (non-default) generator from any layer, it is not processed by subsequent layers. This preference model ensures that your ASDD rules take precedence over automated discovery.

ASDD (Automatic Sensitive Data Discovery)

ASDD uses regex-based classification rules to match columns by name, table, data type, or other metadata patterns. Each generator has associated classification rules that define when it should be applied. ASDD currently runs during application sync operations. It applies to built-in and user defined generators, and can also create a custom generator configuration when rule conditions require it. ASDD is deterministic. A column either matches a rule or it does not; there is no ambiguity or scoring involved.

Configuration

Classification rules are managed per generator. To view or create rules:

  1. Navigate to Synthetic > Generators.

  2. Select a generator to view its associated classification rules.

  3. Rules specify conditions such as exact column name matches, regex patterns, or data type matches.

Example: A rule might specify "if column name exactly matches 'SWIFT_CODE', assign the SWIFT Code generator regardless of table."

Deep Search

Deep Search uses semantic (vector) similarity to compare column names and table metadata against built-in generator labels. For example, it might compare a column named "emp_id" against labels like "employee identification number", "tax ID", etc., and suggest the closest match. Deep Search works only on columns that still have default generator assignments after ASDD completes. If ASDD already assigned a meaningful generator to a column, that column is not re-evaluated by Deep Search. Deep Search runs during both application sync and dataset creation.

Deep Search applies only to built-in generators shipped with DCT. Custom generators you define do not participate in Deep Search and can only be assigned via ASDD rules.

Confidence threshold

Deep Search applies a minimum confidence threshold to ensure matches are meaningful. The default threshold is 70% and can be configured via API (see Configure Synthetic Data settings). Only matches that meet or exceed this threshold are suggested as generation strategies.

Example

A column named "contact_phone" in a customer table might match the "phone number" generator label with 85% confidence, exceeding the 70% threshold. The phone number generator is assigned to the column.

LLM-based generation

For columns not classified by ASDD or Deep Search, the system passes the table structure, column metadata, and context to a configured LLM provider and requests a suggestion for an appropriate generation strategy. LLM-based generation runs during dataset creation for unclassified columns, after ASDD and Deep Search have completed. LLM-based generation creates a custom generator configuration for matched columns; it does not reuse built-in or user defined generators. Unlike ASDD (rule-based) or Deep Search (pattern-based), LLM-based generation leverages semantic understanding of schema context, business meaning, and data relationships to suggest generators.

Requirements

  • License: Enterprise license tier

  • LLM Provider: An LLM provider must be configured (see Configure LLM providers)

  • Feature enabled: LLM-based discovery must be enabled in Admin > AI Settings > LLM Features

Example

A column named "contact_details" in an employee table might be ambiguous to pattern matching, but an LLM could suggest a phone number or email generator based on the broader table context and typical employee data structures.

Viewing assignment methods

After you create or sync a dataset, you can see which assignment method set each column's generator. In the Datasets editor, use the Assignment Method column:

  1. Auto Discovery — ASDD assigned an existing built-in or user defined generator based on regex rules.

  2. Auto Discovery Custom — ASDD created a custom generator configuration instead of selecting a built-in or user defined generator.

  3. User Override — You manually selected a built-in or user defined generator.

  4. User Custom — You manually created a custom generator configuration.

  5. Deep Search — Deep Search selected a built-in generator.

  6. LLM Assigned — LLM-based discovery created a custom generator configuration.

This information helps you understand how each assignment was made and validate that discovery behavior matches your expectations.

Which discovery layer applies to your generator?

Generator Type Discovery layers applied (in order)
Built-in generator ASDD and Deep Search can assign built-in generators.
User defined generator ASDD can assign user defined generators. Deep Search does not assign user defined generators.
Custom generator configuration Can be created by ASDD (Auto Discovery Custom), LLM-based generation (LLM Assigned), or manually by you (User Custom).

What if no layer matches?

If no discovery layer matches a field, the default generator remains assigned. Fields that are database managed are not assignable and do not receive user-editable generator assignments.