Automated disaster recovery

Automated disaster recovery lets you pair two DCT appliances so that one (the secondary) replicates the application database of the other (the primary). If the primary appliance is lost, you promote the secondary to take its place, with minimal data loss. Automated disaster recovery requires a DCT Enterprise license; see Licensing.

It is one of two ways to recover the DCT appliance; the other is the manual Backup and restore process. For how the two compare and when to use each, see Disaster recovery.

Automated disaster recovery protects the DCT appliance's own data, not the datasets on the engines DCT manages.

Overview

Automated disaster recovery is configured and monitored from the appliance Setup Dashboard — the administrative interface you also use to manage the software version, storage, network, and time settings — on a dedicated Disaster Recovery tile.

An appliance is always in one of three disaster recovery roles:

Standalone — the default. The appliance has no disaster recovery relationship.
Primary — the appliance is being replicated by one or more secondaries. From a primary you can view and remove its registered secondaries.
Secondary — the appliance is replicating from a primary. From a secondary you can rotate its credentials, resynchronize it, or promote it to become a new primary.

A secondary's only role action is to promote itself to primary (this is how you fail over, and how you run a failover test). To return an appliance to standalone, remove it from its primary. To make a promoted appliance a secondary again, reconfigure it to sync from the primary.

How replication works

When you first configure a secondary, it takes a full copy of the primary's database, which can take some time for a large database — the Disaster Recovery tile may not show the secondary as caught up until that completes. After that, the secondary streams the primary's PostgreSQL write-ahead log (WAL), an ongoing record of every database change, and applies it as changes occur to stay closely synchronized. If the secondary falls far enough behind, for example after a prolonged network interruption, the primary may no longer be able to retain the WAL data the secondary still needs. When that happens, the secondary automatically takes a fresh full copy to catch up, then resumes streaming.

Replication uses a pull model: the secondary always initiates the connection, so your firewall must allow it to reach the primary (not the reverse) on these ports:

Port	Used for	Direction
5433 (TCP)	Database (WAL) streaming and resynchronization	Secondary to primary
443 (HTTPS)	Registering the secondary and synchronizing encryption keys	Secondary to primary

Service encryption keys are synchronized separately over the HTTPS connection, each secondary using its own key material, so a promoted secondary can operate without the primary.

Requirements and sizing

License: automated disaster recovery requires a DCT Enterprise license, and is available only on the DCT appliance.
Storage: provision the secondary with twice the primary's storage — a full copy is written alongside the existing one during resynchronization, so both must fit at once.
CPU and memory: a secondary runs most services shut down, so it only needs the minimum appliance specifications (8 CPUs and 16 GB, or more with AI features). After promoting it to primary, increase CPU and memory to match its new workload.
Network: ports 5433 and 443 open inbound on the primary and outbound from the secondary (the secondary initiates the connection).
Versions: the primary and secondary must run the same DCT version and the same major PostgreSQL version when you pair them.

Set up and operate disaster recovery

Configure a secondary — pair an appliance as a secondary and secure the replication connection.
Failover testing — rehearse a failover and verify each service afterward.

Failover, resync, and upgrade tasks are described below.

Performing a failover (promote a secondary)

When a primary is lost, promote its secondary to take over. On the secondary's Disaster Recovery tile, select Promote to primary.

Promotion requires you to acknowledge a warning about running two primaries at once ("split brain"), and it permanently severs the replication relationship with the original primary. There is no undo: to re-establish replication afterward, you configure one of the two appliances as a secondary of the other from the start.

After promotion, the appliance operates as a standalone primary. Complete the post-failover verification steps to confirm each DCT service is fully operational, because some services require manual steps after a failover — see Service coverage and known limitations. To rehearse this before a real event, see Failover testing.

Resync and automatic recovery

If replication falls too far behind or breaks, DCT recovers automatically in most cases. A background process resynchronizes the secondary when it detects a network failure, when the primary can no longer supply the data the secondary needs, or when the replication gap grows too large. To prevent repeated resynchronization attempts from looping, DCT caps automatic attempts and, once that limit is reached, sets the state to Manual intervention required.

You can also trigger a manual resynchronization from the secondary's Disaster Recovery tile if automatic recovery has not resolved the problem.

Upgrading a primary/secondary pair

When you upgrade a paired appliance, upgrade the secondary first, then the primary. Upgrading the secondary first means that if the primary's upgrade runs into a problem, the already-upgraded secondary can be promoted to take over.

As with standard PostgreSQL, a major PostgreSQL version difference between the two appliances is incompatible: a major PostgreSQL upgrade on one side breaks replication until the other side is upgraded to match. Expect replication to show as broken between the two upgrade steps; it resolves once both appliances are on the same major version. Minor PostgreSQL version differences do not break replication.

Monitoring replication health

DCT does not send email or external alerts for replication problems — check the Disaster Recovery tile on the secondary. It shows the replication state, the connected primary, replication lag, the last key-sync time, and certificate-validation status. In an error state, a View error detail option reports the specific cause.

The replication state can be one of the following:

State	Meaning
Streaming	Healthy. Changes are actively streaming from the primary.
Startup / Catching up	Initial synchronization or catch-up is in progress after connecting or reconnecting. This is expected to move to Streaming.
Disconnected	The connection to the primary has dropped. Causes include a network interruption, an expired or changed certificate on the primary, or a PostgreSQL major-version mismatch during an upgrade. Use View error detail to see which. DCT attempts to recover automatically; if it cannot, the state escalates to Manual intervention required.
Slot invalidated	The secondary fell far enough behind that the primary no longer retained the data it needed, usually after a sustained network problem. DCT automatically performs a full resynchronization.
Resync in progress	A full resynchronization is underway, either automatic or manually triggered.
Manual intervention required	DCT could not recover automatically — typically the primary was unreachable for an extended period. Use View error detail, check network reachability and the primary's health, then consider a manual resynchronization.
Unknown	The state could not be determined, for example immediately after a role change.

Managing replicas and credentials

From the primary, the Disaster Recovery tile lists the registered secondaries, showing each one's status, name, and address. You can remove a secondary from here; removing a secondary is also how you return that appliance to standalone.

From the secondary, the Disaster Recovery tile provides a menu of actions for that appliance:

Option	What it does
Rotate password	Rotates the replication password this secondary uses to connect to the primary. Initiated from the secondary; the primary does not rotate a secondary's credentials.
Rotate token	Rotates the replication token this secondary uses to authenticate to the primary, in the same way as the password.
Promote to primary	Promotes this secondary to primary. See Performing a failover.
Manual resync	Triggers a fresh full copy of the primary's database. See Resync and automatic recovery.
Reconfigure secondary	Points this secondary at a different primary to replicate from (or re-applies its connection settings, such as SSL validation). This is also how you return a promoted appliance to a secondary role — for example, after a failover test — by configuring it to sync from the original primary again.

Reconfigure secondary is commonly used for failover testing: after you promote a secondary and finish testing, use it to reassign that appliance as a secondary of the original primary. See Failover testing.

Service coverage and known limitations

Core DCT and virtualization data is stored in the database and is fully protected — it is present on the promoted secondary and operational immediately after a failover.

Synthetic data generation and masking are not currently supported with disaster recovery. Support for these services is planned for a future release. If you rely on synthetic data generation or masking, take this into account before using disaster recovery in production.

A few services keep some configuration outside the database and need a manual step after you promote a secondary:

AI service — re-upload the AI models you had uploaded to the primary.
LDAP authentication over TLS — the LDAP configuration is replicated, but the certificate material used to validate the LDAP server is not. Reconfigure LDAP TLS trust on the promoted appliance.
Network Security trust store — trusted certificate files are not replicated (only their database references are). Re-add any trusted certificates on the promoted appliance.
SAML / SSO — the SSO configuration is replicated, but your identity provider may still point at the former primary's address. Sign in with a local username and password or an API key, then update the identity provider to target the promoted appliance.

For step-by-step guidance on running a failover test and verifying each service afterward, see Failover testing.