ROSA HCP Disaster Recovery with ACM and OpenShift GitOps

This guide demonstrates how to set up an active/passive disaster recovery pattern for applications running on ROSA HCP clusters using Red Hat Advanced Cluster Management (ACM) and OpenShift GitOps (ArgoCD). ACM handles cluster health monitoring and automatic failover detection, while ArgoCD deploys the application to whichever cluster ACM selects.

The pattern works as follows:

• ACM monitors cluster health via klusterlet heartbeats
• A Placement resource selects one healthy cluster at a time (active/passive)
• When the active cluster becomes unreachable, ACM automatically moves the placement to the standby cluster
• ArgoCD detects the placement change and deploys the application to the new target cluster
• DNS is switched manually to point to the new active cluster

Prerequisites

This guide assumes the following are already in place:

• Two ROSA HCP clusters in different AWS regions (referred to as $CLUSTER_EAST in us-east-1 and $CLUSTER_WEST in us-west-2)
• A third ROSA HCP cluster for the ACM hub ($CLUSTER_ACM) with the ACM operator installed
• S3 buckets created in each region for application data, with S3 Cross-Region Replication configured from the primary to the DR bucket
• EFS file systems created with cross-region replication (primary EFS replicating to the DR region), with the AWS EFS CSI Driver Operator installed on both regional clusters. Follow Enabling the AWS EFS CSI Driver Operator on ROSA to set up EFS CSI on each cluster.
• EFS mount targets created in the DR cluster’s worker subnets with NFS (port 2049) allowed in the security group
• IRSA roles for S3 access created on each regional cluster
• AWS CLI, oc CLI, rosa CLI, and helm CLI configured
• A Route 53 hosted zone for the custom domain

Environment Variables

Set the following environment variables. Update the values to match your environment.

Log into the ACM Hub Cluster

All resources in this guide are created on the ACM hub cluster unless otherwise noted.

Import Managed Clusters into ACM

Import each regional cluster so ACM can monitor and manage them.

Import the East Cluster

1. Create the ManagedCluster resource

2. Get a token from the east cluster and create the auto-import secret

   Note: Use oc create secret generic with --from-literal rather than inline YAML. Tokens often contain special characters that break YAML parsing.

3. Wait for the cluster to be imported and available

   Wait until AVAILABLE shows True:

Import the West Cluster

1. Create the ManagedCluster resource

2. Get a token from the west cluster and create the auto-import secret

3. Wait for the cluster to be imported and available

4. Verify both clusters are imported

Create a ManagedClusterSet

Group the regional clusters into a ManagedClusterSet so they can be referenced as a single pool for placement decisions.

1. Create the ManagedClusterSet

2. Add both clusters to the set

3. Create a ManagedClusterSetBinding in the openshift-gitops namespace to allow ArgoCD to use this cluster set

Install OpenShift GitOps on the Hub

Install the OpenShift GitOps operator which provides ArgoCD.

1. Install the operator

2. Wait for the operator to install

   Wait until the PHASE shows Succeeded.

3. Grant the ArgoCD service account cluster-admin privileges

4. Get the ArgoCD admin password

Register Managed Clusters with ArgoCD

Use the ACM GitOpsCluster CRD to register managed clusters as ArgoCD deployment targets. This uses the ACM cluster-proxy so ArgoCD can deploy to managed clusters without direct network access.

1. Create a Placement to select all clusters in the DR cluster set

2. Create the GitOpsCluster resource

3. Verify the clusters appear as ArgoCD cluster secrets

   You should see secrets for both kmc-east and kmc-west.

Configure ACM Placement for Failover

Create the Placement that controls which cluster the application is deployed to. This is the core of the DR mechanism.

1. Create the application Placement

   This Placement selects exactly one cluster from the dr-clusters set. The Steady prioritizer keeps the application on its current cluster unless it becomes unreachable. The tolerations allow 30 seconds after a cluster is tainted as unreachable before the placement moves.

2. Create the ConfigMap that tells the ArgoCD ApplicationSet how to read ACM PlacementDecisions

3. Verify the placement is selecting a cluster

Tune Lease Duration for Faster Failover Detection

By default, ACM checks the klusterlet heartbeat lease every 5 minutes. For a faster demo, reduce the lease duration to 10 seconds on both managed clusters.

Obtain a TLS Certificate (Optional)

If you want to serve the application on a custom domain with a valid TLS certificate, obtain one using Let’s Encrypt with a DNS-01 challenge via Route 53.

1. Request the certificate

   Follow the prompts to create a DNS TXT record in Route 53 for validation.

2. Store the certificate and key in environment variables

Create the ArgoCD ApplicationSet

The ApplicationSet uses a merge generator that combines two sources:

• clusterDecisionResource: reads from ACM’s PlacementDecision to know which cluster to deploy to
• list: provides per-cluster configuration (region, S3 bucket, IRSA role ARN, EFS file system ID)

When the PlacementDecision changes (e.g., failover), ArgoCD automatically deploys the application to the new target cluster and removes it from the old one.

1. Create the ApplicationSet

   First, if you obtained a TLS certificate, prepare the indented cert and key for YAML embedding:

   Then create the ApplicationSet:

   Note: The heredoc substitutes ${VAR} references with your environment variable values. The Go template {{.field}} references use double curly braces and are not substituted by the shell — they are processed by ArgoCD at deploy time. If you did not set TLS_CERT and TLS_KEY, the TLS fields will be empty and the route will use the cluster’s default wildcard certificate.

2. Verify the Application was created and is syncing

   Wait until the application shows Synced and Healthy:

Prepare DR Cluster for EFS Data Continuity

When ArgoCD deploys the application to the DR cluster, the Helm chart dynamically provisions new EFS access points for each PVC. These new access points create fresh, empty subdirectories — the replicated data from the primary EFS lives under different paths. To ensure the DR application sees the replicated data, pre-create static PersistentVolumes on the DR cluster that point to the original data paths.

The demo application uses 3 EFS-backed PVCs:

• shared-flight-data — shared volume mounted by the dashboard and flight recorder
• flight-data-flight-recorder-0 — StatefulSet replica 0
• flight-data-flight-recorder-1 — StatefulSet replica 1

Record the PVC-to-path mapping

On the primary cluster, map each PVC to its EFS access point path. The PV volumeHandle format is <efs-id>::<access-point-id>, and each access point has a root directory path where the PVC’s data is stored:

Export the paths from the output:

Pre-stage static PersistentVolumes on the DR cluster

Log into the DR cluster and create static PVs with claimRef pre-binding. The claimRef reserves each PV for a specific PVC so that when ArgoCD deploys the application, the PVCs bind to these PVs instead of dynamically provisioning new access points:

Log back into the ACM hub:

Set Up DNS

Create a Route 53 A record pointing to the router of the active cluster.

1. Get the ELB hostname and IP for the east cluster

2. Get the ELB hostname and IP for the west cluster

3. Create the DNS record pointing to the primary (east) cluster

4. Verify the application is accessible

Failover Test

Simulate a region failure by stopping the worker instances on the east cluster.

1. Verify the application is healthy

2. Verify EFS replication is healthy and review the most recent replication timestamp before promoting the DR file system

   Confirm that the replication status is ENABLED and that LastReplicatedTimestamp meets your recovery point objective before proceeding. Data written after the last replicated timestamp might not be available on the DR file system.

3. Delete EFS replication to promote the DR replica to read-write

   EFS cross-region replicas are read-only while replication is active. The DR cluster's pods cannot write to the replica file system until it is promoted. Deleting the replication configuration is the only way to promote it — AWS does not have a separate `promote` API. Once deleted, the DR EFS becomes an independent read-write file system. During failback, the guide re-establishes replication from primary to DR.

4. Get the east cluster worker instance IDs

5. Stop the east worker instances

6. Watch for ACM to detect the failure and ArgoCD to deploy to the west cluster

   With the tuned lease duration (10s) and toleration (30s), this should take approximately 40-50 seconds.

   Wait until Available changes from True to Unknown and a new application acm-demo-kmc-west appears with Synced/Healthy status.

7. Switch DNS to the west cluster

8. Flush local DNS cache and verify

Failback

Failing back is a manual process. The Steady prioritizer in the Placement keeps the application on the current (west) cluster even after east recovers, preventing unnecessary flip-flopping.

1. Start the east worker instances

2. Wait for the east cluster to rejoin ACM

   This typically takes 2-3 minutes as the klusterlet pods restart and begin sending heartbeats.

   Wait until it shows True.

3. Force the placement back to the east cluster

   The Steady prioritizer keeps the app on west (the current cluster). To fail back, temporarily add a label selector that only matches the east cluster:

4. Verify ArgoCD deployed to the east cluster

   Wait until acm-demo-kmc-east shows Synced/Healthy.

5. Re-establish EFS replication from primary to DR

   Before allowing production traffic to return to the primary cluster, verify that the application is healthy and that any required DR-side EFS and S3 data has been reconciled. Application health alone is not sufficient to determine that a stateful workload is ready for failback.

   Re-establish EFS replication so it is in place for future failovers. First, disable the overwrite protection that AWS enables on the replica after replication is deleted:

6. Switch DNS back to the east cluster

7. Remove the failback label selector to restore automatic failover

8. Flush local DNS cache and verify

Failover Timeline Summary

Production Considerations

• EFS path mapping: Record and maintain the PVC-to-EFS access point path mapping as part of your DR runbook. In a real disaster, the primary cluster API might not be available to query. Update this mapping whenever PVCs are recreated.
• Data reconciliation before failback: Both EFS and S3 replication are one-directional (primary → DR). Data written during failover must be manually synced or merged back to the primary before re-establishing replication. See the ROSA DR with OADP guide for detailed failback data reconciliation steps.
• ACM hub availability: The ACM hub is a single point of failure for failover detection. In production, deploy the hub with high availability or consider an active-passive hub configuration.
• DNS automation: Replace the manual DNS switch with Route 53 health checks and failover routing policies for fully automated DR.
• Lease duration tuning: The 10-second lease used in this guide is aggressive. For production, balance detection speed against the risk of false positives from transient network issues. A 60-second lease is a reasonable starting point.
• EFS mount targets: Ensure the DR cluster has EFS mount targets in all worker subnets before a disaster occurs. Creating mount targets during a failover adds delay to the recovery process.

Cleanup

1. Delete the ApplicationSet

2. Delete the Placement and ConfigMap

3. Delete the GitOpsCluster and all-clusters Placement

4. Delete the ManagedClusterSetBinding and ManagedClusterSet

5. Detach the managed clusters

6. Delete the DNS record