Conversation
…on module, unit tests One Python package selected by Lambda Handler string (drs_region_switch.<module>.handler). Extracted from drs-mr-demo (proven live 2026-09-10..14) with: - region-explicit clients so the same code deploys to both plan regions - step table (STEPS) as the contract for the CDK construct / Terraform module / docs - recover: adopts the cycle's own launch job via a source-server tag; stateful never adopts a leftover instance; FAILED launches surface the job-log reason - register-target / register-failback-target: instance located via DRS, never by tag - failback-launch: launch-into-source aware (stops the target first, no retag of the original) - reprotect: 'started?' from the recovery instance's own state; RESCAN completion from real fields - retire: DRS-documented cleanup order; resting-state invariant; clears the job tag - 28 unit tests with fake clients, one per live finding
… def instead of lambda)
…t with assertion tests
.projenrc.ts: register drs-ec2 (aws-cdk-lib >= 2.215 for the typed aws_arcregionswitch L1;
TypeScript pinned ~5.9 because a fresh resolve pulls TS 6, whose tsconfig defaults drop
@types/*), build/e2e/cleanup workflows (e2e = make deploy -> status -> rehearse-cycle LEGS=2).
lib/constructs/step-table.ts: the seven-step contract (mirrors lambda/drs_region_switch/__init__.py).
lib/constructs/drs-region-switch-steps.ts:
- DrsRegionSwitchSteps (per region): 7 functions from one asset selected by Handler string,
explicit 1-day log groups, orchestration role with the policy proven live (ViaAWSService
EC2 set, tag-scoped direct StopInstances for launch-into, PassRole to the app + DRS roles).
- DrsRegionSwitchPlanSteps (plan side): typed CfnPlan.StepProperty[] per workflow with the
in-RegionToRun function ARN (deterministic names, no cross-region refs), interleave helper
for the consumer's Aurora/DNS steps, grantInvoke for the plan's execution role.
test/: 10 assertion tests.
…step table and Lambda package as the CDK construct - layer built from lambda/drs_region_switch with archive_file source blocks (no local-exec), seven stub functions per region selected by handler string, explicit 1-day log groups; per-region submodule instantiated once per provider alias (aws.primary / aws.secondary) - orchestration role with the proven policy, statement-for-statement with the construct - outputs: function_arns by region, layer_arns + handlers (layer-only path), plan_role_policy_json, activate_secondary_steps / activate_primary_steps as ordered objects for dynamic "step", activate_primary_split - examples/plan: complete plan with Aurora Global + Route 53 blocks interleaved at the split - terraform fmt clean; validate passes against hashicorp/aws 6.64
…+ scripts bin/app.ts: context-driven wiring (repo idiom: Makefile reads outputs, passes -c key=value; -c stack= selector). lib/: network (x2), iam (app role only), database-primary/-secondary, app-primary (t2.small/BIOS, AWSDRS tag, internal ALB, 3-VPC private zone), alb-secondary, drs-steps (construct per region), plan (typed CfnPlan with Aurora + Route 53 + construct steps interleaved), observer. 11 stacks synth with placeholder context. Makefile: one target per deploy step (make expands $(shell) per recipe, so outputs must be read in a later target than the deploy that produces them); accept-peering via CLI; final pass binds ARC health checks. cleanup.sh: the proven parallel-wave teardown with the new stack names. scripts/: drs-setup, rehearse-*, tunnel from the demo (project/regions from env), plus app-code.sh (upload + in-place SSM refresh) and status.sh.
…ved NagSuppressions)
README follows the repo's section order (architecture, what's deployed, prerequisites, deployment, testing, findings, cleanup, cdk-nag suppressions, license). Diagrams + generator scripts carried over and re-rendered under the pattern name; all drs-mr-demo self-references renamed; GitLab mention dropped.
pg8000.dbapi: connect(database=, timeout=), paramstyle 'format' so the %s SQL is unchanged; its cursor is not a context manager, so the request handler closes cursor and connection in a finally.
…n is needed for make tunnel
…rong account create-drs-service-roles.py used boto3's default credential chain, not the profile the Makefile hands to drs-setup.sh. In the first live run it therefore inspected another account, printed 'exists role' for all six roles, and initialize-service then failed in the target account with 'Failed to attach the following IAM roles to their instance profiles' (found live 2026-09-16; the demo never hit it because AWS_PROFILE was always exported there). The script now takes [aws-profile], builds its session from it, and exits non-zero when the resolved account differs from the requested one.
…DB password
The app instance writes the password into a systemd unit (Environment=DB_PASSWORD=<value>) from a
shell heredoc. % is a systemd specifier, ' and \ are systemd quoting, $ and ` are shell expansions;
the original exclude set ('"@/\\ ') allowed all of them, so a redeploy whose password drew a %
produced a corrupted password in the process environment and a DB login failure (seen live in the
original demo's 2026-09-15 redeploy, diagnosed 2026-09-16). Exclude them at generation time.
…re stale source servers on instance replacement On AL2023 the AWS CLI is a system Python package whose python-dateutil RPM (2.8.1) is older than pg8000's requirement; 'pip3 install flask pg8000 boto3' upgraded it out from under the CLI and the next 'aws ssm get-parameter' in UserData died under set -e with "No module named 'dateutil'" -- the systemd unit was never written and the ALB answered 502 (first live run, 2026-09-16). The app's dependencies now live in /opt/app/venv and the unit runs the venv interpreter; the system CLI is untouched. psycopg2-binary never pulled dateutil forward, which is why the original never hit this. The UserData change replaces the instance, which strands the previous instance's DRS source server still tagged <project>:role=app -- and the plan's Lambdas select by that tag and must never see two. drs-setup.sh now disconnects and deletes any forward-replication server whose instance is not the current app instance before (re)installing the agent.
A UserData change on AWS::EC2::Instance is an in-place stop/start and cloud-init runs UserData only on first boot, so the venv fix (6835af3) never executed on the existing instance -- same id, still 502 (2026-09-16 14:50Z). Hash the boot script into the instance's logical id so CloudFormation replaces it, as the L2 Instance's userDataCausesReplacement does; drs-setup.sh already retires the old instance's source server.
Drop the `on.push.branches-ignore: [main]` override from the per-pattern build workflow generator. A workflow badge without ?branch= reports the latest run on the default branch, and with pushes to main excluded the only run the badge could ever see was a manual dispatch -- the aurora badge has shown a failed 2026-03-23 dispatch for six months while every branch build was green. Build now also fires on a push to main that touches <pattern>/**, i.e. once per merged change to that pattern. The job has contents:read only and no id-token, so it cannot reach AWS. The e2e workflows are untouched: they trigger on pull_request only and still never run on merge.
…nctions The harness decided whether to run the disk-marker round trip by reading a CloudFormation parameter named StatefulEc2 from the drs-steps-secondary stack. That parameter belonged to the original template-based deployment; the CDK port passes the flag as `-c statefulEc2=true` context and the construct bakes it into every step function's environment as STATEFUL_EC2. The describe-stacks query therefore returned an empty string, the log header read `stateful=` (blank), and a fully stateful plan execution ran with the stateless checks only -- no marker written on the serving instance, no marker check after fail-back. Observed live on the 2026-09-18 LEGS=2 run in the e2e account. Read STATEFUL_EC2 from the deployed drs-reverse-replicate function instead, which is what the plan actually runs with, and fail loudly if the value is anything other than true/false so the harness can never silently downgrade to the stateless assertions again.
…t file The app instance's boot script ran under `set -euxo pipefail`, so the Secrets Manager response and the extracted password were echoed into /var/log/cloud-init-output.log and the EC2 console output (readable with ec2:GetConsoleOutput). It then wrote `Environment=DB_PASSWORD=...` into /etc/systemd/system/drsapp.service, a world-readable (0644) file. Now xtrace is switched off while the secret is in flight, and the credentials are piped straight into /etc/drsapp.env created under umask 077, which the unit loads via EnvironmentFile=. The unit itself carries only non-secret settings. app.py is unchanged: it still reads DB_USER/DB_PASSWORD from its environment. Fetching the secret at application start was considered and rejected: the secret lives only in the primary region, so a recovered instance in the secondary region would depend on the impaired region to boot. The on-disk env file rides the DRS replica exactly as the unit file did. Pinned by test/app-primary-secret-handling.test.ts (2 tests, both red against the previous script).
23 *.pyc files under app/__pycache__/ and lambda/**/__pycache__/ were committed because the projen-generated .gitignore had no Python rules. Byte-code is machine-specific build output and has no place in a sample repository. Adds an optional per-pattern `gitignore` list to the projen Pattern interface, sets `__pycache__/` and `*.pyc` for drs-ec2 (the only pattern with Python code), regenerates drs-ec2/.gitignore, and removes the tracked files from the index. `git status` now shows the scripts/__pycache__ directory as ignored rather than untracked.
`pip install flask pg8000 boto3` resolved to whatever PyPI served on the day, so two deploys of the same commit could run different code and a compromised or broken release would be picked up silently. Pins: flask==3.1.3, pg8000==1.31.5, boto3==1.42.97. AL2023's default python3 is 3.9 and boto3 1.43+ requires 3.10, so 1.42.97 is the newest release the venv can take; the comment in the boot script says so for the next bump. THIRD-PARTY-LICENSES gains a Version column with the same numbers. Pinned by test/app-primary-dependency-pins.test.ts: every pip install in the boot script must be exact-version, and each pinned version must appear in THIRD-PARTY-LICENSES (red against the unpinned script).
Neither AWS::EC2::Instance set MetadataOptions, so both accepted IMDSv1 requests -- the SSRF-to-credentials path IMDSv2's session tokens close. Both now carry HttpTokens=required with the endpoint enabled. Every consumer on the boxes already speaks IMDSv2: app.py fetches a token before reading the region and instance id, the AWS CLI in the boot script, cloud-init and the SSM Agent do so natively, and Elastic Disaster Recovery supports IMDSv2-only instances (the FAQ documents it for recovery instances via the launch template). Pinned by test/ec2-imdsv2.test.ts: exactly two instances in the pattern, each requiring tokens (red against the previous stacks).
… drs:* The construct (and the Terraform module mirroring it) granted `drs:*` on `*`, flagged by cfn_nag (F4) and checkov (CKV_AWS_111). The seven step Lambdas call exactly these APIs: DescribeSourceServers, DescribeRecoveryInstances, DescribeJobs, DescribeJobLogItems, GetLaunchConfiguration, UpdateLaunchConfiguration, StartRecovery, ReverseReplication, StopFailback, StopReplication, TerminateRecoveryInstances, DisconnectSourceServer, DeleteSourceServer, DeleteRecoveryInstance, TagResource, UntagResource Resource stays `*`: source servers, recovery instances and jobs are created by DRS at run time and have no predictable ARN. test/drs-actions-contract.test.ts derives the list from the Lambda source (drs* client calls plus paginator names) and asserts the policy grants each by name with no wildcard, grants nothing the code does not call, and that terraform/iam.tf carries the identical set -- so a new boto3 call without a grant fails at `npx projen test`, not as an AccessDenied halfway through a live fail-over. Red against drs:*.
1,854 -> 1,082 words. Structured in the order the Open Source Program Office expects for a sample README: architecture, reuse, resources deployed, prerequisites, deploy, test and rehearse, DRS requirements, cleanup, security, limits, license. Removed: the project-history sections (Findings, "proven in", "measured on", "behaviours worth knowing", cycle-safety comparison) and the narrative failover story. Added: a Security section that points to the repository's security-issue process and states the demo's deviations in plain terms, and a cost notice. Preserved and re-verified against the previous text: every command (13 code lines), every timing figure, every stack row, region and CIDR, the cdk-nag exception list, and the DRS operational requirements. The DRS permissions bullet now names the enumerated action list instead of drs:*. Service names are spelled out on first use.
terraform/README.md: full service names on first use, imperative
instructions, no em-dash asides or rhetorical headings ("Layer-only
use" replaces the question), one action per table cell. Module inputs,
resource counts, step table, prerequisites and version floors are
unchanged and re-checked against versions.tf and examples/plan.
Root README: the drs-ec2 row spells out Amazon EC2, AWS Elastic
Disaster Recovery (AWS DRS), Amazon Application Recovery Controller
(ARC) Region switch, Amazon Aurora Global Database and Amazon Route 53,
and drops "cycle-safe" and "incl.".
…omers
Makefile: header names the services in full and drops the "protected,
dashboarded, failover-ready" claim; target descriptions say what each
target does with its duration ("the proven order" -> "deployment
order"; "Everything" -> the stacks, app code, DRS setup and check it
runs; "Tear down everything" -> what is deleted, where, how long).
Scripts: usage lines now match the arguments each script accepts
([aws-profile|-], optional legs/mode) -- cleanup.sh no longer calls
itself teardown.sh, drs-setup.sh loses the systemd-run wrapper note,
rehearse-switchover.sh and rehearse-cycle.sh describe the rehearsal and
its verification conditions instead of "proving" anything, tunnel.sh
keeps the plugin requirement and drops the laptop/ssh aside.
Diagrams: titles and captions spell out Amazon EC2, AWS Elastic
Disaster Recovery (DRS), Amazon Aurora Global Database, Amazon Route 53
and Amazon Application Recovery Controller (ARC) Region switch on first
use; "presenter" becomes "user"; the CLI-flag chatter leaves the ARC
node. Both PNGs regenerated from the edited generators (diagrams 0.x on
Graphviz) and checked visually; the mermaid sources carry the same
wording.
The suppression text still said drs:* and pointed at 'design.md s8', a planning document that is not part of this repository.
The enumerated grant in e65e362 was derived from the Lambda source, so it could not see the calls DRS makes under the caller's identity (forwarded access session) while it services StartRecovery and StartFailbackLaunch. CloudTrail for the 2026-09-18 rehearsals (run under drs:*) records three such calls against the orchestration role, invokedBy drs.amazonaws.com: drs:CreateRecoveryInstanceForDrs recover-ec2, failback-launch drs:ListTagsForResource recover-ec2, failback-launch drs:DescribeReplicationConfigurationTemplates reverse-replicate, reprotect Without them the recovery launch is denied after conversion. Added to the CDK construct and the Terraform module. The contract test now keeps two sets: the code-derived one and a closed SERVICE_FORWARDED list; every grant must belong to one of them, forwarded actions must not be called by the code, and a wildcard still fails. Proven red against e65e362's grant (fails on drs:CreateRecoveryInstanceForDrs) and green with this change. README count 16 -> 19 with the forwarded-call note for customers writing their own policy.
Contributor
Author
|
Intent: Add |
…ipts make The drs-ec2 e2e assumes arn:aws:iam::<account>:role/github-actions-drs-ec2, which nothing in the repository described; PR #76's first e2e run failed at credential assumption because the role does not exist yet. docs/iam/ now carries the role: a GitHub OIDC trust policy in the shape of the account's existing runner roles, and a least-privilege permissions policy derived from what the runner itself executes. cdk deploy runs without --role-arn, so the runner only assumes the CDK bootstrap roles in the three Regions (the model github-actions-aurora already uses live) and CloudFormation creates the stacks' resources under the bootstrap execution role. The rest is the direct surface of the Makefile, cleanup.sh, scripts/*.sh and the DRS service-role helper: 53 CLI calls across STS, CloudFormation, S3, EC2, IAM, SSM, DRS, ARC Region switch, RDS, ELBv2 and Lambda, plus the helper's four IAM writes, each bounded to the project's names where a name exists. The runner holds none of the DRS launch or reverse-replication authority; the plan's Lambda functions do, under their own role. test/github-actions-role-policy.test.ts derives every call from the scripts (including the Makefile's $(AWS) form and the helper's boto3 calls, excluding the SSM payloads that run on the instance) and fails when the policy lacks the matching action, when it carries a wildcard, when iam:PassRole widens beyond the app instance role, or when the helper's role/policy table and the policy's fences drift apart. Proven red by removing one grant. Known limits are in docs/iam/README.md: the DRS service-linked role on a first-ever initialization, iam:CreateRole on the six DRS service-role names, and Region-bounded rather than tag-bounded DRS and security-group grants.
…ed ones in cleanup The seven step log groups have fixed names (/aws/lambda/drsdemo-*) and CDK's LogGroup default is RemovalPolicy.RETAIN, so `make clean` left them behind and the next deploy's drs-steps change set failed early validation with 'already exists' on all seven (e2e run 35878845232, 2026-09-23). Set RemovalPolicy.DESTROY (Terraform already destroys them) and have cleanup.sh sweep any /aws/lambda/<project>-* group that outlived its stack, after the stacks are gone. Pin: every LogGroup in the construct carries DeletionPolicy/UpdateReplacePolicy Delete (red on Retain before the change).
…t peering accept cleanup.sh's new sweep needs logs:DescribeLogGroups and logs:DeleteLogGroup, the latter fenced to /aws/lambda/drsdemo-* in the two AWS DRS Regions (IAM simulator: project groups allowed, other names and Regions denied). The runner's ec2:AcceptVpcPeeringConnection grant listed only the vpc-peering-connection resource; IAM also evaluates the accepter vpc, so the call was denied (run 35878845232, 19:17Z). It was also redundant: CloudTrail shows CloudFormation accepting every same-account drs-ec2 peering itself while creating the AWS::EC2::VPCPeeringConnection, seconds before the Makefile step, on every deploy since 2026-09-16. Remove the Makefile accept-peering targets and the grant instead of widening it; pin both in the contract test.
…nstance profiles The e2e died at drs-setup: iam:AddRoleToInstanceProfile denied on all four DRS instance profiles (run 35915704468). DRS creates those profiles itself at path '/' during initialize-service (CloudTrail 2026-09-16 13:31Z, invokedBy drs.amazonaws.com), while the helper creates them at '/service-role/' and the grant named only that form; IAM evaluates the existing profile's ARN. The add call also requires iam:PassRole on the role (IAM API reference). - policy: GetInstanceProfile/CreateInstanceProfile/AddRoleToInstanceProfile on both ARN forms of the four profiles; PassRole of the four EC2-trust DRS roles to EC2; CreateServiceLinkedRole for AWSServiceRoleForElasticDisasterRecovery only, because CloudTrail shows initialize-service issuing it under the caller's identity on EVERY call (InvalidInputException once it exists), along with GetInstanceProfile x4. 29 statements. IAM simulator: project profiles allowed at both paths, other profiles denied; PassRole allowed to EC2 only and only for the four roles; only the DRS SLR. - helper: read the profile first and skip the add when the role is already attached, so an initialized account needs no IAM write here. - test: helper call set gains GetInstanceProfile; SERVICE_FORWARDED set carries the initialize-service calls with the evidence; PassRole pin allows exactly the two fenced statements; profile fence covers both path forms. Red x3 on the old policy, 96/96 green.
…g a replication template Third e2e (run 35926926216) reached drs-setup with all 11 stacks up and died at 'drs update-replication-configuration-template': AccessDeniedException 'Unable to verify subnet ... Ensure ec2:DescribeSubnets permission is granted'. DRS validates the template under the CALLER's identity. CloudTrail from the 09-16 setup under Admin gives the full forwarded set for create/update of a replication template: ec2:DescribeSubnets, ec2:DescribeSecurityGroups, ec2:GetEbsDefaultKmsKeyId, kms:DescribeKey on alias/aws/ebs, and a DRY RUN of ec2:CreateSecurityGroup when the default replication-server group does not exist (every run here: the staging VPC is new). IAM authorizes a dry run like the real call; the real group is created later under the DRS service-linked role. The 09-18 rehearsal + teardown show no other forwarded calls under the caller, so cleanup needs nothing more. The additions pushed the inline policy to 10,700 characters, over IAM's 10,240 aggregate quota for a role (Access Analyzer POLICY_SIZE_EXCEEDS_IDENTITY_POLICY_QUOTA; put-role-policy would have refused). Rewritten compactly with the same grants: short Sids, unscoped reads merged, the three DRS resource statements merged, project reads merged, security-group create+delete merged, and the redundant sts:GetCallerIdentity grant dropped (no permission needed). 21 statements, 9,352 characters. A grant-by-grant diff against the previous document shows nothing lost except GetCallerIdentity; the 'added' rows are cross-applications inside merged statements (IAM simulator confirms they are inert) plus the new grants. Tests: SERVICE_FORWARDED gains the five template-validation actions with the evidence; the security-group pin covers the merged statement; a size pin keeps the rendered policy under 9,800 so the next discovery fails in the test, not at the terminal; Sid references updated. 101/101. Access Analyzer zero findings. IAM simulator: every fence unchanged (profiles at both paths, PassRole to EC2 only, DRS SLR only, SG probe and kms read bounded to the two DRS Regions, no rule writes).
…le validating a template The dry-run ec2:CreateSecurityGroup that DRS issues under the caller's identity when creating or updating a replication configuration template carries a tagSpecificationSet (AWSElasticDisasterRecoveryManaged, aws:drs:managed, Name). EC2 authorizes tags on creation as a separate ec2:CreateTags on the security group being created, which the runner policy did not grant, so e2e run 35940960822 (attempt 2) failed at drs-setup: DRS reported "no permission for ec2:CreateSecurityGroup" while the CloudTrail errorMessage on the CreateSecurityGroup event names ec2:CreateTags on security-group/*. Grant ec2:CreateTags on security groups in the two DRS Regions, conditioned on ec2:CreateAction = CreateSecurityGroup so the runner can never retag an existing resource. 22 statements, 9,619 characters rendered. Contract test: ec2:CreateTags joins the SERVICE_FORWARDED set with the evidence; the no-tag-writes pin becomes a pin on the exact fence (one statement, two Regions' security-group/*, the CreateAction condition). Red on the previous policy, 103/103 green. Access Analyzer: zero findings. IAM simulator: allowed at CreateSecurityGroup time in us-west-2; denied with no CreateAction, in us-east-1, and at RunInstances time.
…re returning
`make deploy` is documented as ending at the resting state ("wait for CONTINUOUS"),
but drs-setup.sh returned as soon as the source server was tagged. The e2e runs
`make rehearse-cycle LEGS=2` immediately after, and its leg-0 baseline asserts the
resting-state invariant with a 5-minute settle budget sized for the minute DRS needs
after a fail-back, not for a fresh install's initial sync. Run 36028055691 (attempt 2,
2026-09-24): agent registered 18:03:04Z, rehearse-cycle started 18:04:22Z, failed at
18:11Z with "forward replication is INITIAL_SYNC" without executing the plan. The
server reached CONTINUOUS at 18:23Z, 20 minutes after registration.
Add step [7] to drs-setup.sh: poll dataReplicationState every 30 s, return on
CONTINUOUS, bounded at 45 min; fail loud (with dataReplicationError and the
non-succeeded initiation steps) on STALLED, DISCONNECTED, STOPPED or PAUSED, which
do not progress on their own. Uses only drs:DescribeSourceServers, already granted
to the runner role, so no policy change.
Contract test runs the exact loop under a fake `aws` that walks scripted states:
progress -> returns 0 at CONTINUOUS; STALLED/DISCONNECTED -> exit 1 fast; no
progress -> exit 1 after 90 polls. Pins the loop's position (after tag-resource)
and the Makefile promise. Red x4 against the previous script, 108/108 green.
… security group DRS creates the "default Replication Server Security Group" outside CloudFormation and terminates the replication server asynchronously after delete-source-server. The wave-3 sweep tried the delete once and moved on, so when DRS was slow the VPC delete failed with "has dependencies" on both del() attempts and the network stack was left DELETE_FAILED, which the next `make deploy` cannot update. e2e run 36040999672 (2026-09-24, the first run whose pre-flight cleanup had to retire a CONTINUOUS source server): sweep DependencyViolation 19:01:51Z, DRS TerminateInstances on the replication server 19:12:44Z, VPC DELETE_FAILED 19:31:20Z and 19:48:00Z, drsdemo-net-secondary left DELETE_FAILED, Deploy failed at net-1 three minutes later. sweep_drs_sgs now retries the delete every 15 s until every DRS-created group in the VPC is gone, bounded at 20 min, then warns and continues so the stack deletes still run. The DependencyViolation is the wait signal, so the runner role stays at ec2:DescribeSecurityGroups + ec2:DeleteSecurityGroup (no policy change). Contract test runs the function under a fake aws whose delete refuses N times: 45 refusals -> 46 deletes then success; 0 -> one delete; never -> 80 polls, WARN, exit 0. Red x3 against the previous script, 113/113 green.
…e types for every grant elasticloadbalancing:DescribeTargetHealth accepts no resource types, so IAM ignores the Resource element and an ARN-scoped grant denies at run time. The policy scoped it to the secondary target group's ARN; the rehearsal's leg-0 baseline (rehearse-cycle.sh, the first call to reach the target-group check under the runner) failed with AccessDenied on every one of its 20 polls in e2e run 36051833765 (2026-09-24 21:43Z). The IAM simulator returns implicitDeny for the previous grant with or without a resource ARN and allowed for the new one; Access Analyzer did not flag the old grant. Move the action to UnscopedReads; drop the targetgroup ARN from ProjectReads. 22 statements, 9,530 characters. No other change. Make the whole class fail at test time: test/fixtures/action-resource-types.json carries the AWS Service Reference's resource types for all 61 granted actions (refresh with test/fixtures/refresh-action-resource-types.mjs), and the contract test requires every granted action to be in the fixture and every ARN-scoped action to accept resource types. Red on the previous policy (exactly ProjectReads: DescribeTargetHealth), 162/162 green. README documents the rule and the refresh step.
… data path register-target waits for the recovered instance to report healthy, but /health returned ok unconditionally, so a recovered instance with a wrong security group, route or writer endpoint was marked healthy and the plan moved on. /health now runs SELECT 1 against the endpoint published in SSM and answers 503 on failure. It checks reachability and authentication only, not the writer role, so the primary does not flap during a planned Aurora switchover.
…etwork-online The boot script runs under set -e and depends on the package repositories, SSM, S3 and Secrets Manager. One transient failure ended it before the systemd unit existed, leaving nothing for Restart=always to restart. Every network-dependent call is wrapped in a bounded retry (5 attempts, 10 s growing), and the unit orders After=/Wants=network-online.target. The new test runs the retry function extracted from the synthesized script under bash. UserData changed, so the next deploy replaces the instance by design; drs-setup.sh retires the old source server.
drs-setup.sh: exit 1 when the instance is not SSM Online, when the agent never registers a source server (everything after that loop was skipped and make deploy exited 0 with nothing protected), and when the project tag does not land (it is the plan's selection contract; the write was best-effort and is now read back). rehearse-switchover.sh: completedWithExceptions (a skipped step) fails the leg and lists the skipped steps; the post-failover checks are PASS or FAIL and any FAIL exits non-zero after the fail-back has returned the estate to rest. Makefile help: status is informational; drs-setup takes 15 to 25 min live. New tests run the changed fragments under a fake aws; red against the previous scripts.
… discarding them stop_failback and disconnect were called through _try([], ...), a throwaway list, so their failures vanished. _try now logs every failure with the call and error, and best_effort=True is explicit for the two calls that may fail while DRS is already tearing the instance down. Two tests, red against the previous retire.py.
… DRS timing Prerequisites: deploy only into a sandbox account, never production or an account with other DRS workloads, because the fail-back retires every recovery instance and FAILBACK server in the two Regions. Security: the app runs as root and as the Aurora master user, and the demo console has no authentication. Deploy: DRS takes 15 to 25 min to CONTINUOUS (measured 2026-09-24/25), make deploy ~75 min. register-target: healthy now means the database is reachable.
The public workflow YAML no longer carries the test account id. E2E_ACCOUNT in .projenrc.ts is now the expression ${{ vars.E2E_ACCOUNT_ID }}, resolved by GitHub in the with: value of configure-aws-credentials; all six generated AWS-facing workflows regenerated. A fork sets one repository variable to point the workflows at its own account. Pin: the drs-ec2 workflows contain no 12-digit id and the IAM docs keep their placeholder.
…sources drs-retire listed every recovery instance in the secondary and every FAILBACK server in the primary and stopped, terminated, disconnected or deleted them, and register-failback-target picked the highest-id RUNNING recovery instance among all FAILBACK servers; the orchestration role holds those grants on '*'. A second DRS workload in the same account and Regions was retired by this plan's fail-back. Ownership now flows from the project tag: recovery instances are owned through the forward server that launched them (including a server a stateful re-protect tagged app-retired), FAILBACK servers through the owned recovery instance that feeds them, primary-side records through those FAILBACK servers or the protected primary. register-failback-target requires exactly one owned RUNNING fail-back instance instead of guessing by sort order. Five tests with an unrelated workload present; four are red against the previous code. README caveat updated.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The repository has no sample for the most common enterprise disaster-recovery shape: a stateful Amazon EC2 application that must fail over to a second AWS Region and, later, fail back onto the same instance with the data it wrote while away. Customers assemble AWS Elastic Disaster Recovery (AWS DRS), Amazon Aurora Global Database, Amazon Route 53 and Amazon Application Recovery Controller (ARC) Region switch by hand, and the fail-back half is where they get stuck.
Why it matters
Fail-back is the step DR drills skip. Without a worked example, teams either never rehearse it or discover the undocumented prerequisites (the launch-into target must be stopped, the disk must re-protect before retire) during a real event.
What changed
New
drs-ec2/pattern. One command (make deploy) builds eleven stacks acrossus-east-2,us-west-2andus-east-1: a Flask application on EC2 protected by AWS DRS, an Aurora Global Database, Route 53 failover records, an SSM Session Manager bastion, and an ARC Region switch plan whose steps are seven Lambda functions.make rehearse-cycle LEGS=2fails over and fails back without operator input; in stateful mode a file written on the DR instance rides reverse replication home onto the original EC2 instance id.The reusable part is
lib/constructs/drs-region-switch-steps.ts(CDK) and the equivalentterraform/module: the seven step functions, their orchestration role, and the ARC plan wiring. Everything else indrs-ec2/is the sample application and the harness around it (Makefile,scripts/,cleanup.sh,app/,lambda/,docs/,test/,THIRD-PARTY-LICENSES).The orchestration role grants the 19 DRS actions it needs, not
drs:*. Sixteen are derived from the Lambda source by a contract test; three (drs:CreateRecoveryInstanceForDrs,drs:ListTagsForResource,drs:DescribeReplicationConfigurationTemplates) are calls AWS DRS makes under the caller's identity during a launch, proven from CloudTrail and listed explicitly.Repository-level edits register the pattern:
.projenrc.ts(new pattern entry, per-patterngitignoreoption, and pattern build workflows now also run on pushes tomainso README badges trackmain), the generated.github/workflows/drs-ec2-build.yml,drs-ec2-e2e.ymlanddrs-ec2-cleanup.yml, a/drs-ec2entry in.github/dependabot.yml, the rootREADME.mdrow, and projen bookkeeping in.gitattributes,.gitignoreand.projen/files.json.aurora-build.ymlands3mrap-crr-build.ymllose theirbranches-ignore: [main]for the badge reason above; the build job hascontents: readonly and noid-token, so it cannot reach AWS.flowchart LR subgraph L2["Fail-back leg (stateful)"] A[drs-reverse-replicate]:::added --> B["drs-failback-launch<br/>lands on the original instance id"]:::added --> C[register-failback-target]:::added --> D[aurora-switchover-back]:::added --> E[dns-flip-back]:::added --> F[drs-reprotect]:::added --> G[drs-retire]:::added end classDef added fill:#DCFCE7,stroke:#16A34A🟩 added · 🟨 changed · 🟥 removed · 🟦 unchanged
Every step is idempotent and re-entrant under ARC's retry loop; each inspects live state, takes at most one action, and returns
RetryLateruntil its target state holds.Dependencies: new
drs-ec2/package.jsonandpackage-lock.json(CDK 2.215.0, cdk-nag 2.x, TypeScript 5.9); the EC2 application pinsflask==3.1.3,pg8000==1.31.5,boto3==1.42.97, all listed indrs-ec2/THIRD-PARTY-LICENSES.Tests
test/(Jest, 181 tests across 11 suites, including the role-policy contract below): stack synthesis in both stateful modes; the DRS action contract (code-derived set plus the three service-forwarded actions, no wildcard, Terraform grants the identical set); IMDSv2 required on both instances; the boot script never traces the database secret and the unit file carries no credential; pinned Python versions matchTHIRD-PARTY-LICENSES; no compiled byte-code tracked.lambda/tests/(pytest, 35 tests): each step's state machine against stubbed DRS/EC2/RDS responses, including ownership: an unrelated DRS workload in the same account and Regions is never selected, stopped, terminated or deleted by fail-back or retire.test/green-signals.test.ts,test/app-primary-boot-resilience.test.ts,test/workflow-account-id.test.ts: the changed shell fragments run under a fakeaws(drs-setup.shfails when no source server registers or the tag does not land;rehearse-switchover.shfails oncompletedWithExceptionsand on any failed post-check); the boot script'sretryfunction runs under bash; the published workflow YAML carries no account id.make lint: cdk-nag (AwsSolutions pack) with documented suppressions, ruff,bash -non every script.terraform fmt -checkandterraform validateon the module andexamples/plan.test/github-actions-role-policy.test.ts: derives every AWS call the e2e runner makes from the Makefile,cleanup.sh,scripts/*.shand the DRS service-role helper, and fails whendocs/iam/github-actions-role-policy.jsondoes not grant the matching action by name, carries a wildcard, or widensiam:PassRole.Manual verification
Proven live in the e2e account on 2026-09-18 from an empty account:
make deploy(~60 min), statelessrehearse-cycle LEGS=4, statefulrehearse-cycle LEGS=2with disk-marker evidence (2 marker(s) present: state carried across leg 2on the original instance id), and a one-passmake clean(33 min, zero residue). Leg timings are indrs-ec2/README.md.The e2e workflow assumes
arn:aws:iam::563688183446:role/github-actions-drs-ec2;docs/iam/holds its trust and least-privilege permissions policies and the commands that create it. The runner only assumes the CDK bootstrap roles forcdk deploy(the modelgithub-actions-auroraalready uses) plus the direct calls the scripts make, and the contract test pins that surface.Green end to end under that role on 2026-09-25 (e2e run 36065929282, head
3f591b1, 2h40m): pre-flight cleanup of a full prior estate incl. a CONTINUOUS source server 33m42s; Deploy 70m20s (11 stacks,drs-setupincl. its wait for CONTINUOUS);Resting state12s;rehearse-cycle LEGS=221m51s (baseline invariant held; leg 1us-west-2/a1d709fcdb2fb5b112m11s; leg 2 fail-backus-east-2/960150894322586c9m04s; invariant held again, primary EC2 identity preserved); cleanup 32m35s. Post-run account check: zerodrsdemo-*stacks in all three Regions, zero DRS source servers or recovery instances, zero/aws/lambda/drsdemo-*log groups, zero DRS-created security groups. The five earlier attempts each died one step further along, all in plumbing rather than the DR path (retained log groups, DRS instance-profile path, forwarded-access grants for the replication template, tags-on-create for the DRS security group, initial-sync timing, replication-server SG teardown ordering, and an ARN-scopedDescribeTargetHealthgrant IAM ignores); each fix carries a red-then-green test.Green again on the review-fix head on 2026-09-25 (e2e run 36186235669, head
d6e3ff1, 2h16m): pre-flight cleanup 1m30s; Deploy 76m57s (instance replaced by the new boot script,drs-setupasserted SSM Online, source-server registration and the tag read-back, then waited 24 min for CONTINUOUS);Resting state12s;rehearse-cycle LEGS=223m09s (baseline invariant held; leg 1us-west-2/09f89edbc66f345313m10s; leg 2 fail-backus-east-2/aa26784fdc1b6c7f9m25s; invariant held again; primary EC2i-079e8d8cdce981293preserved), which is the first live pass of the database-aware/healthbehindregister-targetand of the ownership-scopeddrs-retire; cleanup 32m34s. Post-run account check at 23:00Z: zerodrsdemo-*stacks in all three Regions, zero DRS source servers or recovery instances, zero/aws/lambda/drsdemo-*log groups, zero DRS-created security groups, zerodrsdemoinstances, global cluster and app-code bucket gone. The seven commits after3f591b1are the review fixes: database-aware health check, boot-script retries, fail-louddrs-setup.shandrehearse-switchover.sh, logged retire cleanup failures, workload-scoped fail-back and retire, the e2e account id moved to theE2E_ACCOUNT_IDrepository variable, and README notes (sandbox account only, demo privilege choices, measured timings).Security review: this sample orchestrates DRS recovery and holds
iam:PassRole; the OSPO security review is requested separately and is a merge gate.Screenshots
N/A: no user-visible UI. Architecture and fail-back diagrams are in
drs-ec2/docs/.no linked issue: new sample pattern, tracked outside GitHub.