Layer 1: The Landing Zone, and the Org-Service Enablement That Terraform Won't Do For You

Last week I walked through Layer 0 : the organization itself, OUs, four member accounts, guardrail SCPs, a Terraform state backend, and GitHub OIDC, all managed from the management account and nothing else. That layer was deliberately inward: everything it touched lived in one account.

Layer 1 is where the platform stops being a skeleton and grows a nervous system. This is the landing zone, the shared identity, audit, and security backbone that every layer above it, including the EKS platform in Layer 2, quietly depends on. And it’s the first time Terraform reaches across account boundaries and leans on organization-wide services with delegated administration. Those two facts drive almost every interesting decision, and every single one of the gotchas, in this post.

I’m going to be honest up front about where the real value of this write-up is. The architecture is clean and I’m proud of it, but you could reconstruct most of it from a good AWS blog. What you can’t easily reconstruct is the list of ways AWS Organizations service enablement will quietly defeat a Terraform apply, the exceptions with confident-sounding names that send you debugging the wrong thing. That’s the centerpiece. I’ll build up to it.

The cross-account execution model

The first decision Layer 1 forces is a question of identity: Terraform runs as one account, the management account, but it has to create resources inside other accounts. So how does it act inside an account it doesn’t belong to?

Terraform keeps entering as the management identity: locally that’s the SSO profile refplatform-mgmt, in CI it’s the management OIDC role from Layer 0. To actually do work inside a member account, a provider alias assumes that account’s OrganizationAccountAccessRole, the role Layer 0’s account vending dropped into every member account when it created them:

provider "aws" {
  alias  = "security"
  region = var.aws_region
  assume_role { role_arn = "arn:aws:iam::${local.account_ids.security}:role/OrganizationAccountAccessRole" }
}

The rule I hold to religiously here: account IDs are never hardcoded. Not in a tfvars file, not in a locals block, not in a comment. Every Layer 1 stack reads them from the org stack’s remote state:

data "terraform_remote_state" "org" {
  backend = "s3"
  config  = { bucket = var.state_bucket, key = "org/terraform.tfstate", region = var.aws_region }
}
locals { account_ids = data.terraform_remote_state.org.outputs.account_ids }

The only account-ID-shaped secret in the whole flow is var.state_bucket, whose name embeds the management account ID, and that comes in through the gitignored tfvars, the same no-account-IDs-in-git discipline from Layer 0. Everything downstream resolves from that one read.

There’s a nice property to this pattern that pays off at plan time: if a member account’s OrganizationAccountAccessRole were renamed or missing, the assume-role fails loudly and immediately, before anything is touched. The blast radius of a misconfigured account is exactly that account, surfaced clearly, not a half-applied mess.

Delegated administration: keep the management account boring

The second cross-cutting decision is where the org’s security services are administered. The AWS-blessed answer, and the one I follow, is: not the management account. The management account should stay minimal and boring; it’s the account with the keys to the entire kingdom, and you want as few moving parts in it as possible.

So for each org-wide security service, the management provider registers the dedicated security account as the delegated administrator, and from then on the service’s org-wide configuration is managed through the security provider alias:

The one exception is org CloudTrail, which must originate in the management account, so the trail is created there, but it writes to a bucket the security account owns. Which brings us to the audit backbone.

Centralized audit lands in the security account

The security account does double duty as the log archive. It’s low-privilege by design (no workloads run there), so putting the audit trail there means a compromise of a workload account can’t reach back and tamper with the evidence. (The audit trail as a diagram : every account, one write-isolated archive.) Two things live there:

• A KMS-encrypted, versioned, public-access-blocked S3 bucket receiving the org CloudTrail, management events from every account in the org. The bucket policy is scoped tightly to the org trail via an aws:SourceArn condition. (Versioned is not the same as immutable, which is why the audit trail later gained S3 Object Lock in WORM mode. A separate post.)
• The AWS Config organization aggregator, giving one pane of glass over resource configuration across all five accounts.

CloudTrail is deliberately limited to management events by default; that’s the free first copy. Data events (S3 object-level, Lambda invocations) are behind enable_data_events and default off, because they’re voluminous and they cost real money. That’s a running theme: every paid service in Layer 1 is behind an enable flag (enable_guardduty, enable_securityhub, enable_config, …) defaulting to true but flippable, so the whole platform can be stood down to near-zero between demos and rebuilt from code. AWS Config is the single largest variable cost and always the first knob I reach for when parking the platform.

The identity model

Identity Center lives in the management account, so terraform/identity is a management-only stack (no cross-account writes), which makes it the safe first exercise of the remote-state-read pattern. The model is deliberately small:

• Four permission sets: Administrator (AdministratorAccess), PowerUser (PowerUserAccess), ReadOnly (ViewOnlyAccess), Billing (Billing + Cost Explorer).
• Four groups: platform-admins, developers, auditors, billing.
• Access is granted to groups, never to users. Individuals get access by landing in a group; the permission sets attach to groups per account.

The default assignment matrix is expressed as a Terraform for_each over this:

Note the shape of the developers row: PowerUser in dev, ReadOnly in prod, and no access at all to the management, security, or shared-services accounts. That’s the whole point of the multi-account model expressed as one table. SSO stays the sole human access path; there are no IAM users anywhere in this platform, and the bootstrap admin from Layer 0 simply joins platform-admins.

The centerpiece: org-service enablement is imperative, and it bites

Here’s the thing nobody tells you clearly enough. When you enable a service like CloudTrail or GuardDuty in the console, AWS quietly does a bunch of AWS Organizations plumbing for you behind that friendly button: it enables trusted service access, sometimes registers delegated admin, sometimes creates service-linked roles. Terraform does none of that for you. The provider’s CreateTrail, CreateDetector, CreateAnalyzer calls assume the org plumbing is already there, and when it isn’t, they fail with exceptions whose names range from mildly helpful to actively misleading.

There’s no clean, standalone Terraform resource for most of this org-service enablement either. The one native path is the aws_organizations_organization singleton, which our org stack intentionally reads via a data source rather than manages. So the general prerequisite, run once per service, is imperative:

aws organizations enable-aws-service-access --service-principal <svc>.amazonaws.com

For this platform, the full set that has to be enabled is: sso, cloudtrail, guardduty, securityhub, access-analyzer, config, and config-multiaccountsetup. Miss any one and the corresponding apply falls over.

Let me tell you the specific stories, because the general rule doesn’t prepare you for how each one fails differently.

CloudTrail: CloudTrailAccessNotEnabledException

This was the first one, and the cleanest. The terraform/logging apply created seven of its eight resources happily, then the aws_cloudtrail resource threw:

CloudTrailAccessNotEnabledException: your organization hasn't enabled CloudTrail service access

Root cause, exactly as above: CreateTrail doesn’t auto-enable CloudTrail trusted access the way the console does. The fix was one command and a re-apply:

aws organizations enable-aws-service-access --service-principal cloudtrail.amazonaws.com

The re-apply is worth pausing on because it’s a pattern for the whole layer: because the other seven resources were already in state, the second apply created only the trail. Partial applies are normal here. You fix the one org-service quirk and re-run; Terraform picks up exactly where it left off. Don’t panic and destroy.

Access Analyzer: the SLR that needs to exist in the management account

This one is subtler and cost me real time. In terraform/security, creating the ORGANIZATION Access Analyzer from the delegated-admin (security) account threw:

ConflictException: Access Analyzer Service Linked Role is not in the organizational management account

Read that carefully. Enabling trusted access created the AWSServiceRoleForAccessAnalyzer service-linked role in the delegated-admin account, which is where I was creating the analyzer, so it looked like everything should be fine. But an org analyzer also requires that SLR to exist in the management account, and nothing had put it there.

The fix is to manage the management-account SLR explicitly, on the default (management) provider:

resource "aws_iam_service_linked_role" "access_analyzer" {
  aws_service_name = "access-analyzer.amazonaws.com"
}

And then a second, sneakier failure: even after creating the SLR, the analyzer still failed, because IAM is eventually consistent and the freshly created role wasn’t visible yet to the Access Analyzer control plane. This is the classic trap where your Terraform is correct and your apply still fails. The fix is a deliberate pause between the SLR and the analyzer:

resource "time_sleep" "wait_for_slr" {
  depends_on      = [aws_iam_service_linked_role.access_analyzer]
  create_duration = "30s"
}

I don’t love time_sleep. A hardcoded wait is a code smell: it papers over a timing problem by guessing a duration instead of checking for readiness. But IAM propagation has no “wait until ready” signal to check, so a deliberate pause is the honest tool. The analyzer then depends_on the sleep, so Terraform runs the wait before it tries to create the analyzer.

Security Hub: the legacy standard that silently enables itself

Security Hub’s gotcha isn’t a failure; it’s a silent success of the wrong thing, which is worse because verification is the only way you catch it. After apply, I checked the enabled standards expecting two (FSBP + CIS 1.4, the two I’d declared) and found three. A CIS 1.2.0 standard was enabled and completely unmanaged by my Terraform.

Root cause: aws_securityhub_account defaults enable_default_standards = true, and at enable time that auto-subscribes the legacy default standards, CIS 1.2.0 among them. So the fix is to flip it off:

resource "aws_securityhub_account" "this" {
  enable_default_standards = false
  lifecycle { ignore_changes = [enable_default_standards] }
}

The ignore_changes there is not decoration; it’s the actual trap. That argument is create-only / ForceNew: it only takes effect when the account is first enabled, and if Terraform ever decides to change it, it does so by destructively replacing aws_securityhub_account, which tears down your whole Security Hub configuration. So you set it false for fresh builds and then guard it with ignore_changes so Terraform never tries to reconcile it on an existing account. The already-subscribed CIS 1.2.0 I killed out-of-band:

aws securityhub batch-disable-standards --standards-subscription-arns <arn>

AWS Config: the exception that lied about being a propagation problem

This is my favorite, because I diagnosed it wrong for half an hour and the wrong diagnosis was completely reasonable.

Creating the Config organization aggregator threw:

OrganizationAccessDeniedException: This action can only be performed if you are a
registered delegated administrator for AWS Config...

Now, AWS Config genuinely lags Organizations. Registering a Config delegated administrator is honored by Organizations instantly but takes roughly 10 to 30 minutes to propagate into AWS Config itself, and during that window the aggregator legitimately fails with exactly this exception. I’d even documented that in the plan and added a time_sleep for it. So of course my first read was: propagation, just wait.

Except it didn’t clear. The error persisted well past 30 minutes, and I could reproduce it directly from the CLI, which propagation delay wouldn’t do indefinitely. The actual root cause: AWS Config’s delegated-admin check for the aggregator requires the account registered under config.amazonaws.com, and I had only registered it under config-multiaccountsetup.amazonaws.com. Config’s org features are split across two service principals, and the aggregator specifically wants the first one. The fix was to register the second principal too:

resource "aws_organizations_delegated_administrator" "config_service" {
  account_id        = local.account_ids.security
  service_principal = "config.amazonaws.com"
}

The aggregator created immediately after that. (I’d done the registration out-of-band first to unblock myself, so I terraform imported it afterward to keep code and state honest.) The lesson I took from this one isn’t about Config; it’s that a plausible known failure mode (propagation lag) will happily mask a different real bug, and “it looks exactly like the slow thing” is not proof it is the slow thing. Reproduce it directly and time it out.

One operational footnote from that debugging session: I’d left a background retry loop running against the aggregator, and it eventually died with InvalidGrantException, because a long-running background job outlives the local SSO token. aws sso login --profile refplatform-mgmt and re-run. Worth knowing before you trust an overnight retry loop.

CI/CD foundation: OIDC everywhere, a human gate on prod

Layer 0 gave GitHub Actions OIDC into the management account. To actually deploy workloads, CI needs to reach the workload accounts and a registry, still with zero stored credentials. The model (ADR-0005) is deliberately un-clever: each deployment-target account gets its own GitHub OIDC provider and its own github-actions-deploy role, assumed directly by the workflow with a short-lived token. No central hub role that chains into the others; that would be a long-lived, broadly-trusted thing to protect, and I’d rather not have one.

Security here comes from tight trust, not narrow permissions. The OIDC trust condition is scoped per account:

That last row is the interesting one. Prod’s token can only be minted for a GitHub Environment called prod, and that environment carries a required-reviewer protection rule, so a prod deploy physically cannot run until a human approves it, before the OIDC token is ever issued. It’s the CI equivalent of the plan/review/approve discipline I use on Terraform. That environment is repo config, not Terraform, reproduced with gh:

gh api --method PUT repos/<ORG>/<REPO>/environments/prod \
  --input - <<'JSON'
{ "wait_timer": 0, "reviewers": [{ "type": "User", "id": <YOUR_GH_USER_ID> }],
  "deployment_branch_policy": null }
JSON

The registry is a single central ECR in shared-services, with repository policies granting cross-account pull to the workload accounts (and later, EKS node roles), and lifecycle policies expiring old/untagged images so it doesn’t grow unbounded. One registry to build, scan, and sign into; pulled by many. The deploy roles themselves carry broad permissions (AdministratorAccess) for now; the exact set EKS/IRSA/networking needs isn’t known until Layer 2, and narrowing them to least-privilege is an explicit, tracked Layer 2 task. The safety today is the tight trust and the short-lived sessions, not permission breadth. (That task is now done: Layer 2 attached permission boundaries to every privileged principal, capping what these roles can reach even while the attached policy stays broad.)

Networking: centralized-egress hub-and-spoke

The network (ADR-0006) is built the way a real production multi-account platform runs one, not the shortcut: a centralized-egress hub-and-spoke anchored on a Transit Gateway in shared-services. (Diagram here , which is much easier to follow than prose.) The egress VPC in the hub owns the NAT tier; the workload VPCs have no NAT at all; their private subnets default-route 0.0.0.0/0 to the TGW, which forwards to the egress VPC’s NAT. One NAT tier for the whole org, which is both the production pattern and cheaper than NAT-per-VPC once you’re past three VPCs. The module defaults to one NAT per availability zone, the production-shaped answer, but this reference environment overrides it to a single NAT and takes roughly two thirds off the NAT bill. That seam between demo and production is a variable, not a shortcut.

Segmentation lives in the TGW route tables, not in security groups, a cleaner, auditable control. The workloads route table (dev and prod attached) has a default route to the egress attachment and no route between dev and prod: they’re isolated by construction. Addressing follows a simple IPAM plan: a 10.0.0.0/12 supernet, one /16 per account (10.0 hub, 10.1 dev, 10.2 prod, 10.3+ reserved), each workload /16 carved into per-AZ /19 private subnets sized for EKS pod density and small /24 public subnets for ALBs. Private AWS connectivity comes from free S3/DynamoDB gateway endpoints everywhere plus interface endpoints (ECR api+dkr, STS, CloudWatch Logs) that let nodes pull images and assume roles without traversing NAT. Those are implemented but off by default: they carry a flat hourly per-AZ charge and only earn it once real EKS nodes are running, which is Layer 2’s problem. And VPC Flow Logs from every VPC feed straight into the security log bucket, extending the same audit backbone down to the network layer.

Networking had its own org-service tax, consistent with the theme: attaching a cross-account Transit Gateway relies on AWS RAM org sharing, and there’s an onboarding lag when a member account is first brought into RAM’s resource-sharing; the share and the cross-account attachment don’t light up the instant you create them. Same shape of problem as everything else in this layer: the org-level enablement has to settle before the account-level resource will take.

What I’d tell someone forking this

The architecture (cross-account assume-role, delegated admin to a boring security account, group-based SSO, OIDC-per-account CI, hub-and-spoke egress) is reusable and, honestly, mostly the standard patterns done properly.

The thing that will actually cost you a day is org-service enablement. Terraform will not enable trusted service access or delegated administration for you the way the console does, the exceptions it throws when you forget are inconsistent and sometimes misleading, and at least one of them (OrganizationAccessDeniedException on the Config aggregator) looks exactly like a benign propagation delay while being a genuine second-service-principal bug. Pre-enable every service principal before you apply, expect partial applies and just re-run after each fix, reach for time_sleep when IAM eventual consistency bites, and guard the ForceNew arguments so a future plan doesn’t destructively replace your security services. Do that and the landing zone comes up clean.

Next in the series: Layer 2, where all of this (the network, the registry, the deploy roles, the audit backbone) finally gets an EKS platform dropped on top of it.

• AWS
• AWS Organizations
• Terraform
• CloudTrail
• GuardDuty
• Security
• Networking
• DevOps
• SRE