Last week I laid out the whole platform : five accounts, an EKS-based internal developer platform, built in the open in layers. This week I want to go all the way down to the floor, Layer 0, the org bootstrap, because it’s where the most interesting tension in any infrastructure-as-code project lives: something has to exist before Terraform can run, and I refuse to store a secret in a public repo to make that happen.

Layer 0 is the accounts, the OUs, the SCPs, the Terraform state backend, and the GitHub OIDC federation. It’s the least glamorous layer and, honestly, the one I’m most careful about, because everything above it inherits its mistakes. This post is the honest version, including the manual steps I can’t code away and the four things that didn’t go exactly to plan.

The chicken-and-egg problem

Infrastructure as code has a founding paradox: the tool that manages your infrastructure needs infrastructure to run. Terraform wants to store its state in S3. But the S3 bucket doesn’t exist yet, and the thing that creates it is the same Terraform run that needs somewhere to put its state. You can’t remote-store the state of the stack that creates your remote store.

There’s no clever trick that makes this disappear. There’s only being honest about it. So docs/bootstrap.md is a deliberately short, deliberately manual list of six ClickOps steps, and everything after step 6 is code:

  1. Create the management account (dedicated email alias, business payment method).
  2. Lock down root: MFA immediately, no IAM users, no access keys, and then never touch root again except for the handful of tasks that require it .
  3. Enable AWS Organizations (all features).
  4. Enable IAM Identity Center in the home region; create the admin user; from here on all human access is SSO via aws configure sso.
  5. Verify email aliasing (you+aws-mgmt@…, you+aws-security@…, and so on, one inbox, many addresses).
  6. Run Terraform locally to create the state backend, then migrate.

That’s the whole manual surface. The management account exists to run three things: Organizations, Identity Center, and billing, and nothing else. Everything past that boundary is Terraform.

Why raw Organizations instead of Control Tower

The default enterprise answer here is AWS Control Tower with Account Factory. I chose raw AWS Organizations managed directly in Terraform, with OUs, member accounts, and SCPs as first-class resources in terraform/org/. That decision is written up in ADR-0002 , and the reasoning matters:

  • Forkability. A Control Tower landing zone can’t be forked. A Terraform org module can. The entire point of this platform is that you can clone it, set your variables, and reproduce the structure. Control Tower is fundamentally not that.
  • Legibility. Control Tower provisions a lot of machinery behind the scenes. A reference platform should show the machinery, not hide it. If you want to understand SCP evaluation or account vending, Control Tower is exactly the wrong teacher, because it abstracts away the parts worth learning.
  • Scale honesty. Control Tower earns its complexity at enterprise account counts. At five accounts, it’s overhead without payoff.

The trade-off is real and I own it: there’s no Account Factory, so account vending is a Terraform resource rather than a self-service catalog, and guardrails are SCPs I wrote by hand, fewer of them, but fully visible. If this platform ever needed dozens of accounts, I’d revisit the call, and that revisit would get its own ADR.

The account and OU structure

The shape is small and opinionated (there’s an interactive account landscape if you want to click through what lives where):

Management (org root: Organizations, Identity Center, billing only)
├── Security OU
│   └── security        (GuardDuty/Security Hub delegated admin, log archive)
├── Infrastructure OU
│   └── shared-services (CI/CD roles, ECR, networking hub)
└── Workloads OU
    ├── workloads-dev
    └── workloads-prod

In terraform/org/main.tf, the three OUs hang off the org root, and four member accounts get vended into them via aws_organizations_account. A couple of deliberate details: close_on_deletion is left false because closing an account is a human decision, not a terraform destroy side effect; and root passwords on vended accounts are never set. If I ever need root on a member account, I use the reset flow against its alias, set MFA, and walk away.

The guardrails live in terraform/org/scps.tf. I keep them few, valuable, and firm: an SCP is a ceiling, not a grant, so the set stays small and readable. There are exactly three managed policies, each attached to all three top-level OUs via a setproduct fan-out:

  • region-allowlist: deny any regional API call outside the approved regions, with global services (IAM, Organizations, STS, CloudFront, Route 53, billing, WAF, Shield) exempted via not_actions.
  • deny-root-user: deny everything when the principal ARN is a member account’s root.
  • deny-leave-and-close: deny organizations:LeaveOrganization and account:CloseAccount.

I’ll come back to that third one, because it has a story.

On regions: ADR-0003 pins the home region to us-east-1 (where IAM, CloudFront, Route 53, and billing are anchored anyway, and where Identity Center is enabled, and Identity Center’s home region is effectively permanent, so it was a deliberate choice, not a default). The allowlist permits us-east-1 plus us-east-2 as an adjacent DR region, so workloads have somewhere to fail over to without editing the guardrail later. Any new region is a conscious SCP change, by design.

The standout engineering detail: a partial S3 backend that keeps the account ID out of git

Here’s the piece I’m proudest of, and it started as a conflict between two hard rules.

The state bucket name embeds the management account ID. In terraform/bootstrap/main.tf it’s literally "${var.name_prefix}-tf-state-${data.aws_caller_identity.current.account_id}". That’s good hygiene: globally-unique, unguessable, tied to the account. But the backend block that points at that bucket lives in versions.tf, which is committed to a public repo. And the platform’s hardest rule is: never commit account IDs. Terraform backend blocks can’t interpolate variables, so I couldn’t just reference var.something and move on.

The resolution is a partial backend configuration. versions.tf holds only the non-sensitive settings; the account-specific bucket is supplied at init time from a gitignored backend.hcl. Here’s the committed part:

# terraform/bootstrap/versions.tf
terraform {
  required_version = ">= 1.10"

  backend "s3" {
    key          = "bootstrap/terraform.tfstate"
    region       = "us-east-1"
    use_lockfile = true
    encrypt      = true
  }
}

Notice what’s not there: no bucket. That comes from backend.hcl, which is gitignored, and which every fork recreates from a committed backend.hcl.example carrying a <MGMT_ACCOUNT_ID> placeholder:

# backend.hcl (gitignored)
bucket = "refplatform-tf-state-123456789012"

Every terraform init, in every stack, is then:

terraform init -backend-config=backend.hcl

This mirrors the terraform.tfvars / terraform.tfvars.example convention already in the repo, so the pattern is consistent and forkable rather than a one-off hack. The account ID exists in exactly one place on my machine and zero places in version control.

Two more things worth calling out in that same block:

S3 native state locking, no DynamoDB. use_lockfile = true (Terraform ≥ 1.10) uses conditional writes on the state object itself for locking. For years the standard answer was a DynamoDB lock table, an extra resource, extra IAM, extra thing to forget. That’s gone. One less moving part in the most fragile layer of the stack.

The bootstrap stack is the one exception to the init flow. Because it creates the bucket, its first run uses local state, then migrates in with terraform init -backend-config=backend.hcl -migrate-state. The org stack has no such dependency (the bucket already exists), so I initialized it directly against S3 from its very first init. That keeps org state, which contains member account IDs, off local disk entirely.

Zero stored AWS credentials: GitHub OIDC

The other half of the bootstrap stack is federation. (Here’s the credential flow as a diagram , showing how CI reaches AWS with nothing stored.) The whole platform runs on a single design principle: there are no IAM users and no access keys anywhere in this organization. Humans authenticate via Identity Center SSO. Workloads use IAM roles. And GitHub Actions authenticates via OIDC.

terraform/bootstrap/github-oidc.tf creates an aws_iam_openid_connect_provider for token.actions.githubusercontent.com and an IAM role whose trust policy is scoped to this specific repo:

condition {
  test     = "StringLike"
  variable = "token.actions.githubusercontent.com:sub"
  values   = ["repo:${var.github_org}/${var.github_repo}:*"]
}

GitHub Actions assumes that role at runtime with a short-lived token. There is nothing to leak, nothing to rotate, nothing sitting in a secrets store waiting to be exfiltrated.

I’ll be candid about the current shape: that role attaches AdministratorAccess, and the trust sub is scoped to :*, any branch or PR. This is the widest it will ever be, and it’s a deliberate Layer 0 starting point, not the end state. Narrowing to specific branches/environments (repo:ORG/REPO:ref:refs/heads/main, repo:ORG/REPO:environment:prod) and least-privilege policies is tracked for a later layer. (It got done: Layer 2 put permission boundaries on every privileged principal, which I’ll cover when I get there.) Which brings up the deployment posture: every Layer 0 apply ran locally, behind a human SSO approval gate. The GitHub Actions workflow (terraform-plan.yml) runs plan only, on pull requests, via the OIDC role. It never applies. Automated apply-via-CI, with environment protection rules, is deferred to Layer 1 as a design choice. The role and the AWS_ROLE_ARN repo variable are already in place so CI can take over the moment I make that call.

The war stories

Layer 0 was smoother than I expected, mostly reconciliations and decisions rather than hard API failures. But the reconciliations are the interesting part, because they’re where “everything is code” stops being a slogan and starts being work. All four are logged in docs/layer0-issues.md.

1. The invisible SCP. Post-apply, I verified the org’s SCPs and found a fourth policy, DenyLeaveAndCloseAccount, attached to the org Root, denying organizations:LeaveOrganization and account:CloseAccount. It wasn’t in Terraform. It wasn’t in the repo. The console’s “enable all features” flow had auto-created and attached it as a recommended guardrail. A sensible policy, but an invisible one, a guardrail nobody reading the repo could see, which is exactly the failure mode “everything is code” exists to prevent. Rather than import it, I folded its intent into my own managed policy: I extended deny-leave-org to also deny account:CloseAccount, renamed it deny-leave-and-close so it’s a strict superset of the console version, then detached and deleted the console SCP. The org now has exactly three managed SCPs plus the default FullAWSAccess, and every one of them is in git. (A fourth, an audit-protection SCP that stops anyone disabling the audit backbone, arrives later when I harden the platform for audit-readiness.)

2. The account ID that almost leaked. This is the partial-backend story from above, but it’s worth naming as an issue and not just a design flourish, because it started as a genuine conflict: the handoff notes literally said “hardcode the bucket name in versions.tf,” and doing that would have committed the management account ID to a public repo. The partial backend config is the fix, and it’s now the standard pattern for every stack in the platform.

3. The quota that didn’t fire. New organizations often ship with a low default quota (~4 to 5) on concurrent account creation, and my plan creates four accounts at once. I fully expected a CONSTRAINT_VIOLATION. It never came. All four vended cleanly, with workloads-dev simply taking about 80 seconds to come up asynchronously. No Service Quotas increase, no Support ticket. I’m documenting it anyway, because a fork might hit the limit, and if you do, request an increase for “Accounts” under AWS Organizations in Service Quotas and re-apply; Terraform will create only the accounts that don’t yet exist.

4. The gitignored lockfile. My initial .gitignore ignored .terraform.lock.hcl. That’s backwards. Terraform’s own recommendation is to commit the lockfile so provider versions and checksums are pinned in VCS; ignoring it is a reproducibility gap, especially for CI, which would resolve providers differently than my laptop. I un-ignored the lockfiles and regenerated them with multi-platform checksums so CI (Linux) and both flavors of dev Mac agree:

terraform providers lock \
  -platform=linux_amd64 \
  -platform=darwin_amd64 \
  -platform=darwin_arm64

The .gitignore now carries an explicit comment saying the lockfile is committed on purpose, because the natural instinct is to ignore it, and I wanted the next person (probably me) to not “fix” it back.

One nice property of Layer 0: it hit no trusted-access or delegated-admin failures, because it only ever operates in the management account and doesn’t enable any org-wide services. That whole class of pain starts in Layer 1, which is exactly where I’m headed next week.

What Layer 0 buys

When this layer is done, I have five accounts in a legible OU tree, three hand-written guardrails I can read in a single file, a state backend that locks itself without a DynamoDB table, and a CI identity with zero stored credentials, and not one account ID committed to a public repo. Every decision above is written down in an ADR, so the reasoning outlives my memory of it.

That’s the floor. Next week: Layer 1, the landing zone, identity, logging, security tooling, and the first real taste of org-wide trusted access, which is where the deployment log gets a lot more exciting.

Share:
Enjoyed this? Buy me a coffee

Comments