The pipeline sounds like two lines. Plan on the pull request so a human can read it. Apply after merge so what shipped is what was reviewed.
Then you run it against real infrastructure and find the parts nobody mentions: a lock left behind by a cancelled job, a plan that was accurate forty minutes ago, an apply that got halfway and stopped, and a run-all that quietly touched eleven units when the diff changed one.
This assumes you already have a structure worth running. This is about what happens when CI drives it.
What the pipeline is actually for
Not automation. You could apply from a laptop.
It exists so that the change that ships is the change that was reviewed, applied with credentials no individual holds, leaving a record of who approved it. Every decision below follows from that, and any convenience that breaks it is not worth having.
Plan on the pull request, apply on merge
The split matters more than it looks.
On the pull request, run plan. Post the output where the reviewer will see it, as a comment rather than a link into CI logs that nobody opens. The plan is the artefact under review, not the HCL diff. A three line change to a variable can produce a plan that replaces a database.
After merge, run apply.
The rule worth enforcing in the pipeline definition itself: the job that can apply must not be reachable from a pull request, including from a fork, including via a manual trigger with a branch input. Every "just this once" path around this eventually gets used at 6pm on a Friday.
Two things to be honest about.
A plan is a snapshot. It was true when it ran. If the PR sits for two days, or another apply lands first, it is fiction. Re-plan immediately before apply and treat a material difference as a stop, not a warning.
Approving a plan is not the same as reading it. A plan showing 4 to add, 0 to change, 0 to destroy is scannable. A plan showing 60 changed resources is not, and everyone approves it anyway. That is an argument for smaller units, not for better discipline.
Only run what changed
At forty units, planning everything on every pull request takes long enough that people stop reading the output.
Scope the run to what the diff touched:
# Directories with changed .hcl files in this PR
git diff --name-only origin/main...HEAD \
| grep '\.hcl$' \
| xargs -r -n1 dirname \
| sort -u
Two adjustments before you trust it.
A change to root.hcl or env.hcl affects everything beneath it. If the diff touches a shared file, widen the scope to that whole subtree rather than the file's own directory.
A changed unit's dependents may also change. If a VPC unit's outputs shift, everything consuming them is affected. run-all on the subtree handles this; a per-directory loop does not.
Whatever you build, print the list of units the job selected before it runs them. That printed list is the real safety check, and it costs nothing.
Credentials: stop storing a key
If your pipeline has a long-lived AWS access key in its secrets, replace it with OIDC federation. GitHub Actions, GitLab and most CI providers support it.
The mechanics: CI presents a signed identity token, AWS trusts your CI provider as an identity provider, and a role gets assumed for the life of that job. The credential expires when the job ends.
Three reasons this is better, in order of how much they will matter to you:
- There is nothing to steal. A key in CI secrets can be exfiltrated by any workflow change that manages to print it. A token that expires in an hour and is bound to one repository is a much smaller prize.
- There is nothing to rotate. Key rotation is a task that gets skipped, and skipped rotation is how you end up with a three year old key in six pipelines.
- The trust policy is a real boundary. You can scope the role to a specific repository, a specific branch, and a specific environment. The production role can be made unassumable from a pull request context, which turns your earlier policy decision into something enforced rather than agreed.
Assume a different role per environment. Same reasoning as separate state buckets: a staging job should not hold credentials that can reach production, regardless of what it was asked to do.
Locks, and the job you cancelled
Terraform takes a state lock during plan and apply. A job killed mid-run, by a cancel button or a runner timeout, can leave that lock behind. The next run fails with a lock error and a lock ID.
Resist the reflex to add -lock=false, and resist automating force-unlock. Both convert a loud failure into a silent one, and the failure mode of a broken lock is two applies writing state at once.
Practical handling:
- Set a CI timeout longer than your slowest apply, so the runner does not kill jobs mid-write.
- Make cancellation rare. Do not auto-cancel in-progress apply runs when a new commit lands.
- When a lock is genuinely stale, unlock manually, by a person, after checking that nothing is still running.
Newer Terraform supports native S3 locking, which removes the separate lock table but not this problem. The lock still exists, it just lives elsewhere.
The apply that got halfway
This is the failure that actually hurts, and the one most pipelines have no answer for.
run-all apply across twelve units. Nine succeed. The tenth fails on a quota limit. Eleven and twelve never run.
Your infrastructure is now in a state that matches neither the old commit nor the new one. There is no rollback: the nine that succeeded are real, and reverting the branch means applying the inverse, which is its own risky change.
What helps:
Design for resume, not rollback. Fix the cause, run again. Terragrunt skips units already matching their configuration, so a second run converges. This only works if your modules are genuinely idempotent, which is worth verifying before you need it.
Fail the pipeline loudly and specifically. The notification should name which units applied and which did not. "Pipeline failed" sends someone digging through logs during the exact window when the answer matters.
Order by blast radius. Apply foundational units first. A failure after the VPC is done is recoverable. A failure that leaves half a VPC is worse.
Never mark it green. A partially applied merge is not success, and a pipeline that reports otherwise trains people to ignore it.
Pin versions so CI and laptops agree
If CI runs one Terragrunt version and an engineer runs another, you will eventually get a plan that differs between them, and you will spend an afternoon on it.
Pin the Terragrunt binary, the Terraform or OpenTofu binary, and every provider, in the repository rather than in the pipeline configuration. Terragrunt has a version constraint setting for exactly this. Providers get a lock file. Commit it.
The pipeline should install the pinned version, not the latest one. latest in a setup step means your infrastructure tooling upgrades itself on a schedule set by other people.
A minimal shape
on: pull_request
1. checkout, resolve changed unit directories
2. print the selected units
3. assume the read-only role via OIDC
4. terragrunt run-all plan on that scope
5. post the plan as a PR comment
on: push to main
1. checkout, resolve the same scope from the merge commit
2. assume the apply role via OIDC, scoped to this branch
3. re-plan, compare against what was approved
4. terragrunt run-all apply
5. on failure, notify with the exact list of applied and unapplied units
Nothing there is clever. The value is in what it makes impossible: applying from a pull request, applying an unreviewed plan, applying with a credential that outlives the job.
The part CI does not solve
A pipeline enforces the path a change takes. It does not detect a change that never took the path.
Someone fixes production in the console at 2am. Nothing in this pipeline notices. The next plan will show it, sometimes as a surprising deletion, and by then the person who made the change may not remember. That is drift, it is a separate problem, and a scheduled plan that reports differences is the usual first answer.
DevLift generates Terragrunt from your existing structure and opens a pull request rather than applying, so the plan is reviewed before anything changes and every apply carries a name. Book a walkthrough, or read the structure that holds up at ten environments.