Somebody is setting up a new environment. Staging already works, so they open its configuration, select all, copy, and paste into the new one. Then they go through and change the values that look environment specific.
That last sentence is the whole problem. "The ones that look environment specific" is a judgement made at speed, by a person who did not write most of these keys, about a list that has grown for two years.
The failures this produces are quiet ones. Not a crash on boot, which you would catch. A service that starts cleanly, answers health checks, and writes to the wrong database for six hours.
What actually goes wrong when you copy?
Four things, in roughly the order you meet them.
A value points at the wrong environment and nothing complains. A queue URL, a bucket name, a database host. The application does not know these are wrong. It connects successfully, because the resource exists, and it is the wrong one. This is the expensive failure and it is the one copying causes most.
A key is missing and the default is worse than an error. Someone adds FEATURE_STRICT_VALIDATION=true in staging and forgets production. Production picks up the code default, which is false, and now validation is off in the only place it mattered. A missing key that crashes on boot is a good outcome. A missing key with a permissive fallback is an incident waiting for traffic.
A stale key stays forever. Nobody deletes anything, because deleting a variable you did not add is how you break something at 2am. So the list grows, and every subsequent copy carries the accumulated debris of every previous one. Environments end up with keys no code has read since last year, and nobody can tell which those are.
A secret is now in two places. Rotating it means remembering both. Somebody will not.
Which values should never be copied?
Sort every key into three buckets before you automate anything. The sorting is the work; the tooling is easy afterwards.
Same everywhere. Log format, feature toggles that are genuinely global, a public API version. These are safe to copy and arguably should not be per environment at all: if a value is identical in five places, it is a constant living in the wrong file.
Different per environment, by design. Endpoints, bucket names, queue URLs, replica counts, log levels, timeouts. These must exist in every environment and must hold different values. These are the ones that need a diff, not a copy. You want to know that a key exists in both places, and you specifically do not want the values to match.
Never copied. Production credentials, signing keys, anything a regulator would ask about. These should not be readable by the person setting up staging, which means the copy operation should not be able to see them either.
The bucket that causes incidents is the middle one, because it is the only one where copy looks correct and is not.
Where should the values live?
Not in your configuration UI's database, and not in a repository.
The pattern that holds up: values live in the platform's own secret store, and everything else holds references. On AWS that means SSM Parameter Store or Secrets Manager. Your deployment configuration says DB_PASSWORD -> /prod/payments/db_password, and the value itself is fetched at deploy or at runtime by something holding an IAM role.
Three properties fall out of that, and they are why it is worth the indirection:
The value has one home. Rotation is one write. Nothing needs to be updated in six other systems, so nothing gets missed.
Access is IAM, not application logic. Who can read the production password is answered by a policy you can audit, not by who has a login to your dashboard.
Your config is safe to look at. A list of parameter paths can be screenshotted, pasted into a ticket and committed to git. A list of values cannot. That difference makes everyday work dramatically less fraught.
What does promotion mean if it is not copying?
Copying moves values. Promotion moves the shape: the set of keys, and the intent behind each one.
Concretely, promoting staging to production should:
- List the keys that exist in staging and not in production.
- For each one, ask which bucket it is in.
- Copy only the "same everywhere" values.
- For "different per environment", create the key and require a value. Do not guess, and do not carry the staging value over as a default, because a default that works is a default nobody replaces.
- For "never copied", create a reference to a production secret path and leave the value to whoever holds that access.
The output of a promotion is not a completed environment. It is a list of decisions somebody has to make, most of which are one word long. That is the correct output, and it is slower than a copy button by about four minutes, once.
How do you tell config from secrets when the list is already a mess?
You will not do this perfectly, so do it by consequence rather than by category.
Ask one question per key: if this value appeared in a screenshot in a public ticket, would anyone care? That splits the list faster than trying to classify by name, and it catches the cases naming conventions miss, like a webhook URL with a token in the path, or a connection string with a password embedded in it.
Two patterns worth grepping for specifically:
- Keys ending in
_URLthat contain credentials. Slack incoming webhooks are the classic: the whole secret is in the path, which also means header-based secret injection cannot protect them. - Keys named
*_CONFIGor*_JSONholding a blob. Nobody audits inside a blob, so secrets accumulate there.
Who should be able to read production values?
Fewer people than can deploy to production, which surprises teams the first time they say it out loud.
Deploying is reviewed, logged and reversible. Reading a credential is none of those. Someone who can ship a change to production through a pull request has left a trail; someone who read the database password has not, unless you built the trail.
Two rules that carry most of the value:
Write-only in the interface people use daily. Setting a value is a normal operation. Reading one back should be rare, deliberate and logged. Most platform UIs get this backwards by default, showing values with a reveal button, because it is convenient.
Separate the environments' access, not just their data. A role that can read staging secrets should not be able to read production ones. If the same role reads both, the boundary between environments is decoration.
What breaks when a variable is missing?
This is worth deciding deliberately, because the default is bad.
Most frameworks read an absent variable as an empty string or a language default, and continue. That turns a configuration error into a behavioural one, discovered later and further from the cause.
Fail loudly at startup instead. Validate the full expected set of keys before the application accepts traffic, and exit if any required one is absent. A container that will not start is a five minute problem. A container that started with PAYMENT_TIMEOUT unset is a much longer one.
The same argument applies to the platform layer: a deploy that would leave a required key unset should not proceed. That check belongs alongside the other things a pipeline should catch before a human sees it.
How do you know what changed?
You need two things, and most setups have neither.
A diff between environments. Not the values, the keys and the deliberate differences. "Production has 47 keys, staging has 51, here are the 4 extra and here are the 6 whose values differ by design." That report answers most configuration questions before anyone has to ask them.
A history of who set what, when. Not the value, the fact of the change. When a service starts behaving differently at 14:00 and nothing deployed, the answer is frequently that somebody changed a variable at 13:58. Without a record, that takes an hour to find. With one, it takes a minute.
This is the same class of problem as configuration drift in infrastructure: the state of the system stopped matching what anyone believes about it, and nothing announced it.
The short version
Do not build a copy button. Build a diff.
Sort your keys into same everywhere, different by design, and never copied. Keep values in the platform's secret store and references everywhere else. Fail loudly on a missing key. Keep a record of who changed what.
The copy button is the feature everyone asks for, and it is the one that eventually writes staging's queue URL into production on a Friday afternoon.
DevLift manages environment variables and secrets as references into SSM and Secrets Manager rather than storing values, so rotation happens in one place and access stays an IAM question. Book a walkthrough.