Teams put a vault in place, feel safe, and stop thinking. But a secret sitting in a vault is not the risk. The risk is the copy that the app pulled at boot and left in an environment variable, the one in the CI log, and the one a developer pasted into a config three months ago that nobody can rotate without breaking production.
The vault is plumbing, rotation is the product
A secret you cannot rotate in under an hour without a deploy is a liability, full stop. We learned this when a database password leaked and the rotation took two days because four services read it from baked images. Now the test for any new secret is simple: can we change this value right now and have everything pick it up within minutes? If not, it goes back.
- Inject at runtime, never bake into images or commit to config
- Give every secret an owner and a known rotation cadence, even if it is yearly
- Prefer short-lived credentials a workload requests on demand over long-lived keys
- Keep an inventory of who reads each secret, so revocation is not archaeology
Delivery is where it leaks
Most leaks we have cleaned up were not breaches of the vault. They were the handoff. A secret printed in a build log, an env dump in an error report, a backup that included the rendered config. We treat the path from vault to process as the dangerous part and audit it specifically: scrub logs, block env dumps in error handlers, and never let a secret transit through a place that gets archived.
Assume every secret will leak once. Design so that a leaked value is cheap to revoke, not catastrophic to live with.
Short-lived beats well-guarded
The best secret is one that expires before anyone can use the leaked copy. Where the platform allows it, we moved from static keys to workload identity that mints a credential good for fifteen minutes. A scraped fifteen-minute token is a non-event. A scraped permanent key is an incident and a weekend.
The boring discipline that pays off
We run a quarterly drill where we rotate a real production secret on purpose, during business hours, and time it. The first drill is always embarrassing. By the third one, rotation is a non-event, which is exactly the state you want to be in before the day it is not a drill.