Secrets and Key Management

Lesson 6 ended on a crucial point: encryption is only as strong as the key, and signing is only as trustworthy as the signing key. That makes secrets management — how you handle API keys, database passwords, tokens, and cryptographic keys — one of the highest-leverage security topics there is. A perfect encryption scheme with the key committed to GitHub is worthless. And secrets leak in remarkably mundane ways, which is good, because mundane problems have mundane, learnable fixes.

How Secrets Leak (It’s Always the Boring Way)

Before the fixes, know your enemy. Secrets rarely leak through sophisticated attacks; they leak through everyday developer convenience.

The usual suspects:

• Hardcoded in source and pushed to git. The classic. And deleting the line in a later commit does not remove it — it’s still in the history, retrievable by anyone with the repo. Public-repo secrets are scanned and abused by bots within minutes of being pushed.
• Printed in logs and error traces. A config dump, an exception that includes the connection string, a debug log of the request headers (with the auth token). Logs are widely readable and long-lived (Lesson 11), so a secret in a log is a leaked secret.
• Pasted in Slack, emails, tickets, screenshots. Convenient, and permanent in someone’s chat history.
• Shipped in client code. A mobile app or JavaScript bundle is fully inspectable; any “secret” embedded in it is public. (A secret the client needs is a sign the architecture is wrong — the call should go through your backend.)

The Managed Lifecycle

The fix is to stop treating secrets as strings you scatter around and start treating them as managed objects with a lifecycle: generate → store → inject → rotate → audit.

• Generate strong, random secrets from a CSPRNG (Lesson 6). Don’t reuse one secret across environments or services.
• Store them in a proper secrets manager — HashiCorp Vault, AWS/GCP/Azure Secrets Manager, or your platform’s encrypted secret store. These encrypt secrets at rest, control who can read them, and log access. If you’re small and can’t run a full vault yet, the minimum bar is environment variables injected by your platform at deploy time — never a .env file committed to the repo.
• Inject at runtime, not at build time. The application receives the secret from the environment or a mounted file when it starts; the secret never lives in the code, the image, or the repo.
• Rotate on a schedule and immediately on any suspected exposure. Short-lived, automatically rotated credentials (dynamic secrets, cloud role credentials) are best — a secret that’s only valid for an hour is far less valuable to steal.
• Audit and revoke. Know who (and which service) read which secret and when, so you can investigate and cut off access fast. The ability to revoke instantly is as important as the ability to issue.

Two rules of thumb sit on top of this:

• Automate secret scanning. Run a scanner (gitleaks, trufflehog) in CI and enable push protection (GitHub and others offer it) so a committed secret is caught before it ever reaches the shared history. This turns the most common leak into a blocked commit.
• A secret that ever touched git history is compromised — rotate it. Don’t just remove the line and move on. Assume it was captured (bots are fast), revoke it, and issue a new one. Scrubbing history is good hygiene but rotation is the actual fix.

Envelope Encryption: Protecting Data Without Handling the Master Key

When you need to encrypt data at rest (sensitive database fields, files), you face a bootstrapping problem: the data is protected by a key, but then that key needs protecting, and so on. The industry-standard answer is envelope encryption, which is how cloud KMS (Key Management Service) works — and it’s worth understanding because it shows up everywhere.

The scheme uses two layers of keys:

• A Data Encryption Key (DEK) — a fresh, random key that actually encrypts your data (fast symmetric encryption). It’s stored right next to the ciphertext, but it is itself encrypted.
• A Key Encryption Key (KEK), the master key, which lives inside the KMS / HSM and never leaves it. The KEK’s only job is to encrypt (“wrap”) the DEKs.

This indirection buys three big properties:

1. A server compromise doesn’t hand over the master key. Your application only ever holds wrapped DEKs and asks the KMS to unwrap them. The KEK — the key that could decrypt everything — never touches your servers, so stealing a server doesn’t steal the crown jewels.
2. Rotation stays cheap. To rotate the master key you re-wrap the small DEKs, not re-encrypt terabytes of data. This makes regular key rotation actually feasible instead of a dreaded megaproject.
3. Central audit and revocation. Every decrypt goes through the KMS, so you get a log of who decrypted what, and you can revoke access in one place.

The best part: as an engineer you mostly just call encrypt()/decrypt() (or use envelope encryption helpers) against the KMS, and the two-layer dance is handled for you. You get strong key management without personally juggling master keys — which, given how badly manual key handling tends to go, is exactly what you want.

A Few More Habits

• Least privilege for secrets, too (Lesson 3, Lesson 10). A service should be able to read only the secrets it needs, not the whole vault.
• Don’t log secrets, ever (Lesson 11). Scrub tokens, keys, and passwords from logs and error reports; many logging libraries support redaction filters.
• Separate secrets per environment. A leaked staging key should never unlock production.
• Prefer workload identity over long-lived keys. On cloud, let a service assume a short-lived role (Lesson 10) instead of carrying a static access key around — there’s then no long-lived secret to leak.

Pitfalls to Avoid

1. Committing secrets to git. The most common leak; bots find public ones in minutes. Use a secrets manager or injected env vars, and enable push protection.
2. Deleting a leaked secret instead of rotating it. It’s still in history and likely already captured. Rotate and revoke.
3. A .env file in the repo. That’s committing secrets with extra steps. Keep it out of version control; inject at runtime.
4. Logging secrets. Config dumps, header logs, and verbose errors leak tokens. Redact them.
5. Embedding secrets in client code. Mobile/JS bundles are public. Route the call through your backend instead.
6. Never rotating. Long-lived static secrets are high-value targets. Rotate on a schedule and on any exposure; prefer short-lived credentials.
7. Hand-rolling key management. Use a KMS with envelope encryption so the master key never touches your servers.

Quick Recap

• Secrets leak in boring ways — git history, logs, Slack, client code — so manage them as a lifecycle: generate → store (vault) → inject at runtime → rotate → audit.
• Never commit secrets; enable secret scanning / push protection. A secret that touched git history is compromised — rotate it, don’t just delete the line.
• Use envelope encryption / KMS for data at rest: a per-item DEK wrapped by a master KEK that never leaves the KMS — so a server breach doesn’t expose the master key, rotation is cheap, and every decrypt is audited.
• Apply least privilege to secrets, never log them, separate them per environment, and prefer short-lived workload identity over static keys.

Next lesson: pulling the application-security threads together at the API layer — the OWASP API Top 10, mass assignment, and rate limiting as a token bucket.