Secrets management is the category of security problems that engineers cause themselves, often knowingly, because the correct approach requires more setup than DATABASE_URL=postgres://prod:password@....

The cost of getting this wrong is asymmetric. Setting up proper secrets management takes a few hours. A credential leak can result in a compromised production database, unauthorized API usage, or breach notification requirements. The math favors spending the time.

How Credentials Leak

Committed to version control: the most common path. A database password, API key, or service credential gets put in source code or a config file that ends up in a repository. Even if the commit is later reverted, Git history retains it. Even if the repository is private, access to the repository grants access to the secret. GitHub scans for common secret patterns and notifies you, but the leak has already happened.

Logged in application logs: code that logs request parameters, environment variables, or error details often accidentally logs secrets. Log aggregation systems (Datadog, Splunk, CloudWatch Logs) that receive these logs now contain the secret in searchable form.

Exposed in CI/CD: CI systems that print environment variables during debugging, steps that echo configuration to console, or failing builds that dump their environment in the error output.

Left in deployment artifacts: Docker images that COPY configuration files including secrets, build artifacts that embed connection strings, or container images pushed to public registries.

Stolen from running systems: if an attacker gets code execution on your server (through application vulnerabilities, SSRF, container escape), environment variables are trivially readable.

Why Environment Variables Are Not Enough

Environment variables are better than hardcoded strings in source code. But they’re not a secrets management solution.

Environment variables are typically:

• Visible to all processes on the system
• Logged by many tools and frameworks when debugging is enabled
• Passed to child processes by default
• Stored in deployment configuration (docker-compose.yml, Kubernetes manifests, CI/CD configs) which are often version-controlled
• Long-lived - they don’t rotate automatically

The question is: where do those environment variables come from? If the answer is “a file in the repository” or “manually configured in each environment,” you haven’t solved the problem - you’ve moved it.

What Proper Secrets Management Looks Like

A secrets store as the source of truth: HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, Azure Key Vault. These are purpose-built systems for storing, versioning, auditing access to, and rotating secrets.

Short-lived credentials where possible: instead of a database password that never changes, your application gets a short-lived credential (token valid for 1 hour) from the secrets manager. Even if leaked, it expires quickly. AWS IAM roles, Vault dynamic secrets for databases, and Kubernetes ServiceAccount tokens all implement this pattern.

Access control: not every service should have access to every secret. Production credentials shouldn’t be accessible from development environments. The payment service doesn’t need the analytics database password.

Audit logging: every secret access is logged. You can answer “who read this credential and when.”

Rotation: secrets change on a schedule, or immediately when a leak is suspected. Systems that require manual rotation almost never get rotated.

Practical Starting Points

For small teams or simple systems: AWS Secrets Manager or similar cloud-native solution. Store secrets there, fetch them at application startup using the SDK, keep them in memory (don’t write them to disk).

import boto3

def get_secret(secret_name: str) -> dict:
    client = boto3.client('secretsmanager', region_name='us-east-1')
    response = client.get_secret_value(SecretId=secret_name)
    return json.loads(response['SecretString'])

# At startup - not at module level in case of import-time issues
db_config = get_secret('prod/myapp/database')

For Kubernetes: external-secrets-operator or Vault Agent Injector. These sync secrets from your secrets manager into Kubernetes Secrets (or inject them as environment variables) without putting secrets in your manifests.

For local development: a .env file that’s in .gitignore, with a .env.example that contains placeholder values. The .env file never gets committed. Every developer creates their own with development credentials.

# .gitignore
.env
.env.local
*.env

# .env.example (committed)
DATABASE_URL=postgres://localhost:5432/myapp_dev
STRIPE_SECRET_KEY=sk_test_your_key_here

Dealing With Existing Leaks

If a secret has been committed to a repository - even briefly, even in a branch - assume it’s compromised. The correct response:

1. Rotate the secret immediately. Don’t investigate first. The window between discovery and rotation is the risk window.
2. Audit access logs for unexpected usage of the leaked credential.
3. Remove the secret from version history (git filter-repo or BFG Repo Cleaner) to prevent future accidental use, but treat this as cleanup, not security mitigation. The secret was already accessible.

Tools like truffleHog, gitleaks, and detect-secrets can scan repositories for secrets patterns. Running these in CI catches leaks before they’re pushed to shared branches.

The Pattern to Internalize

Secrets should flow in one direction: from a secure store into running processes, in memory, never persisted to disk or version control. Every place you’re currently storing a secret by other means (config files, environment variable definitions in manifests, hardcoded values) is a place to replace with a reference to a secrets manager.

The operational burden of this is real but manageable. The alternative - managing the aftermath of a credential leak - is worse.