Secrets management & Vault

01What counts as a secret, and why .env files are not enough

A secret is anything that grants access: database passwords, API keys for payment or email providers, OAuth client secrets, JWT signing keys, TLS private keys, SSH keys, encryption keys and webhook signing secrets. The Twelve-Factor approach of putting configuration in environment variables was a big step forward from hard-coding, and .env files on a server are still a reasonable start. But as a system grows, problems appear: copies of the same .env on laptops, CI and servers; no record of who read what; no easy rotation; secrets visible in process listings, crash dumps or error pages; and the ever-present risk of committing the file. Secrets management solves this by treating secrets as data with an owner, an access policy, an audit log and a lifecycle.

• Never in Git, Docker images, client bundles or logs
• Each secret has an owner, a scope and a rotation plan
• Access is granted by identity and recorded

02How a secret manager works

Think of a bank vault with a guard and a ledger. The secret manager stores secrets encrypted at rest, usually under a key protected by a cloud KMS or hardware security module. Applications authenticate with an identity — a cloud IAM role, a Kubernetes service account, a CI job's OIDC token, or an AppRole — instead of a shared password. Policies map identities to paths they may read: the orders service can read secret/prod/orders/* and nothing else. Every read and write is written to an audit log. The main options: HashiCorp Vault (source-available under the BSL licence since 2023, now part of IBM), OpenBao (an open-source fork of Vault under the Linux Foundation), cloud services such as AWS Secrets Manager, Google Secret Manager and Azure Key Vault, and developer-focused platforms like Doppler, Infisical or 1Password Secrets Automation.

03Vault basics: KV secrets and policies

Vault organises everything by paths. The key/value (KV) secrets engine stores static secrets with versioning, so you can roll back. Policies, written in HCL, grant capabilities such as read, create, update or list on paths. Applications log in through an auth method that matches where they run — kubernetes, aws, jwt/oidc for CI, or approle — and receive a short-lived Vault token limited by policy. Development mode (vault server -dev) is for learning only: it runs in memory, unsealed, with a root token. Production needs storage (integrated Raft storage is the common choice), TLS, auto-unseal with a cloud KMS or HSM, audit devices enabled, and the root token revoked after setup. The same commands work with OpenBao's bao CLI.

1# Store and read a versioned secret (KV v2)
2vault kv put secret/prod/orders db_password='use-a-generated-value'
3vault kv get -field=db_password secret/prod/orders
4 
5# orders-policy.hcl — read-only access to one app's secrets
6path "secret/data/prod/orders/*" {
7  capabilities = ["read"]
8}
9 
10vault policy write orders orders-policy.hcl
11vault audit enable file file_path=/var/log/vault_audit.log

04Dynamic secrets: credentials that expire on their own

The strongest idea in Vault is dynamic secrets. Instead of one long-lived database password shared by every instance, Vault creates a unique database user for each application instance on request, with a lease — say one hour. When the lease expires or is revoked, Vault deletes that user. A leaked credential is useless soon after, every connection is attributable to one instance, and revoking access for a compromised service is a single command. The same pattern exists for cloud credentials (temporary AWS keys), PKI (short-lived TLS certificates) and SSH (signed certificates instead of authorised keys). Cloud providers offer similar ideas natively: IAM roles for workloads and IAM database authentication avoid stored passwords entirely. Applications must handle renewal, which client libraries and agents such as Vault Agent do for you.

1# One-time setup by an admin
2vault secrets enable database
3vault write database/config/app-db plugin_name=postgresql-database-plugin \
4  connection_url="postgresql://{{username}}:{{password}}@db.internal:5432/app" \
5  allowed_roles="orders-ro" username="vault_admin" password="initial-admin-pass"
6vault write -force database/rotate-root/app-db   # Vault now owns the admin password
7 
8vault write database/roles/orders-ro db_name=app-db default_ttl=1h max_ttl=24h \
9  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";"
10 
11# Each app instance asks for its own short-lived user
12vault read database/creds/orders-ro

05Getting secrets into applications

There are three common delivery patterns. Direct API: the app uses an SDK to read secrets at startup and renews leases — flexible but couples code to the vault. Sidecar or agent: Vault Agent (or a cloud equivalent) authenticates, fetches secrets and renders them into a file or environment for the app, renewing them automatically; the app just reads a file. Platform sync: in Kubernetes, the External Secrets Operator or the Secrets Store CSI driver syncs from the secret manager into pods. In CI/CD, avoid storing long-lived cloud keys at all: GitHub Actions, GitLab and others can issue an OIDC token per job, which the cloud or Vault exchanges for short-lived credentials. For Docker Compose on a VPS, a simple, sound pattern is root-owned env files with mode 600 outside the repository, or Docker secrets mounted as files.

1# GitHub Actions: no stored cloud keys — exchange the job's OIDC token
2permissions:
3  id-token: write
4  contents: read
5jobs:
6  deploy:
7    runs-on: ubuntu-latest
8    steps:
9      - uses: aws-actions/configure-aws-credentials@v4
10        with:
11          role-to-assume: arn:aws:iam::123456789012:role/deploy-prod
12          aws-region: eu-central-1
13      - run: aws secretsmanager get-secret-value --secret-id prod/orders --query SecretString --output text > /dev/null

06Rotation, least privilege and leak response

Rotation means replacing a secret with a new one on a schedule and after any suspected exposure. To rotate without downtime, support two valid versions at once: create the new credential, deploy it, confirm nothing uses the old one, then revoke the old. Cloud secret managers can automate rotation for databases; for third-party API keys, rotation is usually a scripted runbook. Least privilege means each service, person and pipeline gets only the secrets it needs, in the environment it needs, ideally read-only. Prevention: run secret scanners such as gitleaks or TruffleHog in pre-commit hooks and CI, and enable your Git host's push protection. When a secret leaks, the order matters: revoke or rotate it first, then remove it from history, because rewriting Git history does not un-leak a key that has already been cloned or scraped.

• Rotate: create new → deploy → verify → revoke old
• Scan: pre-commit + CI + push protection
• Leak: revoke first, clean history second, review access logs third

07Vault architecture: seal, storage and high availability

Vault's security model rests on the barrier: everything that leaves Vault for storage is encrypted with a keyring, protected by a root key. Sealing means Vault holds no keys in memory. With Shamir unseal, the root key is split into shares (for example five, with three needed), held by different people; with auto-unseal, a cloud KMS or HSM decrypts it automatically at startup, which is the norm in production because restarts should not need humans at night. Integrated storage (Raft) replicates data across typically three or five nodes; one is the leader, others are standbys that forward requests, and quorum is required for writes. Back up with Raft snapshots and test restores. Enterprise and HCP editions add performance and disaster-recovery replication across regions. Operationally, Vault is tier-zero infrastructure: if it is down, apps cannot start, so plan for caching, sensible lease lengths and monitoring of seal status, leader changes and lease counts.

1# Production-style server config (excerpt)
2storage "raft" {
3  path    = "/opt/vault/data"
4  node_id = "vault-1"
5}
6listener "tcp" {
7  address       = "0.0.0.0:8200"
8  tls_cert_file = "/etc/vault/tls/vault.crt"
9  tls_key_file  = "/etc/vault/tls/vault.key"
10}
11seal "awskms" {
12  region     = "eu-central-1"
13  kms_key_id = "alias/vault-unseal"
14}
15api_addr     = "https://vault-1.internal:8200"
16cluster_addr = "https://vault-1.internal:8201"

08Secret zero: how the first credential is trusted

Every secrets system faces the 'secret zero' problem: to fetch secrets, an app must first authenticate, so what secret does it use for that? The modern answer is to rely on an identity the platform already proves. On AWS, the instance or task role signs a request Vault or Secrets Manager can verify. In Kubernetes, the pod's projected service-account token is validated against the cluster. In CI, the job's OIDC token is signed by the CI provider. On bare VPS servers there is no such platform identity, so options include AppRole with a role id baked into configuration and a secret id delivered once by the deployment tool with a short TTL and single use, or a TLS client certificate issued at provisioning. Whatever you choose, bind the identity to narrow policies and short tokens, and make the bootstrap credential worthless after first use.

09Encryption as a service and envelope encryption

Beyond storing secrets, Vault's transit engine and cloud KMS services encrypt data for applications without ever revealing the key. The app sends plaintext, receives ciphertext, and stores it in its own database; decryption requires a policy-checked call. Keys can rotate, and old ciphertexts stay decryptable and can be re-wrapped to the new version. For large data, use envelope encryption: generate a random data key, encrypt the data locally with it (for example AES-256-GCM), then encrypt the data key with the KMS key and store it next to the data. You get fast local encryption, a central audit trail, and the ability to cut off access by disabling one master key. This pattern underpins field-level encryption of personal data, encrypted backups and secrets in GitOps tools such as SOPS.

1# Transit: the app never sees the key
2vault secrets enable transit
3vault write -f transit/keys/customer-pii
4 
5vault write transit/encrypt/customer-pii plaintext=$(printf '%s' 'AM12345678' | base64)
6# -> ciphertext: vault:v1:8SDd3WHDOjf7mq69CyCqYjBXAiQQAVZRkFM13ok481zoCmHnSeDX9vyf7w==
7 
8vault write transit/decrypt/customer-pii ciphertext='vault:v1:8SDd3WHDOjf7...'
9vault write -f transit/keys/customer-pii/rotate   # new version, old data still readable

10Kubernetes secrets: what base64 is not

A Kubernetes Secret is base64-encoded, not encrypted; anyone who can read Secrets in a namespace, or read etcd backups, sees the values. Harden it in layers. Enable encryption at rest for etcd, ideally with a KMS provider. Restrict RBAC so only the workloads and few operators who need Secrets can get or list them — 'list' on Secrets effectively reveals them all. Avoid committing Secret manifests to Git; use External Secrets Operator to sync from a central manager, or Sealed Secrets or SOPS for GitOps where values are encrypted in the repository. Prefer mounting secrets as files over environment variables, so values can rotate without restarting and are less likely to appear in crash dumps. Finally, use workload identity so pods can call cloud APIs and secret managers without any static keys in Secrets at all.