Managing Environment Variables and Secrets in DevOps

Config and secrets are not the same thing, and treating them as the same thing is how credentials end up in public repositories.

In 2025, GitGuardian detected 28.65 million new hardcoded secrets in public GitHub commits. That is a 34 percent rise on the previous twelve months, and the steepest one year climb across five editions of the company's annual report. The more uncomfortable number is what happened next, because 64 percent of the valid secrets leaked back in 2022 were still not revoked when the 2026 report was published. The leak is not really the incident. The credential that nobody ever turned off is the incident, and it stays an incident for years.

Two more findings from that research reframe how most teams think about this. Internal repositories are around six times more likely to contain a hardcoded secret than public ones, at roughly 32 percent against 6 percent, which means the private repository your team treats as safe is statistically the more dangerous place. And roughly 28 percent of incidents began somewhere other than a repository altogether, inside collaboration tools such as Slack, Jira, and Confluence, where a credential pasted into a thread to unblock a colleague lives on indefinitely.

This guide is about closing those gaps at a practical level. It starts with a distinction that decides everything, walks through the surprisingly many ways an environment variable escapes a running process, then covers what to actually do in local development, Docker, Kubernetes, CI/CD, and Terraform, before finishing with the two things teams skip most: rotation, and the first hour after a leak.

What we mean by environment variables and secrets

Managing environment variables and secrets is the practice of separating an application's configuration from its code, storing the sensitive parts in a system designed to protect them, delivering both to the right process at the right moment, and controlling who can read them, how long they live, and how quickly they can be replaced.

The phrase is usually said as though it describes one activity. It describes two, and the difference between them is the foundation everything else rests on.

Configuration and secrets are not the same thing

Here is the distinction that quietly determines whether the rest of your practice works.

Configuration is anything that changes between environments but causes no harm if somebody reads it. A log level, a feature flag, a region name, a public API endpoint, a connection pool size. If this leaked, you would shrug.

A secret is anything that grants access. A database password, an API key, a signing key, a token, a certificate private key. If this leaked, you would need to act, and the action would be revocation rather than embarrassment.

The reason this matters so much in practice is that most teams pick one delivery mechanism, usually environment variables, and push everything through it. That works technically, and then the secret inherits the security posture of the configuration, which is to say almost none.

The six ways an environment variable leaks

This is the section worth reading twice, because none of these involve committing a file to a repository and most engineers have not thought about several of them.

The first path is the process listing. On Linux, a process's environment is readable at /proc/<pid>/environ. The kernel restricts this to the same user or root, but that is a lower bar than people assume in a container running everything as one user, or on a host where a support engineer has sudo. Anyone who can run commands in that context can read every secret in every process's environment.

The second path is child processes. Environment variables are inherited by default, so every subprocess your application spawns receives the whole environment. A build script that shells out, an application that invokes a helper binary, a debugging tool someone runs inside the container, all of them get the database password whether or not they need it.

The third path is crash reporting and error tracking. Many error handlers capture the environment as context to help with debugging, which is genuinely useful right up until the moment a stack trace containing your production credentials is uploaded to a third party service and sits in a searchable dashboard that dozens of people can access.

The fourth path is container inspection and image layers. Running docker inspect on a container prints its environment variables in full, and anyone with access to the Docker socket can do this. Worse, if a secret was passed as a build argument or set with an ENV instruction in a Dockerfile, it is baked into the image layers and travels with that image to every registry and every machine that pulls it.

The fifth path is CI/CD logs. Build systems echo commands, print environment dumps when debugging is enabled, and store logs for a long time. A single env or printenv added during troubleshooting can write your entire secret set into a log that is retained for a year and readable by everyone with repository access.

The sixth path is developer machines and collaboration tools. A .env file copied to a laptop for local testing, a credential pasted into a Slack thread to unblock a teammate, a screenshot in a Jira ticket. GitGuardian's finding that roughly 28 percent of incidents begin somewhere other than a repository is describing exactly this, and none of it is caught by scanning your code.

The maturity ladder

Most teams sit somewhere on this ladder without having consciously chosen a rung. Finding yours is a useful five minutes.

Two observations about this ladder are worth more than the ladder itself.

Level 1 is where most teams live and it is not shameful, because it is a genuine improvement over level 0 and it works fine for local development. The mistake is staying at level 1 for production, where the blast radius of a copied file is entirely different.

Reaching level 4 sounds exotic and is more achievable than it looks, because the tooling has matured considerably. A dynamic secret is one that does not exist until something asks for it, lives for minutes or hours, and expires automatically. Vault's database secrets engine will create a database user on demand with a short lease, and cloud providers do something similar when a workload assumes a role instead of holding a key. A credential that expires in an hour is a credential that mostly cannot be leaked in a way that matters, which sidesteps the rotation problem rather than solving it.

Local development, done properly

Start here, because habits formed locally propagate everywhere.

Use a .env file for local configuration, keep it out of version control, and commit a template alongside it so that a new joiner knows what they need without being handed anything sensitive.

cat > .env.example <<'TEMPLATE'
DATABASE_URL=postgres://user:password@localhost:5432/appdb
STRIPE_API_KEY=
LOG_LEVEL=debug
FEATURE_NEW_CHECKOUT=false
TEMPLATE

printf '.env\n.env.local\n*.pem\n*.key\n' >> .gitignore
git check-ignore -v .env

The first command writes a template containing the variable names with placeholder or empty values, which documents the shape of the configuration without exposing anything real. The second appends the sensitive filename patterns to .gitignore, including private key extensions that are easy to forget. The third command is the one people skip, and it verifies that .gitignore actually matches the file, printing the rule responsible, because assuming a pattern works and confirming it works are different things.

Three habits go with this. Never put real production credentials in a local .env file, since local machines are backed up, synced, and occasionally lost, and a development database with fake data is almost always sufficient. Install a pre commit secret scanner so that a mistake is caught before it becomes a commit rather than after it becomes history. And when you do need a shared local secret, fetch it from your secrets manager with a short lived token rather than passing files around.

Containers, where secrets get baked in

Docker introduces a failure mode that catches almost everyone once, which is the difference between build time and run time.

Anything you pass with ARG or set with ENV in a Dockerfile becomes part of the image. Image layers are immutable and shareable, so a secret set during a build travels to your registry and to every host that pulls the image, and deleting the file in a later layer does not remove it from the earlier one. The layer is still there.

FROM node:20-alpine
WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=secret,id=npmrc,target=/root/.npmrc npm ci --omit=dev
COPY src ./src
USER node
CMD ["node", "src/server.js"]

That Dockerfile solves the common case of needing a private registry credential during a build. The --mount=type=secret syntax, provided by BuildKit, makes the credential available as a file only for the duration of that single RUN instruction, and it is never written into any layer. You pass the value at build time with a flag pointing at a local file, and the resulting image contains no trace of it.

At run time the picture is different, because the container genuinely needs its configuration. Passing values with the -e flag or an env file works and is the normal approach, and the important habit is knowing that docker inspect will display all of them to anyone with access to the daemon. For anything highly sensitive, mounting a file rather than setting a variable is meaningfully safer, since a mounted file can have restrictive permissions and does not appear in inspection output or get inherited by every child process.

Kubernetes, where base64 is not encryption

Kubernetes has a purpose built object for this and a widespread misunderstanding about what it provides.

A Secret object stores data base64 encoded. Base64 is an encoding, not encryption, and decoding it requires no key and no privilege. Anyone who can read the Secret object can read the value with a single command, and by default Secrets are also stored unencrypted in etcd, meaning anyone with access to the etcd data or a backup of it has everything.

kubectl create secret generic app-db \
  --from-literal=DATABASE_PASSWORD='s3cr3t-value' \
  --namespace prod

kubectl get secret app-db -n prod -o jsonpath='{.data.DATABASE_PASSWORD}' | base64 -d

kubectl auth can-i get secrets --namespace prod --as system:serviceaccount:prod:web

The first command creates a Secret from a literal value, which is fine for a demonstration though in practice you would source it from a file or a manager. The second retrieves the stored value and decodes it, and running this once is worth more than any amount of explanation, because seeing your password print to the terminal in one line makes the point permanently. The third command is the one to build a habit around, since it asks whether a specific ServiceAccount can read Secrets in a namespace, and the answer tells you whether your access controls are what you believe them to be.

Four things turn Kubernetes Secrets from a filing convention into an actual control.

Enable encryption at rest in the API server configuration so that Secret data is encrypted in etcd, ideally with keys held in a cloud key management service rather than a local file.

Restrict RBAC properly, because the default of many roles being able to list Secrets across a namespace defeats everything else. A workload should be able to read the specific Secrets it needs and nothing else.

Prefer mounted volumes over environment variables when delivering secrets to pods. A mounted Secret appears as a file with controlled permissions, it is not visible in docker inspect or a process environment, it is not inherited by child processes, and it updates automatically when the Secret changes, whereas an environment variable is fixed for the life of the pod.

Bring secrets in from a real manager rather than storing them in the cluster as the source of truth. The External Secrets Operator is the common pattern here, syncing values from Vault or a cloud secret manager into Kubernetes Secrets so the manager remains authoritative and the cluster holds a cache.

For teams doing GitOps, where everything must live in Git, two tools solve the encrypted commit problem. Sealed Secrets encrypts a value so that only the controller running in your specific cluster can decrypt it, making the encrypted form safe to commit. SOPS encrypts values inside otherwise readable YAML using a key from a cloud key management service, so the file structure stays reviewable in a pull request while the values stay protected.

Build the secrets management foundation properly

Vault is the tool most teams eventually reach for, and it rewards structured learning because its concepts, including auth methods, secrets engines, leases, and policies, are genuinely different from a simple key value store. The HashiCorp Certified Vault Associate course on KodeKloud covers Vault fundamentals, authentication methods, dynamic secrets engines, access policies, and production deployment, which maps directly onto levels three and four of the maturity ladder above.

CI/CD, the highest value target

Your pipeline holds credentials for everything it can deploy to, which makes it the single most attractive place in your infrastructure. It is also where the most impactful modern improvement is available.

The traditional approach stores long lived cloud access keys as CI variables. Those keys do not expire, they are readable by anything that can run a job, and they are copied into every build environment. The better approach is workload identity federation using OpenID Connect, where your CI provider proves the identity of a specific job to your cloud provider, which returns a short lived credential scoped to that job. No long lived key exists anywhere, so there is nothing to leak and nothing to rotate.

name: deploy
on:
  push:
    branches: [main]

permissions:
  id-token: write
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: production
    steps:
      - uses: actions/checkout@v4
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/deploy-production
          aws-region: eu-west-1
      - run: ./scripts/deploy.sh

Several details in that workflow are doing security work. The id-token: write permission is what allows the job to request an identity token, and without it the federation cannot happen. The contents: read line follows least privilege by granting only what is needed rather than accepting broad defaults. Naming an environment lets you attach protection rules such as required reviewers, so a production deployment can require human approval. And the credentials step exchanges the job's identity token for a temporary role session rather than reading a stored key, meaning there is no secret in this repository at all.

Three further habits matter in pipelines. Scope credentials to environments so that a pull request build cannot touch production. Rely on your platform's masking to redact secret values from logs, while remembering that masking is best effort and a value that has been transformed, encoded, or split across lines can slip through. And avoid passing secrets as command line arguments, since arguments are visible in process listings, whereas environment variables and files are marginally better.

Terraform, and the state file problem

This one surprises people and deserves its own section because the consequence is severe and the cause is not obvious.

Terraform stores the full state of your infrastructure in a state file, and that state includes the values of resources it created. If Terraform creates a database with a password, that password is in the state file in plain text. Marking a variable as sensitive prevents it appearing in console output, and it does not encrypt anything in the state.

The practical consequences follow directly. Your state backend is as sensitive as the most sensitive credential in your infrastructure, so it needs encryption at rest, strict access control, versioning, and audit logging, and it should never sit in a repository or an unrestricted bucket.

A better pattern avoids the problem where possible. Rather than having Terraform generate and store a password, have it create the resource and let a secrets manager own the credential, or use dynamic credentials so no long lived password exists to store. Where Terraform must handle a secret, read it from a secrets manager at plan time using a data source rather than passing it as a variable, so the value is fetched rather than persisted in your variable definitions.

Rotation, where programmes quietly fail

Everything above is about preventing exposure. This section is about limiting how long an exposure matters, and it is the part almost everyone postpones.

The GitGuardian figure that 64 percent of valid secrets leaked in 2022 remained unrevoked in 2026 is not really a story about detection. Those organisations mostly knew. What they lacked was a repeatable path from knowing to revoking, because rotating a credential in a real system means finding every consumer, updating them all, and confirming nothing broke, and nobody wants to do that at 4pm on a Friday.

Three practices make rotation survivable.

Design for two valid credentials at once. If your system can accept both an old and a new credential during a transition window, rotation stops being a synchronised cutover and becomes a rolling change. You add the new one, migrate consumers gradually, confirm the old one is unused through access logs, then revoke it. Most cloud providers support two active access keys per identity specifically to enable this pattern, and many APIs support multiple valid keys for the same reason.

Keep an inventory of what uses what. The reason rotation is frightening is uncertainty about what will break, and that uncertainty is a documentation problem rather than a technical one. A simple record of every secret, its owner, its consumers, and its last rotation date converts a scary operation into a checklist.

Rotate on a schedule before you need to. A team that has rotated a credential deliberately in a quiet week can rotate it under pressure. A team whose first rotation happens during an incident is learning the procedure at the worst possible moment. This is exactly the same argument as practising a restore from backup rather than trusting that backups exist.

Dynamic secrets are the structural answer to all of this, and it is worth naming plainly. A credential that expires in an hour does not need a rotation procedure, because expiry is rotation, automatically and continuously. The work moves from operating a rotation process to setting up dynamic issuance once.

The first hour after a leak

Assume this will happen, because at the volumes in the opening statistics it happens to competent teams regularly. Having a sequence written down beforehand is the difference between a contained incident and a slow one.

Revoke first, investigate second. The instinct is to work out how it happened, and that instinct costs you time during which the credential still works. Disable or rotate the credential immediately, then investigate with the pressure off. A revoked credential cannot be used regardless of who saw it.

Assume it was used. Credentials exposed publicly have been observed being picked up within minutes by automated scanners. Check access logs for the credential from the moment of exposure, looking for unfamiliar addresses, unusual regions, and unexpected API calls.

Find every copy. The same value is probably in more places than the one you found. Check other repositories, CI variables, container images, configuration management, wikis, and chat history, since the collaboration tool paths are exactly the ones scanners miss.

Rewrite history if appropriate, but not first. Removing the value from Git history is worth doing for hygiene, and it does nothing about exposure since the credential has already been seen. Rotation is the control, and history rewriting is cleanup.

Write down what the leak path was. The value of an incident is the change it produces. A secret in a commit means you need pre commit scanning. A secret in a log means you need output masking. A secret in an error report means you need a denylist in the error handler. Each path has a different fix and treating them all as carelessness fixes none of them.

Choosing a tool

There is no single right answer, and the honest comparison is about what each option costs you to operate.

A reasonable progression for most teams is to keep .env files strictly for local development, adopt the secret manager of whichever cloud you already use for production, add the External Secrets Operator if you run Kubernetes, and consider Vault when you genuinely need dynamic secrets or a single system spanning multiple clouds. Skipping to Vault before you have the operational capacity to run it well is a common and expensive detour.

Practise this on a real cluster

The difference between reading that Kubernetes Secrets are base64 encoded and decoding one yourself in a terminal is the difference between knowing and understanding. The Certified Kubernetes Security Specialist (CKS) course on KodeKloud covers secrets handling, RBAC, and cluster hardening in depth, and the KodeKloud playgrounds give you clusters where you can create, decode, mount, and lock down Secrets without touching anything that matters.

Where to start

1. Write down every secret your team uses, where it lives, who can read it, and when it was last rotated, because the inventory will be shorter and more alarming than expected.
2. Separate configuration from secrets in one service, moving the harmless values somewhere convenient and the sensitive ones somewhere controlled.
3. Turn on secret scanning for your repositories including the private ones, since those are statistically the more likely to contain something.
4. Add a pre commit scanner to one repository so mistakes are caught before they become history.
5. Replace one long lived CI credential with OpenID Connect federation, because that single change removes a permanent key from existence.
6. In a cluster, decode a Secret with base64 -d in front of your team, then check whether encryption at rest is enabled and who can list Secrets.
7. Rotate one credential deliberately this quarter while nothing is on fire, and write down the procedure while you do it.

Conclusion

Those opening statistics describe a problem growing faster than the developer population, and the reason is not carelessness. It is that secrets are easy to create, easy to copy, and structurally hard to track, so every new service, integration, and automation adds credentials faster than anyone retires them.

What actually helps is unglamorous and specific. Separate configuration from secrets so each gets appropriate handling. Know the leak paths that have nothing to do with committing a file, because those are the ones your scanning will never catch. Use the right mechanism for each stage, meaning environment files locally, mounted secrets in clusters, federated identity in pipelines, and a real manager as the source of truth. Then build the two habits nearly everyone skips, which are rotating credentials on a schedule and revoking before investigating when something goes wrong.

Start with the inventory this week. It takes an afternoon, it is uncomfortable in a useful way, and everything else on this list becomes obvious once you can see what you actually have.

Ready to Build Real Secrets Management Skills?

Secrets management sits at the intersection of infrastructure, identity, and operational discipline, which makes it a genuinely valuable specialism. The HashiCorp Certified Vault Associate course on KodeKloud covers authentication methods, dynamic secrets engines, and access policies from the ground up, the Certified Kubernetes Security Specialist (CKS) course covers cluster secrets and RBAC in depth, and the KodeKloud playgrounds let you practise every command in this guide safely. Pick one and start today.

FAQs