Blue green and canary deployments both buy the same thing, a cheap way to be wrong. The switching is easy. Noticing, and the database, are not.
Highlights
- Blue green and canary deployments are two answers to one question, which is how you replace a running version without users paying for your mistake.
- Blue green keeps two complete environments and switches all traffic at once, so rollback is nearly instant and you pay for double the infrastructure.
- Canary sends a small slice of real traffic to the new version first, so the blast radius is small and you need genuinely good metrics for it to mean anything.
- DORA research puts elite performing teams at multiple deployments per day with change failure rates under five percent, which is only sustainable when a bad release is cheap to reverse.
- Neither strategy helps if you cannot tell that the new version is worse, which makes detection the part most teams underinvest in.
- A canary percentage has to be large enough to produce a usable signal, and one percent of low traffic tells you almost nothing.
- Database schema changes are the real constraint, because code reverts in seconds and a dropped column does not.
Deploying used to mean a maintenance window, a rehearsed sequence, and somebody watching a dashboard at two in the morning. DORA research now puts elite performing organisations at multiple deployments per day with change failure rates under five percent, and that combination is only possible when a bad release is cheap to reverse rather than carefully avoided. The strategy you use to deploy is really a statement about how much a mistake costs you.
Blue green and canary are the two answers you will hear most, and both are simple enough to explain in a paragraph. What makes them interesting is not the mechanism, which is the easy half. It is that both depend entirely on your ability to notice something is wrong, and that both are constrained by the one thing you cannot roll back. This guide covers the mechanisms plainly, then spends real time on those two problems.
What a deployment strategy is
A deployment strategy is the agreed sequence by which you replace a running version of your software with a new one, covering how traffic moves from old to new, how you decide whether the new version is healthy, and how you get back to the old one when it is not.
That third clause is the one people skip, and it is the one that determines whether the other two matter.
Start with what you probably already have
Two simpler strategies come before either headline option, and knowing them makes the comparison land.
Recreate stops the old version, then starts the new one. There is downtime, it is trivial to reason about, and for an internal tool nobody uses at night it is perfectly adequate. Do not let anyone tell you this is always wrong.
Rolling update replaces instances a few at a time, so some serve the old version while others serve the new one, and the service stays up throughout. This is the Kubernetes default and it is a sensible starting point for most services. Its weakness is that rollback means another rolling update, which takes as long as the deployment did, and that both versions run simultaneously whether or not your code is ready for that.
Did you know
A rolling update already puts two versions of your code in production at the same time, which surprises people who think of that as a canary problem. If your new version writes a field the old version does not understand, a rolling update will produce inconsistent behaviour for the duration of the rollout, and nobody planned for it because nobody thought of a rolling update as a strategy with implications. The backward compatibility discipline that canary makes obvious was always required. Rolling updates just hid the requirement.
Blue green, explained simply
You run two complete production environments. One serves live traffic and the other sits idle. To deploy, you install the new version on the idle environment, test it there properly, then move all traffic across at once. The environment you just left stays running and untouched.
The appeal is the reversal. If something is wrong, you switch traffic back, and that takes as long as one routing change rather than as long as a deployment. The old version is still running and still warm.
Its costs are real and worth stating plainly. You pay for two environments, which for a large system is a serious number. The switch is all or nothing, so every user meets the new version at the same moment and any problem affects everyone immediately. And you must keep the old environment running long enough to be useful, which means resisting the urge to reclaim it the moment the switch looks fine.
Watch out
If you switch traffic by changing a DNS record, your rollback is not instant no matter what the diagram says. Resolvers cache answers for the length of the record's TTL, and you cannot flush somebody else's cache, so some users keep reaching the old environment for minutes after you switch and keep reaching the new one for minutes after you switch back. Lower the TTL a day in advance if you must use DNS, and prefer a load balancer or service level switch where the change takes effect immediately.
Canary, explained simply
Instead of moving everyone, you move a few. A small percentage of real traffic goes to the new version while the rest continues to the old one. You watch error rates, latency, and whatever business metric matters, and if things look healthy you increase the share in steps until the new version carries everything.
The appeal is blast radius. A problem that would have hit every user hits five percent of them, and you find out before it is everyone's problem.
Its costs differ in kind rather than in degree. You need a traffic layer that can split by weight, which on Kubernetes means ingress annotations, a service mesh, or a progressive delivery controller. You need metrics good enough to distinguish a real regression from normal variance at a small sample size. And both versions run against the same database simultaneously, for as long as the rollout takes, which makes backward compatibility mandatory rather than advisable.
Typical step patterns run something like five percent, then twenty five, then fifty, then everything, with more cautious changes starting at one percent and moving in smaller increments.
Watch out
A canary percentage has to be big enough to produce a signal, and this is the detail that makes canary deployments quietly useless for low traffic services. If you serve one hundred requests a minute, a one percent canary receives one request a minute, and you will not reliably detect a five percent increase in error rate from that within any sensible window. Work out how many requests your smallest step actually receives, and how long you would need to watch it to see the size of problem you care about. If the answer is hours, either raise the percentage or accept that your canary is decorative.
Blue green and canary deployments side by side
The row that decides most real choices is the last but one. Canary requires your code and your data to tolerate two versions running together for an extended period. Blue green mostly does not, which is why teams with awkward schema constraints often reach for it despite the cost.
Learn the deployment mechanics on a real cluster
Both strategies are far easier to reason about once you have watched a rolling update replace pods one at a time and seen what happens when a readiness probe fails. The Certified Kubernetes Administrator (CKA) course on KodeKloud covers Deployments, rollouts, probes, and Services through hands on labs, which is the ground everything here sits on.
The part everyone underinvests in
Here is the uncomfortable thing about both strategies. Each one gives you a cheap way to reverse a bad release. Neither one tells you that the release is bad.
A canary at five percent is worthless if nobody is watching the right numbers, and a blue green switch is worthless if the problem only becomes visible an hour later. The mechanism is the easy half of the problem, and detection is where the actual engineering lives.
Four things are worth watching during and after any release.
Error rate, compared against the version still serving the rest of the traffic rather than against an absolute threshold. A relative comparison is far more sensitive than a fixed number.
Latency percentiles, not the average. A change that leaves the median untouched and doubles the ninety ninth percentile is invisible in a mean and extremely visible to users.
Saturation, meaning whether the new version is using noticeably more memory or connections than the old one for the same work.
One business metric, such as checkouts started or logins completed. Technical metrics can all look fine while the feature silently does nothing useful.
In the wild
A team ran a canary at ten percent for twenty minutes, saw clean error rates and latency, and promoted to full traffic. The regression appeared forty minutes later, because the bug was in a scheduled job that ran hourly rather than in the request path. The canary was working exactly as designed and the observation window was shorter than the interval at which the broken code executed. It is worth asking, before promoting anything, whether your watch period is long enough to include everything the new version actually does.
Automate the rollback, not just the deployment
This is the single highest value change most teams can make, and it is a change to the reverse direction rather than the forward one.
If rolling back requires a person to notice, decide, and run something, it is slow. When it is slow, people hesitate to deploy. When people hesitate, changes accumulate into larger batches, and larger batches are riskier, which justifies more hesitation. The cycle is self reinforcing and the way out is making reversal automatic and boring.
A progressive delivery controller does this for canary by comparing metrics against a baseline at each step and aborting the rollout when a query fails. Argo Rollouts is the most widely adopted, with Flagger a common alternative. For blue green, the equivalent is a script that flips the routing back, tested often enough that nobody is nervous about running it.
The hard part, which is the database
Everything above concerns code, and code is the easy thing to reverse. A schema change is not.
You can switch traffic back to the old version in seconds. You cannot un drop a column in seconds, and if the old code expects a column your migration removed, your instant rollback fails at exactly the moment you need it.
The pattern that resolves this is expand and contract, and it splits one migration into three deployments.
Expand. Add the new structure without removing anything. Add the column, keep the old one, and make sure the currently running code is entirely unaffected by the change. Deploy this first, on its own.
Migrate. Deploy code that writes to both the old and new structure and reads from the new one. At this point either version of the application can run safely, because the old shape still exists and is still maintained.
Contract. Only after the new code has been running long enough that you are confident, and only in a later deployment, remove the old structure.
ALTER TABLE orders ADD COLUMN customer_email text;
UPDATE orders SET customer_email = email WHERE customer_email IS NULL LIMIT 1000;
ALTER TABLE orders DROP COLUMN email;Those three statements belong to three different deployments rather than one migration file. The first is purely additive, so the code currently running is unaffected and can keep operating exactly as it did. The second backfills the new column in bounded batches, and it runs outside the deployment so it can take as long as it needs. The third removes the old column, and it belongs days later once the new code has proven itself, because running it any earlier is what breaks your ability to reverse.
What matters is that at every point in that sequence, the version of the application currently running and the version immediately before it can both operate against the schema as it stands. That property has a name, N minus one compatibility, and violating it at any single step means your rollback will not work when you try it.
Watch out
Backfilling a large table in one statement will lock it, and a lock on a busy table during a deployment turns a routine migration into an outage. Batch the update, run it outside the deployment, and let it take as long as it takes. The temptation to do the whole thing in one migration script is strongest exactly when the table is large enough that doing so is dangerous.
How to choose
Work down these questions and stop at the first clear answer.
Is this an internal tool nobody uses at night? Recreate is fine. Spend your effort elsewhere.
Do you have no traffic splitting and no metrics yet? Rolling update, which is the Kubernetes default, plus proper readiness probes. Build the foundations before the strategy.
Do you need to be able to reverse in seconds, and can you afford duplicate infrastructure? Blue green.
Do you have enough traffic for a small percentage to be meaningful, and metrics you actually trust? Canary, and grow into automated analysis.
Two things are worth saying about that table. Most teams reading this want a rolling update with good probes rather than either headline strategy, because the foundations pay off more than the sophistication. And the two are not exclusive, since a common production pattern is a blue green switch preceded by a short period of partial traffic, which gets you instant reversal and an early warning.
Automate the delivery side properly
Once the strategy is chosen, the value comes from running it the same way every time rather than by hand. The Argo CD course on KodeKloud covers GitOps delivery, syncing, and the declarative approach that progressive delivery builds on, and the KodeKloud playgrounds give you clusters where you can run a rollout, break it deliberately, and watch it revert.
Common mistakes
Where to start
- Find out what your current strategy actually is, since many teams are running the platform default without having chosen it.
- Time a rollback in a non production environment, because the number will be worse than you expect and it is the number that matters.
- Add readiness probes that reflect real dependencies rather than confirming the process started.
- Pick one business metric to watch during releases, and put it on the same dashboard as your error rate.
- If you run canary, calculate the request volume your smallest step receives and decide whether that is a signal.
- Adopt expand and contract on the next schema change, splitting it across three deployments even though it feels like overhead.
- Automate the reversal path and run it deliberately once, so nobody is learning the procedure during an incident.
Conclusion
Blue green and canary are both easy to describe and both solve the same problem, which is making a bad release cheap. Blue green buys instant reversal with duplicate infrastructure and an all at once switch. Canary buys a small blast radius with traffic splitting complexity and a hard requirement for metrics you trust.
What neither buys you is knowing that something is wrong, and that is where the real work sits. A reversal mechanism you never trigger because nobody noticed is not a safety net. Watch relative error rates rather than thresholds, watch percentiles rather than averages, include one metric a product person would recognise, and make sure your observation window is longer than the interval at which your new code actually runs.
Then deal with the database, because that is the constraint that decides what any of this can promise. Expand, migrate, and contract across three deployments keeps every version compatible with the one before it, and without that discipline your instant rollback is a diagram rather than a capability.
Ready to Run This Properly on Kubernetes?
Deployment strategy is where cluster mechanics, observability, and delivery automation meet, which makes it a good reason to shore up all three. The Certified Kubernetes Administrator (CKA) course on KodeKloud covers Deployments, rollouts, and probes in depth, the Argo CD course covers the GitOps delivery layer that progressive rollouts build on, and the KodeKloud playgrounds give you clusters where a failed rollout costs nothing. Start with one today.
FAQs
Q1: What is the difference between blue green and canary deployments?
They differ in who meets the new version first. Blue green keeps two complete production environments, one live and one idle, and you deploy to the idle one, test it there, then move all traffic across at once, which means every user experiences the new version at the same moment and rollback is a single routing change back. Canary keeps one environment and sends a small percentage of real traffic to the new version alongside the old one, increasing the share in steps while you watch metrics, which means a problem affects only that slice rather than everyone. The trade is different in kind rather than in degree. Blue green costs you duplicate infrastructure and gives you near instant reversal with an all or nothing switch. Canary costs you a traffic layer that can split by weight and metrics good enough to detect a regression at small sample sizes, and it gives you a much smaller blast radius. The practical decider is often the database, since canary requires both versions to run against the same schema for the whole rollout.
Q2: Which should I use, and do I need either one yet?
Most teams reading this want a rolling update with proper readiness probes rather than either headline strategy, because the foundations matter more than the sophistication. Work down four questions. If this is an internal tool nobody uses overnight, recreate is genuinely fine and you should spend the effort elsewhere. If you have no traffic splitting and no reliable metrics, use a rolling update, which is the Kubernetes default, and invest in probes and observability first. If you need to reverse in seconds and can afford duplicate infrastructure, blue green earns its cost. And if you have enough traffic for a small percentage to be statistically meaningful along with metrics you trust, canary is available and is where high performing teams tend to end up. The two are not exclusive either, since a common production pattern is a blue green switch preceded by a brief period of partial traffic, which gives you instant reversal plus an early warning.
Q3: What do I need in place before trying a canary deployment?
Three things, and missing any one makes the exercise decorative. You need a traffic layer that can route by weight, which on Kubernetes usually means canary annotations on your ingress controller, a mesh such as Linkerd or Istio, or a controller built for the job like Argo Rollouts or Flagger. You need metrics you trust, specifically error rate compared against the version still serving the rest of the traffic, latency at the ninety fifth and ninety ninth percentiles rather than the average, and at least one business metric a product person would recognise. And you need enough traffic that your smallest step produces a usable signal, which is the requirement people skip. If you serve one hundred requests a minute, a one percent canary receives one request a minute and will not reliably reveal a five percent error increase in any sensible window. Calculate the volume your first step actually receives before deciding the percentage, and raise it if the answer is too small to observe.
Q4: Why do people say the database is the hard part?
Because code reverts in seconds and a schema change does not. You can switch traffic back to the previous version almost instantly, but if a migration dropped a column that the previous version still expects, that instant rollback fails at exactly the moment you need it. The pattern that resolves this is expand and contract, split across three separate deployments. First expand, adding the new structure while removing nothing, so the currently running code is entirely unaffected. Then migrate, deploying code that writes to both the old and new shape and reads from the new one, at which point either version of the application can run safely. Then contract, removing the old structure only in a later deployment once you are confident. The property this preserves is that the running version and the one immediately before it can both operate against the schema as it stands, which is known as N minus one compatibility. Violating it at any step means your rollback will not work. One practical note: backfill large tables in batches outside the deployment, since a single locking statement on a busy table turns a routine migration into an outage.
Q5: How long should I watch a canary before promoting it?
Long enough to include everything the new version actually does, which is usually longer than teams assume and is not a fixed number. The common failure is watching a request path for twenty minutes, seeing clean metrics, promoting to full traffic, and meeting a regression forty minutes later because the bug lived in an hourly scheduled job rather than in the request handler. So the useful question is what the new code does on a timer, on a queue, or in a batch, and whether your observation window is long enough to have exercised it at least once. Beyond that, the window needs to be long enough for your traffic volume to produce a statistically meaningful sample at the current percentage, which is a calculation rather than a feeling. As a starting shape, watch each step until you have seen enough requests to detect the size of regression you care about, and extend the final step before full promotion to cover at least one cycle of any periodic work.
Q6: What is progressive delivery, and how does it relate to these?
Progressive delivery is the broader idea that a release should reach users gradually and with automated judgment rather than in one supervised step, and canary is its most common form. In practice it means a controller shifts traffic in defined increments, runs metric queries against a baseline at each increment, and aborts the rollout automatically when a query fails, so no human has to notice and decide. Argo Rollouts is the most widely adopted implementation, with Flagger a common alternative, and both sit on top of whatever traffic layer you already run. It usually pairs with feature flags, which separate deploying code from exposing behaviour, so a feature can ship to production switched off and then be enabled for a small group independently of any deployment. The reason this matters is the single highest value change most teams can make, which is automating the reversal rather than only the deployment. When rollback is manual it is slow, when it is slow people deploy less often, and when people deploy less often the batches get larger and riskier. Automating the reverse direction breaks that cycle.
Discussion