# Canary Deployments

*2026-10-11*

> Safely releasing cross-cutting changes in legacy cloud environments


<small><i>(Originally drafted in September 2023 during my tenure at Xero. Republished here in October 2026 with updated context and removed proprietary references)</i></small>

> "Don't save the canary. Fix the coal mine." — [Seth Godin](https://www.azquotes.com/quote/1447846)

Modernisation isn't just about using the latest technologies; it's also about using modern engineering processes to deliver value safely to customers.

In the early days of web software engineering, deploying software generally entailed taking application binaries, zipping them up, copying the zip file to a physical server, unzipping the files, and updating the server to use the new binaries.

This byzantine approach to deployments had the potential to be rife with human error: getting a simple configuration setting value incorrect could bring down the entire application.

Fortunately, the world has moved on substantially.

While feature flags are great for managing change at a granular code level, sometimes there are cross-cutting changes that simply cannot be feature flagged.

Servers in the cloud are [no longer thought of as 'pets'](https://www.hava.io/blog/cattle-vs-pets-devops-explained) but instead as 'cattle' – a rather crude way of describing cloud servers as ephemeral rather than static resources.

The flexibility of the cloud environment provides us with a plethora of different approaches when building and deploying software to end users.

However, in most cases you won't actually know how changes will behave until they get deployed and start taking production traffic.

For cross-cutting concerns-such as Identity and Authorisation-it can be incredibly difficult to release such risky changes without customers noticing or a production incident being raised.

## The Authorisation Service Problem

At many organisations, to provide customers with differing product offerings (SKUs), there needs to be a central authority that dictates what products and features a customer is allowed to use as part of their subscription.

At the time, the features that customers would see were often defined by what was known as the 'User Roles Matrix' and this enabled or restricted functionality depending on roles like 'User', 'Advisor', 'Payroll Admin', and so on.

Ironically, to ensure consistency across products, this 'User Roles Matrix' was then copied verbatim across numerous applications in a company's portfolio-including mobile apps.

With the introduction of a centralised Authorisation Service, this changes gatekeeping attributes to use `RequiredActions` rather than `AuthorizedRoles`:

```csharp
[Route("GetInvoice")]
[HttpGet]
[Gatekeeper(RequiredActions = new[] { nameof(Actions.ViewInvoices) })]
public JsonResult GetInvoice(Guid invoiceId, ...)
```

This is a significant, cross-cutting change: if the authorisation actions specified on each endpoint are incorrect, then this has the potential to lock users out of performing certain operations.

## The AuthZ SDK Challenge

As part of building the Authorisation Service, teams often build an AuthZ SDK-a cross-platform library that enables products and teams to easily integrate with the Authorisation Service.

In a .NET application it is nigh on impossible to use both versions of the same library at the same time; the application uses either vOld or vNew of a library.

While it's certainly possible to use binding redirects to have a newer version of a library masquerade as an older version, attempting to use [ILMerge](https://github.com/dotnet/ILMerge) to run both versions concurrently for different code paths is both incredibly confusing and incredibly difficult to accomplish.

This is especially true when considering that those libraries also have their own dependencies such as gRPC libraries.

Given authorisation is a cross-cutting concern, we need a way to safely release this change after all pre-production testing has been completed.

In addition to these changes, the SDK NuGet package itself also has an update, and while we can do as much pre-production testing as possible, nothing compares to real production traffic.

How can we get this change out safely to customers?

## The Canary In The Coal Mine

![Canary Cage](canary-cage.jpg#center)

Carbon monoxide in a closed environment such as a coal mine can be deadly.

Luckily, the canary bird is particularly sensitive to carbon monoxide-a substance which led to numerous mining accidents in the [aftermath of industrialisation](https://review.gale.com/2020/09/08/canaries-in-the-coal-mine/).

To reduce risk, coal miners would use a canary in a cage as an 'early warning system': if the canary showed signs of distress or died then this would alert the miners to the presence of the deadly gas before it could quickly build up in the body.

In software engineering, a [canary release](https://martinfowler.com/bliki/CanaryRelease.html) is a way of testing changes to an application with a subset of users before those changes are released to a wider audience.

Feature flags enable real-time, granular, and low-level changes to code behaviour while the application is running.

![Feature Flags](feature-flags.png#center)

A canary, on the other hand, takes this concept an abstraction higher by effectively 'feature flagging the entire application' at the infrastructure level.

![Canary Release](canary-release.png#center)

If there are any issues with the canary then it can be safely removed, traffic is redirected back to the original live instances, and the 'blast radius' of the canary changes is therefore reduced.

## Deploying to EC2-Based Infrastructure

At many companies, most legacy .NET Framework applications are deployed to Windows-based EC2 instances.

Although this method of deploying and releasing software to customers has always been excruciatingly slow, we can still leverage the overall flexibility of the cloud to enable canary releases.

In most cases all EC2 instances are created behind a load balancer.

![Infrastructure before canary release](infra-before-canary-release.png#center)

With this particular arrangement there are two possible approaches when releasing a new artifact version:

### [CodeDeploy](https://docs.aws.amazon.com/codedeploy/latest/userguide/welcome.html)
* Drain traffic from an EC2 instance behind the load balancer
* Deregister the EC2 instance from the load balancer
* Update the EC2 instance with the new version
* Re-register the EC2 instance to the load balancer
* Rinse and repeat for all other EC2 instances

### [Blue/Green Deployment](https://docs.aws.amazon.com/whitepapers/latest/overview-deployment-options/bluegreen-deployments.html) (Infrastructure Stack Release)
* Create an entirely new load balancer and corresponding stack of EC2 instances
* Deploy the new version to the EC2 instances
* Switch over traffic from the old load balancer to the new load balancer
* Destroy the old EC2 instances

However, both of these strategies are effectively atomic operations and both result in all EC2 instances having an identical configuration with the same version of the application.

## Server Canarying

Canarying on EC2 entails deploying a specific artifact version to a specific EC2 instance.

![Infrastructure after canary release](infra-after-canary-release.png#center)

If the newer version of the application experiences issues then we can simply remove the instance from the load balancer and destroy it.

With a reduced number of EC2 instances taking traffic, AWS Auto Scaling will then kick in to replace the removed instance with a fresh one that has the previous artifact version deployed.

## Implementation: Two Build Pipelines

For EC2 implementation this is typically achieved through two CI/CD builds with the following steps:

### 1. Get Least Used Instance

Retrieve credentials and execute a custom script to find the least used EC2 instance behind the Load Balancer:

* Find all Elastic Load Balancers with the specified server class
* Retrieve the RequestCount metric from CloudWatch for all load balancers to determine the least used ELB
* Retrieve the CPUUtilisation and NetworkIn metrics from CloudWatch for all EC2 instances behind the load balancer
* Calculate the least used EC2 instance by combining these stats to find the lowest
* Return the ID of the EC2 instance with the lowest CPUUtilisation and NetworkIn metrics

### 2. Canary Deploy

Retrieve credentials, validate the provided EC2 Instance ID, deploy the specified artifact version to the instance, and publish a release event:

1. Validate that the server class is for a live AWS EC2 Instance
2. Perform canary deployment:
    * Execute an [AWS Systems Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/what-is-systems-manager.html) document to remove the instance from the load balancer
    * Execute an AWS Systems Manager document to deploy the specified artifact version to the EC2 instance
    * Update the [APM Application Name](https://newrelic.com/platform/application-monitoring)
    * Execute an AWS Systems Manager document to add the instance back to the load balancer
3. Publish a release event via SNS topic

Once the artifact has been deployed it can then be monitored through an APM dashboard specifically for canary monitoring:

![New Relic Dashboard](new-relic-dashboard.png#center)

Querying the error logs via log aggregation is then simply a case of filtering by the EC2 Instance ID that was used to deploy to.

## The Release Process

With the canary release process in place we can now safely release SDK changes to a single EC2 instance, monitor the release using an [APM dashboard](https://www.elastic.co/docs/solutions/observability/apm), and remove the EC2 instance from the load balancer if there are any issues.

The process involves:

1. Building a deployment artifact
2. Using the Get Least Used Instance build to find an EC2 instance to deploy to
3. Using the Canary Deploy build to deploy the artifact
4. Monitoring the release using the live canary dashboard

After an initial attempt at upgrading the SDK is merged, the change may be reverted temporarily so production can continue without blocking releases. Several attempts at upgrading and reverting changes may be made before the SDK is released to customers with minimal issues, after which the changes are eventually merged into master.

While the canary release was initially created for releasing the SDK library, this strategy proves to be incredibly useful later on-especially when it comes to upgrading third-party packages.

## Key Takeaways

Canary deployments provide several benefits:

* **Reduced Blast Radius**: Issues affect only a subset of users
* **Real Production Testing**: Nothing beats actual production traffic
* **Quick Rollback**: Simply remove the instance from the load balancer
* **Monitoring Integration**: Works with existing observability tooling

This approach bridges the gap between feature-flagged deployments and full blue/green infrastructure swaps, providing a pragmatic middle ground for legacy cloud environments that still rely on long-running VM instances.