Lighthouse Night-shift by Casey Horner

Photo by Casey Horner on Unsplash

A wise mentor once put it well: "when your database goes down, your platform goes with it". In other words, your service's availability is directly tied to your database's availability.

There are businesses that can tolerate planned database downtime without significant impact. If that describes your situation, dear reader, feel free to move along. You can skip this one.

Still here? Good. Then it's safe to assume that you, like most of us, need to minimise downtime and connection disruption when making changes to your database clusters. And if that's the case, you should know that when it comes to AWS RDS Aurora MySQL, blue-green deployments may not be the best tool for the job.

This brings us to AWS RDS Aurora MySQL. To help with upgrades, AWS offers a Blue-Green Deployment feature that, on paper, should let you upgrade your database with minimal downtime.

What Exactly is a "Blue-Green Deployment"?

At its core, blue-green deployment is a zero-downtime release strategy. You maintain two identical production environments: one active (blue) serving all live traffic, and one idle (green) standing by. To deploy changes, you first update the green environment. After verification, you switch traffic from blue to green in a single atomic operation. The result: zero downtime for users, and the ability to instantly roll back by switching traffic back to blue if anything goes wrong.

The Wikipedia page on Blue-Green Deployment covers the history and variations, but the fundamental mechanism is simple: eliminate service interruption by swapping entire environments rather than deploying changes in-place.

For database clusters, this same pattern promises something every operations team wants: the ability to perform upgrades, schema changes, or configuration updates without ever taking the database offline.

An Opinionated Overview of AWS RDS Aurora Blue-Green Deployments

The Blue-Green implementation documented by AWS is fairly straightforward. It starts with you, dear reader, registering a "resource" (aptly named "Blue-Green Deployment"). This prompts AWS to seamlessly clone your existing database into a Green instance, upgrade it, and configure replication using the Blue database as the source. At this point you will have two fully fledged clusters running side-by-side, ready for the switchover.

However, reality is more complex.

The RDS Proxy Gambit

AWS provides two distinct approaches to the Blue-Green Deployment strategy. One approach swaps Blue and Green databases by renaming Blue out of the way while immediately renaming Green to the original database name. The other, subtler approach relies on the presence of an RDS Proxy fronting the database, taking advantage of it to route database connections to the newly promoted Green (I should say "new blue") while the much slower renaming happens in the background.

The difference between these strategies can be significant:

Without an RDS Proxy, you are at the mercy of the AWS APIs. Your resources can take arbitrarily long times to be renamed. I've seen this take anywhere from 15 to 300 seconds, depending on the AWS API load. Connections get interrupted, and services notice the database is down.

With RDS Proxy, the switchover is nearly instant. Resource renaming is handled internally by AWS. Connections made through the RDS Proxy won't be interrupted. As you're using RDS Proxy, the vast majority of your connections should be going through the proxy anyway (monitoring being a notable exception here). The RDS Proxy pauses connections for a few milliseconds while it re-establishes backend connections and reroutes active sessions. Some database clients get confused by sudden connection changes, especially if they cache connection-bound environment variables. In my experience, this never caused widespread failures or connection stampedes.

AWS does not allow you to choose these strategies. Unless you've carefully read the AWS docs, you probably won't notice there are two approaches. And you won't understand the trade-offs until it's too late. AWS makes the choice for you, depending on whether an RDS Proxy fronts your cluster at the time you create the blue-green deployment:

  • If your RDS Aurora cluster has an RDS Proxy when you create the Blue-Green Deployment, AWS blocks Target Group changes. This prevents RDS Proxy modifications while the deployment is active and uses the proxy for the switchover.
  • If your RDS Aurora cluster lacks an RDS Proxy when you create the Blue-Green Deployment, AWS prevents you from creating one while it exists and adopts the slower, more disruptive resource-renaming strategy for the switchover.

The One-Way Street Switchover

Once the switchover is done, the Blue-Green Deployment has served its purpose: you cannot roll back through it, and the only available action is to delete it. It also continues blocking Target Group changes, which prevents RDS Proxy modifications and creates a barrier to quick rollback if needed.

Pick One by Ian Taylor

Photo by Ian Taylor on Unsplash

Furthermore, the Old Blue database is abandoned by the Blue-Green Deployment. After being renamed out of the way, it stops receiving traffic and falls increasingly behind as your application pushes changes through the new Blue (formerly Green) database

In my view, this significantly reduces the switchover's value. For decades, the MySQL community has exploited the fact that, while not officially supported, replication from a newer source to an older replica database is possible and usually works well, provided you don't attempt any Data Definition Language (DDL) statements. Because of this quirk, you can keep the old Blue database around and up-to-date for the first few hours after the switchover, ensuring there is a viable, cheap, fast rollback path in the unlikely event that a severe issue is spotted.

Having a quick, cheap rollback path back to the previous database has saved me several times. I vividly remember rolling back a database upgrade during my tenure at PagerDuty, hours after what seemed like a successful deployment. We discovered the database was slowing down when establishing new TLS connections due to a bug in that release, something we missed during weeks of staging tests.

More Control, Less Disruption: A Home-Brewed Alternative

I think we can do better than AWS's offering with minimal extra effort. The last time I did this was to upgrade a deployed park of RDS Aurora 2 (MySQL 5.7, then on Extended Support) to Aurora 3.04 LTS (MySQL 8.0 compatible). I suggested that by implementing a switchover script ourselves instead of relying on the AWS implementation, we could:

  • retain control over cloning and upgrading the Green databases, avoiding resource renaming;
  • leverage the RDS Proxies fronting our databases for a seamless switchover experience;
  • avoid uncontrollable AWS API stalls and delays;
  • keep the Old Blue (5.7) database as an up-to-date warm standby, ready to resume the workload at a moment's notice (provided we met certain compatibility criteria).

Let me explain: retaining control over the process of creating database clones helps. Minimising database client connection disruption matters. Avoiding unnecessary stalls while executing critical operations on our production environment prevents problems. And, finally, having a clean, simple, cheap, and fast rollback path back gives you and your team safety and confidence.

Don't Try This at Home

Replicating from newer to older versions carries risk. But with careful preparation, it can succeed. The last time I tried this might have been my final chance: I upgraded about 270 databases in RDS Aurora 2 (MySQL 5.7, on extended support) to Aurora 3.04 LTS (MySQL 8.0 compatible). I'm unsure this would work for upgrading from MySQL 8.0 to 9.0.

Our 5.7 databases happened to meet all the compatibility requirements:

  • Inherited Collations: With a schema originating in 5.7, all tables and columns already used legacy collations like utf8mb4_general_ci or latin1. As long as no new tables or column alterations occurred during the window, the 8.0 primary never generated binlog events with the unrecognised (for the 5.7 replica) utf8mb4_0900_ai_ci collation ID.
  • Migrated Authentication: The replication account, brought over from 5.7, retained the mysql_native_password authentication method. We explicitly avoided creating new users on 8.0, as those would default to caching_sha2_password, which 5.7 replicas cannot handle without extra plugin configuration.
  • Favourable 8.0 Defaults: MySQL 8.0 has binlog_transaction_compression OFF by default, and binlog_row_metadata set to MINIMAL, so it doesn’t pack row events with compressed payloads or metadata headers that would choke 5.7’s SQL thread.
  • Clean DML Only: Our services only used standard, backward-compatible DML and no DDL. This ensured the event stream remained digestible for the older engine.

Cost & Effort Estimate

Of course, nothing in this life is really free. It is time to address my old frenemy, the implementation costs.

Let's start with common components to all approaches:

  • AWS RDS Blue-Green Deployment can be created free of charge.
  • RDS Proxy is required if you want to have a smooth switchover. It is also considered a best-practice when deploying RDS Aurora databases, so you should be using it already.
  • Blue-Green Deployments require two RDS Aurora Clusters for the duration of the operation, whether you choose to use AWS's offering or not.

Then there's the extra you will need in order to implement your own Blue-Green Deployment: a highly reusable script that checks the cluster to be upgraded matches all the necessary pre-requisites and executes the switchover for you, hooking up Old Blue as a replica of New Blue afterwards. The last time I did it, it costed me a sprint to write, test, and document a script like this.

Return on Investment

And here's what I made out of that comparatively small investment.

Remember, dear reader, that there are two main goals here: minimise service interruptions and drastically reduce Mean-Time-To-Recover (MTTR) in case of a rollback.

The last time I went through this situation, restoring the database from a backup required about one hour of service interruption. It would also require manually reconciling data written to the New Blue database, a task so difficult wanted to avoid it at all costs. Implementing our own Blue-Green Deployment strategy reduced our MTTR by approximately 99.17%, from one hour to about 30 seconds.

Our internal availability target for the platform (no, not the one we publicly committed to, that was much lower) was 99.995% or 133.56 seconds per month (Calculation mnemonic: (53 weeks * 7 days * 24 hours * 60 minutes * 60 seconds * (1-0.99995) error budget) / 12 months = 133.56 downtime seconds / month).

I had about 270 databases to upgrade. While not all of these would have caused a site-wide outage, we couldn't be totally sure about what would. So we very conservatively assumed all outages were site-wide for the sake of this calculation.

By delaying database connections at the proxy and reconnecting them to the new database servers, I was able to avoid API delays and fool about 95% of my services into thinking nothing happened. Yes, I had to restart a few services that got confused when the session variables they set in the connection "changed values" (that is, the new connection didn't have the values they expected). This fit neatly into my (overly) adjusted error budget.

Conclusion

I hope my observations and considerations here have given you food for thought. It was never my intention to publish a prêt-à-porter solution. Every company and every situation is unique, and you must exercise judgement about your specific circumstances and requirements before adopting the strategy I describe here. Please feel free to write to me. I'd love to hear your comments, questions, or thoughts on this topic.