Production deployment with Azure App Service slots and Entity Framework Core migrations

I recently made some progress on this blog and wanted to push the latest changes live. I've been using EF Core migrations to assist with any data related changes, and it has worked out very well for me during development. When it's time to push to production I expect it to work just as well, but nonetheless I want to first see what others think about using EF migrations against a production database.  So I found this StackOverflow question Is it OK to update a production database with EF migrations? The short answer is YES! Now having done this myself, I can say that EF migrations used in conjunction with Azure deployment slots makes production deployment really easy. In this post I want to explain my setup and the simple steps I followed to upgrade this site.

Deployment slots

Azure app service deployment slots give you the ability to setup multiple environments like production, staging and test, and quickly swap between them.

1-azure-deployment-slots
My Azure deployment slots and databases

This feature is only available on standard service plan or higher. This site was running on Azure B1 App Service plan, so I had to scale it up to a S1. With the standard plan I created a Staging slot.  Also previously I had a S0 SQL Database as my production database, this time I created a Basic SQL Database to serve as the Staging database.

Here I'm trying to save money so I only created two environments, production and staging, and let staging serve as testing.  Theoretically staging should closely mimic production and even share same database.   

In addition to app services and databases I also have separate Application Insights and Blob Storage accounts for each of the production and staging environment, they are not shown on the diagram.

Deployment steps

2-azure-app-service-deployment-slot-swap
My Azure deployment flow

 

Step 1

Last year I had my v1.0 app deployed from github on to the production app service, and EF Core automatically populated the production database. This week I did the same thing for the Staging app service and database. 

So by this time the Production and Staging environments have the same exact code and DB schema.

Step 2

I deployed the v1.1 alpha to staging slot and let EF upgrade Staging DB to the newer v1.1 schema. Here I actually have a choice of running a SQL upgrade script myself to upgrade the database or letting EF do it, more on this later. 

This step is critical, because if it works then when I push v1.1 to production it should work too.  But I did encounter a BadImageFormatException right after I pushed v1.1 to staging, this error never occurred to me during development and it took me a while to track down what was happening.  The point is by deploying to staging first gave me the chance to fix that.  

At this moment the entire Staging web app and database got the latest code and working well.

Step 3

3-azure-slot-settings
Remember to check mark your slot-specific settings before swapping!

 

It was time to do the swap, but before doing that I made sure I had the slot-specific settings in each production and staging's Application Settings checked. Each environment has their own database, application insights and blob storage, by check marking them these settings are not going with the swap.

4-azure-slot-swap-source-destination
Before swapping setup Source and Destination

Now I was ready to do the swap and I had a choice of doing a Swap or Swap with preview. In either case let's make staging the Source and production the Destination

Doing Swap will start swapping the source and destination immediately, and the first time I did this it took about 1 minute and 15 seconds to complete.  Basically Azure first warms up the staging slot and then completes the actual swap. 

The Swap with preview on the other hand is not immediate, according to the Azure Doc the following happens.

  • It keeps production unchanged so existing workload on that slot is not impacted.
  • It applies configurations of production slot to staging slot, including the production slot-specific settings!
  • It restarts the worker processes on the staging slot using these production configuration elements.
  • When you complete the swap: it moves the pre-warmed-up staging slot into the production slot, and production slot into the staging slot as in a manual swap.
  • When you cancel the swap: it reapplies the settings of the staging slot to the staging slot.

Below is a screen shot that shows when you do Swap with Preview the connection strings used by both slots are the same.

5-azure-swap-preview-changes
Azure slot swap Preview Changes shows both Production and Staging using same connection strings

 

Step 4

When you preview swap, the staging site shows up with production data, in other words Azure warms up the staging slot.  So when you are ready to Complete swap it's actually done more efficiently than previously when I did the direct swap, the complete swap step took only about 20 to 30 seconds.

6-azure-slot-complete-swap
Complete swap after previewing staging

 

EF Core Migrations

For the upcoming Fanray v1.1 I made a series of changes to the database schema, including add and rename columns, make column nullable, update existing data and insert new data.  As I mentioned in step 2 I had a choice on how to do the database upgrade, I could either let EF do it automatically or I can use EF to generate an upgrade SQL script first then run that against the database. In some organizations they prefer to have a DBA look over any SQL upgrade script first.  I have tried both ways and both worked perfectly.

To generate the SQL script you can either run the EF PowerShell command Script-Migration or the dotnet CLI command dotnet ef migrations script

Final thoughts on what happens during the upgrade 

Remember back in step 3 during the swap preview, the staging actually gets all the production's settings.  As a result EF is actually upgrading your production database at this point already.  Based on the success of step 2 I know this upgrade should work, but while that's happening remember the production website is still running v1.0 of the app against the same database which staging is upgrading to v1.1! 

I tested locally my v1.0 app does work with v1.1 DB schema, but this obviously is an issue if there are breaking changes. The site visitors may experience error during that one-minute swap on the live site.  The first thing comes to my mind to remedy this is to use an old trick app_offline.htm, as discussed in this SO question How to use IIS app_offline.htm file with Azure. The downside of doing this is that even though the swap happens pretty quickly but still during that time your site is down to your visitors.

One of the answers on that SO question mentioned "you should be able to virtually eliminate down time with Azure by running multiple instances".  As I explained above I'm not sure that's the case.  The comment left below this answer is more inline with what I have noted.

My co-founder is actually the Azure expert on our team, and we are already running multiple instances with SQL Azure. However, earlier today, he needed to update the DB schema which meant that part of the site was down for several minutes. When I hit the site, I was redirected to my main ErrorPage. But I would have preferred to have had the app_offline.htm file in the root during those few minutes. I was just under the impression that it's non trivial to be doing file I/O related things on an Azure deployment.

Also Azure provides SQL Database backup so if upgrading the production database fails you can restore it from your backup.  This has been my deployment flow so far, but is there a better way or how are you guys approaching the deployment and upgrade of the production database?  Please let me know what you think.