1,500 people at launch, no downtime

The context

A web platform with a members' area was about to open to the public. More than 1,500 people were expected on launch day, almost all of them at the same moment.

The problem

Everything ran on a single server: application, database and services. If that machine stopped, everything stopped.

Then there was a problem nobody could see. In the load tests, sign-ups were queuing up: 95% completed within 8 seconds, but some took as long as 24. For someone signing up on launch day, 24 seconds means a closed page.

The solution

First I looked for the cause, then I thought about the servers.

• The bottleneck in the code. Sign-ups were waiting for one another because of a lock in the code. I fixed that first.
• Servers that add themselves. A load balancer spreads the traffic across a group that grows from 2 to 6 servers depending on the load.
• Managed databases and queues. They are managed cloud services, with the data in a data centre in Milan.
• Releases with a safety net. If something goes wrong after a release, the platform rolls back to the previous version on its own.
• Load tests. I simulated the launch crowd before the switchover, not after.
• The switchover. I moved the platform three days before the launch and checked that the data was identical to the original.

The result

• Sign-ups are 86% faster: 95% complete within 1.1 seconds, compared with 8 before.
• In the load tests, errors were 0%.
• In the checks, 0% of responses contained another user's data.
• The launch went ahead on the planned date.
• During the launch days the platform ran on three servers instead of two, as a precaution.
• The day after, with real traffic, the servers were running at 5% CPU and 12% memory, handling a few hundred requests a minute.
• The next release went through without a single alert.

What to check before a launch

• Before adding servers, find where the work gets stuck: here the bottleneck was in the code.
• Run load tests with the number of people you really expect, before launch day.
• Under load, as well as speed, check that everyone sees only their own data.
• A migration is finished when the data matches, not when the copy ends.

Details have been changed to protect the client's confidentiality.