How I Learned to Stop Worrying and Love the Termination Notification

I recently moved all my compute resources over to Spot Instances in AWS. I love it. If you’re not using Spot in your applications, I hope this will convince you to look into it.

Your standard instance is known as an On Demand instance, meaning you launch and terminate as needed. For this, you pay a set amount per minute. Once launched, the instance will continue to operate until something terminates it.

Reasons for termination:

When a new instance is requested, AWS provides them on a best-effort basis. I have encountered situations where a specific instance type in a specific Availability Zone isn’t available. This happens infrequently, but is always a possibility. When this happens, the only option is to wait for more capacity to become available, or move the workload to a different instance class and/or a different AZ.

Spot Instances are just like On Demand instances, except for a few key points:

As long as the price specified is higher than the current spot market price, and there is capacity, the request will be fulfilled.

Formerly, the Spot Market operated on bids. That meant that another customer launching an instance with a higher bid price than you could cause you to lose your spot instance. Back in November, AWS changed the way Spot works to now only terminate spot instances when there isn’t enough capacity to fulfill an On Demand request or the market price rises above your max specified. The market price previously re-evaluated every few minutes, but now happens less than a handful of times per day, if it happens at all that day. This means the spot price is far more stable than in the past. There may be other customers willing to pay more than you, but if AWS doesn’t re-evaluate the market, the price won’t change and your instance won’t be terminated.

Reserved Instances tell Amazon that you want to run a given On Demand instance for 1 or 3 years, and by reserving it, they give you a discount. Depending on the options, you get on average 30% to 50% off the On Demand price. If you choose to buy that reservation in a specific Availability Zone, then it also provides guaranteed capacity. It’ll always be there for you to use. Getting that reserved capacity comes at a trade-off, you have to use that specific AZ. If you’re experiencing an outage related to an AZ, you can’t move that reserved capacity to another. You’re stuck either buying reservations in multiple AZs or running On Demand in a different AZ. If your footprint is small (like a few instances), running RI might make sense.

Switching to spot increases your operational integrity:

To gain a high level of confidence in available capacity, I use Spot Fleets. Spot fleets allow you to define which instance types you want to use and the most you’d like to pay per instance hour. Fleets also function like Auto Scaling Groups, with options for scaling actions and maintaining instance counts. In a given spot fleet, you have a concept of an Instance Pool, which is a combination of an instance type and an AZ. If you have a spot fleet with 4 instances in 3 AZs, you have (4 x 3) 12 Instance Pools. You can use the AvailableInstancePoolsCount metric in CloudWatch to see how diverse your fleets are. Specifying a max price in your fleet configuration lower than the current market price reduces this metric, so it’s a good metric to monitor for overall health of your fleets.

The first thing needed to properly leverage Spot Instances is automated instance deployment. Either by using configuration management software or machine images to ensure your instances are configured the way you need when they boot up. I’ve used both methods and heavily recommend a software-based approach.

While working on that, you should also look at your applications and make sure they can gracefully shut down within 2 minutes of notification. If they can’t, you’ll want to have an automated process for recovering the state of that application when you lose an instance. It’s just an additional benefit that this helps the health of your apps.

These two steps will get you most of the way there. It’s just a matter of reading the docs and putting it all together. Or, find someone who’s already done it.

If you make the leap to Spot Instance, I’d love to hear about your experiences. Send me a message here, or in email: levi@levimccormick.com

Related posts:

now i have to kee to this daily time line and get through september to april next year to dul 365, with no breaks, maybe a few break s bu t i will do the 365 again

Peep the latest episode of Touchdowns and Tangents on our website below.. “Trading teams is an Ultralight Scheme” is published by Tds & Tangents.

Prophet Muhammad predicted that tunnels will be dug in Mecca as well as buildings exceeding the height of mountains. This has come true. “When the belly of Makkah will be cleft open and through it…