Lower your build costs with Jenkins and EC2 Spot Instances
posted | about 7 minutes to read
tags: amazon web services continuous integration devops jenkins tutorial
Sometimes, a company may choose to avoid continuous integration due to the cost constraints of having a dedicated build server if working on-premises, or the high hourly cost of a powerful build server in the cloud. In a personal project I was working on recently, I ran right up against these cost constraints on a smaller scale - I didn’t want to wait forever for my builds to finish, but I also didn’t want to spend at a higher hourly rate to keep a build server up! Fortunately, Amazon Web Services has a service that’s really tailored to exactly this use case called spot instances. The idea, for those who aren’t familiar with it, is that you can get a powerful instance at a small fraction of its normal hourly rate. I think the idea is that they’re selling their excess compute power so it’s not just sitting there idle. In any case, I’ve been paying about 30% of the normal hourly price for my c4.xlarge instance, which is a really great cost savings - plus, by integrating it with my Jenkins CI instance, I can make sure that the beefy build server only spins up when I need it, saving me from having to keep the server up 24/7.
For this walkthrough, you’ll need an AWS account and a Jenkins server with the Amazon EC2 plugin installed on it.
Setting Up An AMI
The first step that you’ll need to take is actually preparing a system image for the Jenkins server to spin up and connect to to run a build. The first step here is to spin up a new instance in AWS and install required packages. The only “must-haves” for this is installing the OpenJRE 8 package and running the instance start script - other than that, pretty much anything else is optional. I prefer to bake in any libraries I need to build to the AMI and then just run a package update as part of the build process, but that’s kind of up to you.
The instance start script, incidentally, is located in your Jenkins install after you install the Amazon EC2 plugin - further instructions are located on the plugin page under “Configure AMI for Spot Support”. Unfortunately the plugin files do require authentication to get at in a typical Jenkins environment, so I recommend either putting your Jenkins instance in a very restricted security group and making it public, or copying the Python script to your machine manually and putting it in /usr/bin/userdata. The sample script’s written for Ubuntu, so you’ll need to modify it for your own environment if you use another distribution.
One thing that I do think is probably worth doing is setting up AWS CLI integration so that you can upload build artifacts to Amazon S3. That’s a little outside the scope of this blog, but I explain it more in this post. It’s also optional - if your workflow requires something else, you can set up whatever you need for that as well.
Once you’ve got your machine set up, go ahead and select it in the EC2 Management Console. Under the Actions menu, select Image > Create Image. Wait for this to kick off, then go look under your list of AMI’s - you’ll be able to see the AMI ID. Go ahead and write that down somewhere, since you’ll need it for the next step. Also, make sure that you have the private key for accessing the instance available, as Jenkins will also need that.
Finally, go ahead and set up a security group for your build nodes. Typically, this should allow SSH from only the Jenkins server, as they’ll be otherwise completely noninteractive.
Configuring Jenkins
Jenkins CI is probably the most crucial part of this whole thing. It’s a free, open-source program that allows you to mange your builds and includes plugins to integrate with pretty much any service you could want. I use the GitHub plugin to grab new revisions as they’re committed, the Slack and email notification plugins to make sure I know when a build’s failed, and, of course, the Amazon EC2 plugin to manage my build instances. For the purpose of this walkthrough, I’m going to assume that you already have a build script in place, and will limit myself to discussing AWS IAM and Jenkins setup.
Before you even worry about setting up your Jenkins server, you’ll need to create an IAM identity so that Jenkins can talk back to Amazon and spin up spot instances on the fly. Create a “jenkins-ci” user in IAM, and attach the following policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1312295543082",
"Action": [
"ec2:DescribeSpotInstanceRequests",
"ec2:CancelSpotInstanceRequests",
"ec2:DescribeSpotPriceHistory",
"ec2:GetConsoleOutput",
"ec2:RequestSpotInstances",
"ec2:RunInstances",
"ec2:StartInstances",
"ec2:StopInstances",
"ec2:TerminateInstances",
"ec2:CreateTags",
"ec2:DeleteTags",
"ec2:DescribeInstances",
"ec2:DescribeKeyPairs",
"ec2:DescribeRegions",
"ec2:DescribeImages",
"ec2:DescribeAvailabilityZones",
"ec2:DescribeSecurityGroups",
"ec2:DescribeSubnets",
"iam:PassRole"
],
"Effect": "Allow",
"Resource": "*"
}
]
}
This basically allows Jenkins to actually spin up instances for builds.
When you create the user, also make sure you take note of the Access Key and Secret Key - you’ll need them both when configuring the Amazon EC2 Plugin.
This is all of the configuration that we’ll need to do in AWS. Now, open up your Jenkins web console and go to Manage Jenkins > Configure System. Scroll down until you find the “Cloud” section. Click “Add a new cloud”, and select “Amazon EC2”. Give it a name, then next to “Amazon EC2 Credentials” click “Add”, and configure a new credential of type “AWS Credentials”. Use the Access Key and Secret Key that we set up in IAM. Please note that there’s also an option to obtain credentials from the EC2 instance profile; this may be a better solution for you depending on your Jenkins security needs since it abstracts the credentials out of Jenkins, but it’s beyond the scope of this guide. Finally, select a region, and then paste the private key from the image we created earlier. If you’d like, you can now test the connection to make sure that you can connect to EC2 correctly.
Now, we need to set up the AMI in Jenkins. In the AMIs section, click Add, add a description of your image (“APPLICATIONNAME build server” is perfectly fine), and then paste the AMI ID that you saved during image creation. You can click the Check AMI button to make sure that it’s available to Jenkins. Select an instance type that meets your price/performance requirements, and check the “Use Spot Instance” box. It’s worth noting that spot prices may differ between availability zones, so you may need to update this to fit your needs. Add a maximum bid price, formatted as “0.05”, for example, and type in the security group name that you set up previously. I usually set the remote filesystem root to /tmp/jenkins.Configure the rest of the settings to be in keeping with your environment. Define a label - I use “APPNAME-compiler” - and then set usage to “Only build jobs with label expressions matching this node”. The “Idle termination time” field is the number of minutes a build server should be idle before shutting down - I recommend setting this value to 10 to control costs well.
Finally, now that that’s done, you’re ready to set up your Jenkins build job to work with your EC2 spot instance. Create a new job, or if you already have a job set up open it. Set the label for the job to the label you set in the Amazon EC2 plugin configuration, and then set up your job to run as normal. A couple things that might differ - if you need to install libraries on the system, you’ll need to add a shell script to do that, or if you already did that as part of the AMI you may want to update them at the start of the build. Also, you’ll need to upload your build artifacts somewhere that your deployment process can access them; I use Amazon S3, but you can pretty much use whatever.
At this point, you should have a completely configured system to build your jobs on spot instances in Amazon EC2! If you’re running into issues with instances not spinning up, one of the first things to check is if the current spot instance price exceeds your configured bid price in the plugin. If so, you’ll need to update your bid price or the availability zone in which you’re spinning up your instances and cancel the existing spot request in the AWS EC2 console. Once you’ve done that, Jenkins will automatically create a new spot request for you.
I know this was probably a long read, but I hope it’s helped you explore cost control in your build processes! If you have any questions, don’t hesitate to ask them in the comments section below.