March 3, 2017

Creating Amazon EC2 AMIs - the basics

This morning at the Codefest 2010 I was asked to give an impromptu presentation about the basics of getting an EC2 image up and running. Here is a brief summary of how it all works, using the command-line tools. Of course you can use the EC2 console as it can do it all for you, but knowing how the command-line works is a useful thing to do too! The first thing you'll need is an EC2 account. It's easy to sign up for one at Amazon. Once you've got it, you'll receive a pk- and cert-*.pem certificate pair. Keep these safe, you'll need them later! You'll also receive an account number which if you don't know where to find it, take a look through the Amazon billing pages as it'll appear on your activity and invoice summaries. Now for the real work. Log in to the AWS console and select the 'Instances' option in the menu on the left. For this demo, fire up a basic Fedora Core 8 Minimal. You can of course use Ubuntu etc. but the commands in this blog refer to Fedora. Leave all the settings at their defaults. It'll prompt you to create a keypair, which you'll need to save somewhere safe as you'll need this to ssh into the instance later. If you already have one, select it instead of creating a new one. It'll also ask you to set up a Security Group, which is really a firewall but under a different name. Make sure you keep port 22 open else you won't be able to ssh in! If you want to use it as a web server, open port 80, or a MySQL server, port 3306, etc. Check the summary screen in the last stage of launching, then hit Launch. Your instance will be started. Once it's started (when the spinning wheel in the Instances tab has stopped spinning) select it in the instances list and use the 'Instance actions' button to find the 'Connect' menu item. In the window that appears you'll find an ssh command to log into the instance with. You'll need to run that command from the same location as where you saved your ssh keypair. After ssh-ing in, the machine is yours to do what you like with. Install stuff, reconfigure it, but never ever upgrade or modify the kernel as this'll break it horribly (unless you really know what you're doing, in which case you probably won't be reading this blog entry anyway...). After you're happy with the way you've configured your new machine, you will probably want to share it with other people so that they can start up machines exactly like it without having to do all that configuration work themselves. Luckily, this is quite easy! The following steps can be run locally from your desktop if you want, but it's easier (and quicker) to do them inside the instance itself. Note that they involve copying your pk- and cert-*.pem files up to the instance in order to be able to identify yourself to Amazon's API for creating new images. Don't worry, these won't be kept in the final image, so they won't be visible to anyone else. First up, you'll need to install the relevant tools for doing the image creation. These require Java, so install Java first (you can use the Sun one, or you can use the icedtea free version - other free versions haven't worked so far but they may do in future):

yum install java-1.7.0-icedtea export JAVA_HOME=/usr/lib/jvm/jre-1.7.0-icedtea/

Then install the API tools themselves:

wget unzip rm mv ec2-api-tools* ec2-api-tools export EC2_HOME=/root/ec2-api-tools export PATH=$PATH:$EC2_HOME/bin

Now, scp up your pk- and cert-*.pem files to /tmp using a command very similar to the ssh command you used to log in originally. By putting them in /tmp they won't be replicated in the finished instance, but if you do put them somewhere else don't forget to use the -e clause in ec2-bundle-vol later on:

scp -i <mykeypair>.pem {pk,cert}*.pem root@<instance ip address>:/tmp

(You can get the instance IP address from the AWS console). Then in the instance itself, tell it where these are:

export EC2_PRIVATE_KEY=`ls -1 /tmp/pk-*.pem` export EC2_CERT=`ls -1 /tmp/cert-*.pem`

Now we're ready to go! Choose a name for your new image. It might be easier to set it as an environment variable:

export BUNDLE_NAME=codefest-demo

The following commands will construct the image, upload it to EC2, and register the AMI so that you can use it in future (the Amazon access key and secret key refer to additional security features that you will have received when registering your account). If you put your pk- and cert-*.pem files somewhere else, you'll need to add the -e clause to ec2-bundle-vol, e.g. "-e $EC2_PRIVATE_KEY,$EC2_CERT":

ec2-bundle-vol -u <account number> -r i386 -c $EC2_CERT -k $EC2_PRIVATE_KEY ec2-upload-bundle -b $BUNDLE_NAME -m /tmp/image.manifest.xml -a <access key> -s <secret key> ec2-register $BUNDLE_NAME/image.manifest.xml -n $BUNDLE_NAME

Note that the AMI created will be stored in an S3 bucket, and will only be available for you to use. If you want to share it, use the AMI option in the console menu to bring up the AMIs tab, select the AMI you want to share, then use the 'Permissions' button to grant permission to others. You can make it public, or you can share it with specific people (you'll need their Amazon account numbers). That's it! Hope it helps.

Topics: AWS, Big data technology, Bioinformatics, Cloud, EC2