DataStax Developer Blog

Personalizing your own Brisk AMI

By Joaquin Casares -  June 14, 2011 | 2 Comments

This is an informational post on how to take the DataStax AMI code and create your own AMIs. Creating your own self-clustering AMIs can be useful for internal company projects or highly personalized, easy to launch setups. In comparison to Chef and Puppet, self-clustering AMIs allow the end user to skip installation and configuration and launch Brisk on the spot.

The Brisk AMI code can be found at: https://github.com/riptano/BriskClusterAMI.

EBS vs Instance Store

This tutorial shows how to create instance store images rather than EBS images. EBS-backed instances tend to have random disk delays and are not as efficient as instance-store instances as seen when running benchmarks. In the event that you still wish to create EBS-backed instances, all that would be needed is to follow the instructions in the “Prime the Image” section.

For further reading on why to choose image-store instances over EBS instances, please visit:

http://www.mail-archive.com/user@cassandra.apache.org/msg11022.html

Pre Setup

AWS ID

Visit the following link and make note of the “Account Number” in the top right corner under your name. This number, minus the dashes, is your AWS ID and will be used in the next steps.

https://aws-portal.amazon.com/gp/aws/developer/account/index.html?ie=UTF8&action=access-key

Access Keys

In the same link as above, on the “Access Keys” tab, make note of your Access Key ID and your Secret Access Key as they will also be used in the next steps.

Pem Files

In the same link as above, open the “X.509 Certificate” tab and click “Create New.” Download both of those files to your local machine.

Download EC2 Command-Line Tools

Run the following commands on your local machine:

cd
wget http://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip
unzip ec2-api-tools.zip
mv ec2-api-tools-1.4.3.0/ .ec2
rm ec2-api-tools.zip
cd .ec2
open .

Then copy your cert-xxxxx.pem and pk-xxxxx.pem files into ~/.ec2.

Setup EC2 Command-Line Tools

Add the following to your ~/.bash_profile:

# Setup Amazon EC2 Command-Line Tools
export EC2_HOME=~/.ec2
export PATH=$PATH:$EC2_HOME/bin
export EC2_PRIVATE_KEY=`ls $EC2_HOME/pk-*.pem`
export EC2_CERT=`ls $EC2_HOME/cert-*.pem`
export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Home/

Then execute:

source ~/.bash_profile

For more information about EC2 Command-Line Tools on Mac OSX, visit this tutorial.

Connect to a New Instance

Suggested AMI ID’s can be found in the header of .presetup/pre-setup.sh.

When connecting to the new instance be sure to transfer cert-xxxxx.pem and pk-xxxxx.pem to the home folder of your EC2 image before priming the image. (.presetup/pre-setup.sh will clear your current access keys making scp impossible on this machine later.)

Prime the Image

Copy and paste the multiple sections of .presetup/pre-setup.sh into your linux prompt inside of the EC2 instance.

https://github.com/riptano/BriskClusterAMI/blob/master/.presetup/pre-setup.sh

What Happens

.presetup/run.py is executed and the entire image is setup and installed with the additional useful programs. Some of these programs and other linux settings are primed. This is all done by the developer.

.launcher.py is run on startup and launches all the configurations and starts the services when the image is launched by the client.

Saving the Image

The following is our internal code for saving the DataStax Images. Visit this tutorial for more background information about customizing and saving AMI images.

AWSID=<AWS ID>
ACCESSKEYID=<ACCESS KEY ID>
SECRETACCESSKEY=<SECRET ACCESS KEY>
sudo mv *.pem /mnt
sudo su
cd
curl http://s3.amazonaws.com/ec2-downloads/ec2-ami-tools.zip > ec2-ami-tools.zip
mkdir ec2
cp ec2-ami-tools.zip ec2
cd ec2
unzip ec2-ami-tools.zip
ln -s ec2-ami-tools-* current
sudo cat >> ~/.bashrc << EOF
export EC2_AMITOOL_HOME=~/ec2/current
export PATH=${PATH}:~/ec2/current/bin
EOF
source ~/.bashrc
ec2-bundle-vol –help

VERSION=$(head -1 /home/ubuntu/.presetup/VERSION)

rm -rf ~/.bash_history && history -c && ec2-bundle-vol -p brisk_1.0_beta1-ami_$VERSION -d /mnt -k /mnt/pk-xxxxxxx.pem -c /mnt/cert-xxxxxxx.pem -u $AWSID -r x86_64

# Choose only one of the lines from this block depending on which region you are currently logged into
yes | ec2-upload-bundle -b datastax-brisk-cluster-ami-use -m /mnt/brisk_1.0_beta1-ami_$VERSION.manifest.xml -a $ACCESSKEYID -s $SECRETACCESSKEY –location US
yes | ec2-upload-bundle -b datastax-brisk-cluster-ami-usw -m /mnt/brisk_1.0_beta1-ami_$VERSION.manifest.xml -a $ACCESSKEYID -s $SECRETACCESSKEY –location us-west-1
yes | ec2-upload-bundle -b datastax-brisk-cluster-ami-eu  -m /mnt/brisk_1.0_beta1-ami_$VERSION.manifest.xml -a $ACCESSKEYID -s $SECRETACCESSKEY –location EU
yes | ec2-upload-bundle -b datastax-brisk-cluster-ami-aps -m /mnt/brisk_1.0_beta1-ami_$VERSION.manifest.xml -a $ACCESSKEYID -s $SECRETACCESSKEY –location ap-southeast-1
yes | ec2-upload-bundle -b datastax-brisk-cluster-ami-apn -m /mnt/brisk_1.0_beta1-ami_$VERSION.manifest.xml -a $ACCESSKEYID -s $SECRETACCESSKEY –location ap-northeast-1

echo “”"
cd ~/.ec2/
ec2-register datastax-brisk-cluster-ami-use/brisk_1.0_beta1-ami_$VERSION.manifest.xml –region us-east-1 -n ‘DataStax Brisk 1.0 Beta-1 Cluster – AMI $VERSION’ -d ‘Provides a way to automatically launch a Brisk cluster by simply starting the instance.’
ec2-register datastax-brisk-cluster-ami-usw/brisk_1.0_beta1-ami_$VERSION.manifest.xml –region us-west-1 -n ‘DataStax Brisk 1.0 Beta-1 Cluster – AMI $VERSION’ -d ‘Provides a way to automatically launch a Brisk cluster by simply starting the instance.’
ec2-register datastax-brisk-cluster-ami-eu/brisk_1.0_beta1-ami_$VERSION.manifest.xml –region eu-west-1 -n ‘DataStax Brisk 1.0 Beta-1 Cluster – AMI $VERSION’ -d ‘Provides a way to automatically launch a Brisk cluster by simply starting the instance.’
ec2-register datastax-brisk-cluster-ami-aps/brisk_1.0_beta1-ami_$VERSION.manifest.xml –region ap-southeast-1 -n ‘DataStax Brisk 1.0 Beta-1 Cluster – AMI $VERSION’ -d ‘Provides a way to automatically launch a Brisk cluster by simply starting the instance.’
ec2-register datastax-brisk-cluster-ami-apn/brisk_1.0_beta1-ami_$VERSION.manifest.xml –region ap-northeast-1 -n ‘DataStax Brisk 1.0 Beta-1 Cluster – AMI $VERSION’ -d ‘Provides a way to automatically launch a Brisk cluster by simply starting the instance.’

# Duplicate this line and change to add internal-only permissions
ec2-modify-image-attribute -l –region us-east-1 -a EMPLOYEEAWSID \$(ec2-describe-images –region us-east-1 | grep ‘DataStax Brisk 1.0 Beta-1 Cluster – AMI $VERSION’ | cut -f2)

“”" && rm -rf ~/.bash_history && history -c

The above code will move your *.pem files to your /mnt directory, which is never imaged, and install EC2′s AMI Tools. It will then image your instance and upload it to S3 based on which region you are in. It then echos the correct commands to run on your local machine to register the image and modify the permissions for internal-use-only images.

The code is an active project and may change at any point. Do let us know if there is anything you want to see, or submit a pull request for anything you feel will be beneficial. We’re constantly trying to find ways to make this easy to follow and adaptable to all situations.

Feel free to leave any questions and comments in the support forums and we’ll make sure to address all issues and concerns.



Comments

  1. Chris Dodge says:

    Hi,

    Thanks for the info. I would line to make an EBS backed instance of Cassandra on EC2.. This is for development purposes an I’d like to be to turn on/off instances to save money when we are not working on it.

    However your above link to https://github.com/riptano/BriskClusterAMI/blob/master/.presetup/pre-setup.sh is no longer valid. Can you repost? Thank you.

  2. Joaquin Casares says:

    Hello Chris,

    The new scripts are located here: https://github.com/riptano/ComboAMI/tree/2.2/presetup.

    There are now two scripts: pre_1.sh and pre_2.sh. Copy and paste pre_1.sh into your console and pre_2.sh will come through automatically. Hope this helps!

    Joaquin

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>