Personalizing your own Brisk AMI
This is an informational post on how to take the DataStax AMI code and create your own AMIs. Creating your own self-clustering AMIs can be useful for internal company projects or highly personalized, easy to launch setups. In comparison to Chef and Puppet, self-clustering AMIs allow the end user to skip installation and configuration and launch Brisk on the spot.
The Brisk AMI code can be found at: =https://github.com/riptano/BriskClusterAMI.
EBS vs Instance Store
This tutorial shows how to create instance store images rather than EBS images. EBS-backed instances tend to have random disk delays and are not as efficient as instance-store instances as seen when running benchmarks. In the event that you still wish to create EBS-backed instances, all that would be needed is to follow the instructions in the "Prime the Image" section.
For further reading on why to choose image-store instances over EBS instances, please visit:
Visit the following link and make note of the "Account Number" in the top right corner under your name. This number, minus the dashes, is your AWS ID and will be used in the next steps.
In the same link as above, on the "Access Keys" tab, make note of your Access Key ID and your Secret Access Key as they will also be used in the next steps.
In the same link as above, open the "X.509 Certificate" tab and click "Create New." Download both of those files to your local machine.
Download EC2 Command-Line Tools
Run the following commands on your local machine:
mv ec2-api-tools-18.104.22.168/ .ec2
Then copy your cert-xxxxx.pem and pk-xxxxx.pem files into ~/.ec2.
Setup EC2 Command-Line Tools
Add the following to your ~/.bash_profile:
# Setup Amazon EC2 Command-Line Tools export EC2_HOME=~/.ec2 export PATH=$PATH:$EC2_HOME/bin export EC2_PRIVATE_KEY=`ls $EC2_HOME/pk-*.pem` export EC2_CERT=`ls $EC2_HOME/cert-*.pem` export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Home/
For more information about EC2 Command-Line Tools on Mac OSX, visit this tutorial.
Connect to a New Instance
Suggested AMI ID's can be found in the header of .presetup/pre-setup.sh.
When connecting to the new instance be sure to transfer cert-xxxxx.pem and pk-xxxxx.pem to the home folder of your EC2 image before priming the image. (.presetup/pre-setup.sh will clear your current access keys making scp impossible on this machine later.)
Prime the Image
Copy and paste the multiple sections of .presetup/pre-setup.sh into your linux prompt inside of the EC2 instance.
.presetup/run.py is executed and the entire image is setup and installed with the additional useful programs. Some of these programs and other linux settings are primed. This is all done by the developer.
.launcher.py is run on startup and launches all the configurations and starts the services when the image is launched by the client.
Saving the Image
The following is our internal code for saving the DataStax Images. Visit this tutorial for more background information about customizing and saving AMI images.
ACCESSKEYID=<ACCESS KEY ID>
SECRETACCESSKEY=<SECRET ACCESS KEY>
sudo mv *.pem /mnt
curl http://s3.amazonaws.com/ec2-downloads/ec2-ami-tools.zip > ec2-ami-tools.zip
cp ec2-ami-tools.zip ec2
ln -s ec2-ami-tools-* current
sudo cat >> ~/.bashrc << EOF
VERSION=$(head -1 /home/ubuntu/.presetup/VERSION)
rm -rf ~/.bash_history && history -c && ec2-bundle-vol -p brisk_1.0_beta1-ami_$VERSION -d /mnt -k /mnt/pk-xxxxxxx.pem -c /mnt/cert-xxxxxxx.pem -u $AWSID -r x86_64
# Choose only one of the lines from this block depending on which region you are currently logged into
yes | ec2-upload-bundle -b datastax-brisk-cluster-ami-use -m /mnt/brisk_1.0_beta1-ami_$VERSION.manifest.xml -a $ACCESSKEYID -s $SECRETACCESSKEY --location US
yes | ec2-upload-bundle -b datastax-brisk-cluster-ami-usw -m /mnt/brisk_1.0_beta1-ami_$VERSION.manifest.xml -a $ACCESSKEYID -s $SECRETACCESSKEY --location us-west-1
yes | ec2-upload-bundle -b datastax-brisk-cluster-ami-eu -m /mnt/brisk_1.0_beta1-ami_$VERSION.manifest.xml -a $ACCESSKEYID -s $SECRETACCESSKEY --location EU
yes | ec2-upload-bundle -b datastax-brisk-cluster-ami-aps -m /mnt/brisk_1.0_beta1-ami_$VERSION.manifest.xml -a $ACCESSKEYID -s $SECRETACCESSKEY --location ap-southeast-1
yes | ec2-upload-bundle -b datastax-brisk-cluster-ami-apn -m /mnt/brisk_1.0_beta1-ami_$VERSION.manifest.xml -a $ACCESSKEYID -s $SECRETACCESSKEY --location ap-northeast-1
ec2-register datastax-brisk-cluster-ami-use/brisk_1.0_beta1-ami_$VERSION.manifest.xml --region us-east-1 -n 'DataStax Brisk 1.0 Beta-1 Cluster - AMI $VERSION' -d 'Provides a way to automatically launch a Brisk cluster by simply starting the instance.'
ec2-register datastax-brisk-cluster-ami-usw/brisk_1.0_beta1-ami_$VERSION.manifest.xml --region us-west-1 -n 'DataStax Brisk 1.0 Beta-1 Cluster - AMI $VERSION' -d 'Provides a way to automatically launch a Brisk cluster by simply starting the instance.'
ec2-register datastax-brisk-cluster-ami-eu/brisk_1.0_beta1-ami_$VERSION.manifest.xml --region eu-west-1 -n 'DataStax Brisk 1.0 Beta-1 Cluster - AMI $VERSION' -d 'Provides a way to automatically launch a Brisk cluster by simply starting the instance.'
ec2-register datastax-brisk-cluster-ami-aps/brisk_1.0_beta1-ami_$VERSION.manifest.xml --region ap-southeast-1 -n 'DataStax Brisk 1.0 Beta-1 Cluster - AMI $VERSION' -d 'Provides a way to automatically launch a Brisk cluster by simply starting the instance.'
ec2-register datastax-brisk-cluster-ami-apn/brisk_1.0_beta1-ami_$VERSION.manifest.xml --region ap-northeast-1 -n 'DataStax Brisk 1.0 Beta-1 Cluster - AMI $VERSION' -d 'Provides a way to automatically launch a Brisk cluster by simply starting the instance.'
# Duplicate this line and change to add internal-only permissions
ec2-modify-image-attribute -l --region us-east-1 -a EMPLOYEEAWSID \$(ec2-describe-images --region us-east-1 | grep 'DataStax Brisk 1.0 Beta-1 Cluster - AMI $VERSION' | cut -f2)
""" && rm -rf ~/.bash_history && history -c
The above code will move your *.pem files to your /mnt directory, which is never imaged, and install EC2's AMI Tools. It will then image your instance and upload it to S3 based on which region you are in. It then echos the correct commands to run on your local machine to register the image and modify the permissions for internal-use-only images.
The code is an active project and may change at any point. Do let us know if there is anything you want to see, or submit a pull request for anything you feel will be beneficial. We're constantly trying to find ways to make this easy to follow and adaptable to all situations.
Feel free to leave any questions and comments in the support forums and we'll make sure to address all issues and concerns.
DataStax has many ways for you to advance in your career and knowledge.
You can take free classes, get certified, or read one of our many white papers.
register for classes
DBA's Guide to NoSQL