DataStax Support Forums » Topic: How to create a new AMI from DataStax AMI

DataStax Support Forums » Topic: How to create a new AMI from DataStax AMI http://www.datastax.com/support-forums/topic/how-to-create-a-new-ami-from-datastax-ami Software, Support, and Training for Apache Cassandra en-US Mon, 20 May 2013 11:47:20 +0000 http://bbpress.org/?v=1.0.3 <![CDATA[Search]]> q http://www.datastax.com/support-forums/search.php joaquin on "How to create a new AMI from DataStax AMI" http://www.datastax.com/support-forums/topic/how-to-create-a-new-ami-from-datastax-ami#post-162 Wed, 15 Jun 2011 18:42:12 +0000 joaquin 162@http://www.datastax.com/support-forums/ In that case, I would also opt for EBS volumes as well. For demo purposes, having a machine pre-loaded with data and sleeping is priceless. Depending on what you want to see with performance, you may even considering just leaving the data on the root drive at all times and just have your cluster big enough to have the sum total of all the free space handle the load. This way it would just be a matter of launching the clusters, the putting them to sleep before you launch them again. This would allow you to skip connecting and detaching the EBS volumes. But if you don't mind having a slightly longer presetup time, then using separate EBS drives is recommended though. blueplastic on "How to create a new AMI from DataStax AMI" http://www.datastax.com/support-forums/topic/how-to-create-a-new-ami-from-datastax-ami#post-161 Wed, 15 Jun 2011 18:15:54 +0000 blueplastic 161@http://www.datastax.com/support-forums/ Thanks for the updates Joaquin. Tamara's sinking her teeth into the AMI customization scripts. We wanted to use EBS backed instances mainly for the flexibility. We're creating a demo for clients that we don't want up and running 100% of the time. So, we prefer options & flexibility over performance. Regardless of whether we use instance-store of EBS-backed EC2 machines, we'd like to connect EBS volumes to them and store Cassandra's data files on EBS. That way, when we don't need to run the demo, we'll just detach the EBS volumes and terminate all the EC2 nodes. If we need to restart the demo, we'll start up x # of EC2 nodes and reconnect the EBS volumes to them, so the demo data will quickly be available. From our tests it looks like we can use either instance-store or EBS-backed EC2 machines... doesn't seem to matter. If you have any thoughts, please let us know. joaquin on "How to create a new AMI from DataStax AMI" http://www.datastax.com/support-forums/topic/how-to-create-a-new-ami-from-datastax-ami#post-157 Tue, 14 Jun 2011 18:19:04 +0000 joaquin 157@http://www.datastax.com/support-forums/ I just wanted to update this post with the link to the aforementioned blog post. Hopefully it's easier to follow then my quick cover-all-bases-in-10-minutes post above. You can read it all here: <a href="http://www.datastax.com/dev/blog/personalizing-your-own-brisk-ami" rel="nofollow">http://www.datastax.com/dev/blog/personalizing-your-own-brisk-ami</a> joaquin on "How to create a new AMI from DataStax AMI" http://www.datastax.com/support-forums/topic/how-to-create-a-new-ami-from-datastax-ami#post-154 Fri, 10 Jun 2011 21:00:52 +0000 joaquin 154@http://www.datastax.com/support-forums/ Hello, I'm glad to hear you want to develop the AMI further! All of the source code for this version of the AMI is currently here: <a href="https://github.com/riptano/TempBrisk" rel="nofollow">https://github.com/riptano/TempBrisk</a>. You can look at my projects as this will github project will be killed very soon in the near future and we will have a new address for it. Why would you want to use EBS instances over Instance Stores? Instances Stores have better performance and are more reliable without random lag times making them better for production use and benchmarking. If there is a not a strict reason, then you can just use this script to create your own S3 backed AMI: sudo mv *.pem /mnt sudo su cd curl <a href="http://s3.amazonaws.com/ec2-downloads/ec2-ami-tools.zip" rel="nofollow">http://s3.amazonaws.com/ec2-downloads/ec2-ami-tools.zip</a> > ec2-ami-tools.zip mkdir ec2 cp ec2-ami-tools.zip ec2 cd ec2 unzip ec2-ami-tools.zip ln -s ec2-ami-tools-* current sudo cat >> ~/.bashrc << EOF export EC2_AMITOOL_HOME=~/ec2/current export PATH=${PATH}:~/ec2/current/bin EOF source ~/.bashrc ec2-bundle-vol --help VERSION=$(head -1 /home/ubuntu/.presetup/VERSION) rm -rf ~/.bash_history && history -c && ec2-bundle-vol -p brisk_1.0_beta1-ami_$VERSION -d /mnt -k /mnt/pk-xxxxxxx.pem -c /mnt/cert-xxxxxxx.pem -u xxxxx -r x86_64 yes | ec2-upload-bundle -b datastax-brisk-cluster-ami-use -m /mnt/brisk_1.0_beta1-ami_$VERSION.manifest.xml -a xxxxxxx -s xxxxxxx --location US echo """ cd ~/.ec2/ ec2-register datastax-brisk-cluster-ami-use/brisk_1.0_beta1-ami_$VERSION.manifest.xml --region us-east-1 -n 'DataStax Brisk 1.0 Beta-1 Cluster - AMI $VERSION' -d 'Provides a way to automatically launch a Brisk cluster by simply starting the instance.' ec2-register datastax-brisk-cluster-ami-usw/brisk_1.0_beta1-ami_$VERSION.manifest.xml --region us-west-1 -n 'DataStax Brisk 1.0 Beta-1 Cluster - AMI $VERSION' -d 'Provides a way to automatically launch a Brisk cluster by simply starting the instance.' ec2-register datastax-brisk-cluster-ami-eu/brisk_1.0_beta1-ami_$VERSION.manifest.xml --region eu-west-1 -n 'DataStax Brisk 1.0 Beta-1 Cluster - AMI $VERSION' -d 'Provides a way to automatically launch a Brisk cluster by simply starting the instance.' ec2-register datastax-brisk-cluster-ami-aps/brisk_1.0_beta1-ami_$VERSION.manifest.xml --region ap-southeast-1 -n 'DataStax Brisk 1.0 Beta-1 Cluster - AMI $VERSION' -d 'Provides a way to automatically launch a Brisk cluster by simply starting the instance.' ec2-register datastax-brisk-cluster-ami-apn/brisk_1.0_beta1-ami_$VERSION.manifest.xml --region ap-northeast-1 -n 'DataStax Brisk 1.0 Beta-1 Cluster - AMI $VERSION' -d 'Provides a way to automatically launch a Brisk cluster by simply starting the instance.' ec2-modify-image-attribute -l --region us-east-1 -a EMPLOYEEAWSID \$(ec2-describe-images --region us-east-1 | grep 'DataStax Brisk 1.0 Beta-1 Cluster - AMI $VERSION' | cut -f2) """ && rm -rf ~/.bash_history && history -c That's the exact same code, minus a few changes, that I use to build the image. You would then copy the output of the echo into your computer where you have the ec2-tools installed, and register the AMI and open permissions. This entire step takes 20 minutes minimum, but 30 on a relaxed schedule. (There is a bit of waiting.) For both EBS backed instances and Instance-Stores you setup the image by: copy/pasting pre-setup.sh That will run the entire install process, then you would just create the backup via the normal EBS way. On boot, you will have all of your settings and configurations occur. However, if using the EBS volumes and this exact same source code, you're not guaranteed to be able to reconnect if the cluster was shut down then started back up. I was planning on doing a writeup soon on how exactly to branch off the code and since you've shown interest, I'll definitely try to make it sooner than later. Sorry if the code isn't as easy to follow, I've made quite a few changes to it soon, but really the bulk of your coding should occur in .configurebrisk and the .pre-setup folder. If you wish to start playing with the code feel free to start getting ideas. My next version will cleanup the .files and compile that all into a config file. If you wish, send me an email with your AWSID and I'll add you to this exact versions beta so you can see the new dev features. Sorry this was all done over the forum format, but I'll be sure to have something cleaner sooner, than later. Thanks again for showing interest in the AMI and let me know if you want any small changes or software added to the official project. I'd be more than happy in making this product better for everyone! blueplastic on "How to create a new AMI from DataStax AMI" http://www.datastax.com/support-forums/topic/how-to-create-a-new-ami-from-datastax-ami#post-153 Fri, 10 Jun 2011 18:07:09 +0000 blueplastic 153@http://www.datastax.com/support-forums/ Hi, We would like to use the current DataStax Brisk AMI as the foundation of a new custom AMI we're building. Basically, we want to take the DataStax AMI and add some other components to it, like Puppet. Then we want to use our new modified AMI to launch Cassandra clusters on demand. Since the DataStax AMI is an instance-store that's S3 backed, we can't simply highlight a running node (which we created from the DataStax AMI) and choose Instance Actions > Create Image (EBS AMI) in the AWS GUI. However, there are techniques to convert the instance-store to an EBS-backed AMI. One such technique involves connecting a temporary EBS volume to a running instance and using rsync to copy everything from the instance to the EBS volume. Then you take a snapshot of the EBS volume and register the snapshot as an AMI. Full directions: <a href="http://webkist.wordpress.com/2010/03/16/creating-an-amazon-ec2-ebs-ami-from-a-running-instance/" rel="nofollow">http://webkist.wordpress.com/2010/03/16/creating-an-amazon-ec2-ebs-ami-from-a-running-instance/</a> But I have some concerns about doing this. First, if we launch the DataStax AMI into a running instance, a one-node vanilla Cassandra cluster, let's say... then the scripts have already applied a lot of configuration parameters to this node. If we then take this configured one-node cluster and create a new EBS AMI out of it, can we still use it as a basis to launch a 10 node cluster in the future (5 cassandra/5 brisk)? I'm worried we'd be double-configuring the node, which could cause problems. Any thoughts/comments would be much appreciated! - Sameer