Version 6 (modified by 13 years ago) (diff) | ,
---|
DiCloud Tutorial at GEC12
This page summarizes the DiCloud tutorial that was presented at GEC 12. Slides can be found here.
Tutorial sections:
- Pre-requisites
- Install dependent software packages
- Setup DiCloud's MySQL database
- Create Amazon user credentials
- Register a user on Amazon EC2
- Setup environment variables
- Running the DiCloud server
- Using the DiCloud console
- Using the DiCloud web portal
- Running a Gush experiment using DiCloud and Orca
Pre-requisites
- Install VirtualBox: Download the latest VirtualBox software. If you already have VirtualBox installed on your machine, make sure it is version 4.08 or above.
Note to Ubuntu users: The version of VirtualBox distributed with Ubuntu 10.04 is old. Download the latest version from here.
- Download VirtualBox VM image: Download the VirtualBox VM image (gec12-tutorials-final.ova) needed for the DiCloud tutorial from here. A README with a copy of these instructions is also available at the same location.
- Install the gec12-tutorials-final.ova image: Start up VirtualBox, select File->Import Appliance..., and follow the instructions. Accept the default VM settings during the import. To run the virtual machine, go to the Oracle VM VirtualBox Manager window, select the virtual machine gec12-tutorial-final and click the green arrow labeled Start at the top of this window.
If the install was successful, you should see the logon screen for the Ubuntu OS. Passwords for logging into the VM will be handed out at the tutorial.
- Turn off the VM: Turn off the virtual machine by clicking on the power button icon on the bottom right of the screen.
- Note that this release of tutorial software is intended for use at the GEC12 tutorial only. In particular, these directions are for Ubuntu 10.04.3 LTS (Lucid Lynx). To find out your version of Ubuntu, open a terminal and run:
$ cat /etc/issue Ubuntu 10.04.3 LTS \n \l
While the directions may work for other distributions, your mileage may vary.
Install dependent software packages
To install dependent software you need root access. The directions assume the user is able to use sudo to run commands as root.
$ sudo add-apt-repository "deb http://archive.canonical.com/ lucid partner" $ sudo apt-get update $ sudo apt-get install sun-java6-jdk $ sudo apt-get install maven2 $ sudo apt-get install apache2 $ sudo apt-get install libapache2-mod-php5 $ sudo apt-get install php5-mysql $ sudo a2enmod php5 $ sudo apt-get install mysql-server
MySQL server will ask you to set a root password. Remember the password.
Setup DiCloud's !MySQL Server
First, log into !MySQL and create the dicloud database by running the following commands.
$ mysql -u root -p $ mysql> create database dicloud; $ mysql> quit;
Next load the DiCloud schema, create an admin user, and give the user database rights
$ cd ~/Tutorials/DiCloud $ mysql -u root -p < dicloud.schema $ mysql -u root -p $ mysql> INSERT INTO users(id, username, password) VALUES (1, 'admin', SHA1('geni')); $ mysql> INSERT INTO admin_rights VALUES (1, true, true, true, true); $ mysql> create user vise; $ mysql> SET PASSWORD FOR 'vise' = PASSWORD(''); $ mysql> GRANT ALL ON dicloud.* TO 'vise'@'localhost';
Create Amazon user credentials
Note that tutorial participants should not perform this step.
Amazon EC2 requires that each user create and upload an X.509 certificate to Amazon, and use the corresponding private key to access the web services functions. We create the certificate and private key below. We copy versions of these certificates and change the ownership to www-data to allow the DiCloud web portal to read them.
$ cd ~/Tutorials/DiCloud/ec2/keys $ openssl genrsa 1024 > ./geni_pk.pem $ openssl req -new -x509 -nodes -sha1 -key ./geni_pk.pem > ./geni_cert.pem $ ssh-keygen geni-keypair.pem
Register new user on Amazon EC2
Note that tutorial participants will be able to use a DiCloud account, and do not need to create there own account. Tutorial participants will also use a pre-existing DiCloud user, and do not need to perform the steps listed in the rest of this section.
Create an account with Amazon web services by going here and clicking on "Create an AWS Account" in the upper-right portion of the page. Amazon requires that account link to a valid credit card for payment. Once you've created an account go back to here and click on "Sign in to the AWS Management Console" in the upper-right portion of the page. Once logged in, click on the AWS IAM tab (second from the right). Click on "Users" on the left panel, and then click "Create New Users". Create a user named "geni" (or any other name you prefer). Next click on the geni user, and go to the "Security Credentials" tab at the bottom of the page. Here you need to generate Access Keys and download the credentials.csv file that has the Access Key and the Secret Access Key. Also, upload the signing certificate we created at ~/Tutorials/DiCloud/ec2/keys/geni_cert.pem. Finally, click on the "Amazon EC2" tab (third from the left). Then click on the "Key Pairs" link at the bottom of the left panel. Now click "Import Key Pair". Give the new keypair the name "geni-keypair" (or any other name you prefer) and copy the contents of ~/Tutorials/DiCloud/ec2/keys/geni-keypair.pem in the correct place. Copy the credentials file credentials.csv to ~/Tutorials/DiCloud/ec2/keys/ and create the file ~/Tutorials/DiCloud/ec2/keys/geni_aws_credentials.txt, which should look as follows:
geni@geni-tutorial:~/Tutorials/DiCloud/ec2/keys$ cat geni_aws_credentials.txt AWSAccessKeyId=<Your Access Key ID here> AWSSecretKey=<Your Secret Key Here>
Configuring Amazon for Tutorial Participants
Tutorial participants should download their certificate, private key, key pair, credential.csv, and geni_aws_credentials.txt files by performing the following commands.
$ cd ~/Tutorials/DiCloud/ec2/keys $ wget http://geni.cs.umass.edu/geni_cert.pem $ wget http://geni.cs.umass.edu/geni_pk.pem $ wget http://geni.cs.umass.edu/geni-keypair.pem $ wget http://geni.cs.umass.edu/geni-keypair.pub $ wget http://geni.cs.umass.edu/geni_aws_credentials.txt $ wget http://geni.cs.umass.edu/credentials.csv
Enabling Access to Keys via DiCloud's Web Portal
We also create copies of the certificate and private key and change their ownership to www-data so the web portal can access them.
$ cp ./geni_pk.pem ./geni_pk_web.pem $ sudo chown www-data geni_pk_web.pem $ sudo chgrp www-data geni_pk_web.pem $ cp ./geni_cert.pem ./geni_cert_web.pem $ sudo chown www-data geni_cert_web.pem $ sudo chgrp www-data geni_cert_web.pem
Setup Environment Variables
Step 5: Setup environment variables
cd ~/Tutorials/DiCloud nano environment [ Insert correct from geni_aws_credentials.txt AWS_ACCESS_KEY_ID= AWS_SECRET_ACCESS_KEY= ] cat environment >> ~/.bashrc
Add Access Key Id and Secret Access Key to .s3cfg file. Copy .s3cfg file to ~geni
cp ~/Tutorials/DiCloud/.s3cfg ~/.s3cfg
Run the DiCloud server
cd ~/Tutorials/DiCloud/dicloud/bin ./dicloud_server.sh
Step 7: Using the DiCloud console
Open a new terminal. We run ntpdate to make sure the time is accurate. Amazon will fail calls from machines with inaccurate local time.
sudo ntpdate ntp.ubuntu.com cd ~/Tutorials/DiCloud/dicloud/bin ./dicloud_console.sh dicloud - not connected>connect localhost 1099 Connecting to //localhost:1099/dicloud/server dicloud - //localhost:1099>ec2 start /home/geni/Tutorials/DiCloud/ec2/keys/geni_cert$ dicloud - //localhost:1099>help Commands available for the main menu are: add credit amount_of_money Add credit to the AWS account billing history history_size csv|html Get the AWS account activity history in CSV or HTML format (a history of 0 retrie$ connect DiCloudServerHostName RmiPortNumber Connects to the DiCloud server ebs attach cert.pem pk.pem ec2_region ebs_volume_id ec2_instance_id device_name Attach an EBS volume (ebs_volume_id) to the running instance (ec2_instance_id) as$ ebs create cert.pem pk.pem size_in_gb ec2_region ec2_availability_zone Create an EBS volume ebs delete cert.pem pk.pem volume_id ec2_region Delete an EBS volume ebs detach cert.pem pk.pem ec2_region ebs_volume_id Detach EBS volume ebs_volume_id from its EC2 instance ebs probe cert.pem pk.pem volume_id ec2_region ascii|html Probe an EBS volume and get current status information ec2 probe cert.pem pk.pem instance_id ec2_region html|ascii Check the status of a running EC2 instance ec2 start cert.pem pk.pem ami_id instance_nb instance_type ec2_region ec2_keypair_na$ Start a new EC2 instance ec2 stop cert.pem pk.pem instance_id ec2_region ec2_keypair_name Stop an EC2 instance get balance Get the AWS account balance help Print this help message history [<command index>] Display history of commands for the console. quit Quit the console s3 create bucket_name region Create an S3 bucket s3 delete bucket_name region Delete an S3 bucket s3 get s3_bucket_name object_name file_path Get the content of an object named object_name into the specified file from the S$ s3 probe bucket_name region Get information about an S3 bucket s3 put s3_bucket_name object_name file_path Put the content of the specified file in an object named object_name in the S3 bu$ shutdown Shutdown the DiCloud server the console is currently connected to. dicloud - //localhost:1099>get balance Current balance is: $99.53222835338767 dicloud - //localhost:1099>ec2 probe /home/geni/Tutorials/DiCloud/ec2/keys/geni_cert.pem /home/geni/Tutorials/DiCloud/ec2/keys/geni_pk.pem i-b87b74d8 us-east-1 ascii Instance Id: i-b87b74d8 AMI Id: ami-78f21911 Public name: ec2-75-101-245-135.compute-1.amazonaws.com Private name: ip-10-244-46-161.ec2.internal Public IP: 75.101.245.135 Private IP: 10.244.46.161 Instance State: running Instance Type: m1.small Key pair: geni-keypair Start time: 2011-10-17T17:31:41+0000 Availability zone: us-east-1b dicloud - //localhost:1099>quit
You can also script the DiCloud console. An example is below.
$ echo "connect localhost 1099 get balance quit" > ./getbalance.txt $ ./dicloud_console_script.sh -silent < ./getbalance.txt
Example use of S3
dicloud - //localhost:1099>s3 create gec12-tutorial-1 us dicloud - //localhost:1099>s3 put gec12-tutorial-1 test /home/geni/Tutorials/DiCloud/dicloud/bin/dicloud.properties Uploading file /home/geni/Tutorials/DiCloud/dicloud/bin/dicloud.properties as object test in bucket gec12-tutorial-1 dicloud - //localhost:1099>s3 get gec12-tutorial-1 test /home/geni/Tutorials/DiCloud/test Downloading file /home/geni/Tutorials/DiCloud/test from object test in bucket gec12-tutorial-1dicloud - //localhost:1099> dicloud - //localhost:1099>s3 delete gec12-tutorial-1 us
Example use of EBS
dicloud - //localhost:1099>ebs create /home/geni/Tutorials/DiCloud/ec2/keys/geni_cert.pem /home/geni/Tutorials/DiCloud/ec2/keys/geni_pk.pem 1 us-east us-east-1a dicloud - //localhost:1099>ebs delete /home/geni/Tutorials/DiCloud/ec2/keys/geni_cert.pem /home/geni/Tutorials/DiCloud/ec2/keys/geni_pk.pem <volume_id> us-east
Step 8: Using the DiCloud web portal
Go to http://localhost/vise/dicloud.
Step 9: Running a Gush experiment using DiCloud and/or Orca resources
Direction adapted from http://groups.geni.net/geni/wiki/GEC11VMSetup under installing Gush.
apt-get install keychain mkdir /usr/local/geni/gush cd /usr/local/geni/gush wget http://gush.cs.williams.edu/gush-bin-32bit.tgz cd ~ bash cp ~/Tutorials/DiCloud/ec2/keys/geni-keypair.pem ~/.ssh/id_rsa cp ~/Tutorials/DiCloud/ec2/keys/geni-keypair.pub ~/.ssh/id_rsa.pub keychain ~/.ssh/id_rsa echo 'StrictHostKeyChecking no' >> ~/.ssh/config mkdir ~/Tutorials/DiCloud/gush cd ~/Tutorials/DiCloud/gush cp /usr/local/geni/gush/gush-bin-32bit.tgz . tar xzvf gush-bin-32bit.tgz
Verify gush works
./gush -P 15000
Modify directory.xml and omni_config
Attachments (3)
-
dicloud-tutorial.tgz (70.3 MB) - added by 13 years ago.
Source files for tutorial.
-
GEC12tutorial.txt (3.1 KB) - added by 13 years ago.
Commands for tutorial in a text file.
-
GEC12tutorial.pptx (4.4 MB) - added by 13 years ago.
Slides for GEC12 DiCloud tutorial.