wiki:OnTimeMeasureTutorial

Version 24 (modified by yxu@osc.edu, 13 years ago) (diff)

--

Tutorial on OnTimeMeasure Capabilities for Experimenters

Yingxiao Xu, Prasad Calyam
January 31, 2012

Software License: 'GENI Project License (GPL)'

  1. Tutorial on OnTimeMeasure Capabilities for Experimenters
    1. 1. Pre-work
      1. 1.1 Get OnTimeMeasure and Emulab User Accounts
      2. 1.2 Upload Emulab SSH Keys
      3. 1.3 Generate SSL Certificate
    2. 2. Install OnTimeMeasure with Flack
      1. 2.1 Login into Flack
      2. 2.2 Create Slice using Flack
      3. 2.3 Create Sliver using Flack
        1. 2.3.1 Create Sliver by Importing RSpec file
        2. 2.3.2 Create Sliver from Scratch
          1. 2.3.2.1 Set Root Beacon
          2. 2.3.2.2 Set Node Beacons
        3. 2.3.2.3 Submit
      4. 2.4 Confirmation
    3. 3. Configure, Control, Query OnTimeMeasure using Command-line
      1. 3.1 Config IP address and password
      2. 3.2 Check Measurement Service Status
      3. 3.3 Manage Measurement Tasks
      4. 3.4 Measurement Service Control
      5. 3.5 Check Measurement Service Status
      6. 3.6 Query Measurement Results
    4. 4. Configure, Control, Query OnTimeMeasure using Web-portal
      1. 4.1 Researcher Web-portal Login
      2. 4.2 Configuration
      3. 4.3 Control
      4. 4.4 Query
    5. 5 Integrate Custom Metric of Experiment with OnTimeMeasure
      1. 5.1 Compose custom file for custom metric specifications and a parser …
      2. 5.2 Add custom metric specifications file and parser file into …
      3. 5.3 Configure Tasks of Custom Metric Measurement
      4. 5.4 Manage Measurement Tasks with Custom Metric Measurement and …
    6. 6. View OnTimeMeasure Measurement Results in INSTOOLS web-portal
      1. 6.1 Install INSTOOLS
      2. 6.2 Create Page to display measurement result of OnTimeMeasure
      3. 6.3 View Results
    7. 7. Install and Use OnTimeMeasure with Gush
      1. 7.1 Install and Configure GUSH
        1. 7.1.1 Install Gush
        2. 7.1.2 Create Clear Text Certificate
        3. 7.1.3 Configure Gush
      2. 7.2 Create a RSpec File
      3. 7.3 Create Slice and Sliver
      4. 7.4 Edit directory.xml file of Gush
      5. 7.5 Start Gush
      6. 7.6 Use Gush to Configure OnTimeMeasure
      7. 7.7 Use Gush to Control and Query OnTimeMeasure
      8. 7.8 Further Operation
    8. 8. Use of OMNI to add PlanetLab Nodes for OnTimeMeasure slice …
      1. 8.1 Install OMNI
      2. 8.2 Create a RSpec File
      3. 8.3 Create PlanetLab Sliver
      4. 8.4 Get Node Information
      5. 8.5 Install NodeBeacon
      6. 8.6 Further Operation
    9. 9. General Information

This tutorial provides step-by-step instructions for GENI experimenters to learn about the main Instrumentation and Measurement (I&M) capabilities of OnTimeMeasure software/service that can be used in an integrated manner with Flack/ProtoGENI, INSTOOLS, Gush, OMNI and PlanetLab.

The tutorial organization is shown in below Figure.

Section 1 describes the pre-work for the tutorial. In Section 2, we will create an experiment slice with slivers and will install OnTimeMeasure in ProtoGENI/Emulab using the Flack tool. Next in Sections 3 and 4, we will configure, control and query the measurement results using OnTimeMeasure command-line tools and web-portal, respectively. In Section 5, we will integrate a custom metric of an experiment into the OnTimeMeasure framework. In Section 6, we will query measurement results so that they can be displayed within the INSTOOLS web-portal. In Section 7, we will use GUSH to create new a slice with nodes, and configure it to run measurements across the two slices. Finally in Section 8, we use the OMNI tool to add PlanetLab nodes to the first slice.

The general duration of the tutorial is expected to take about 1 hour.

1. Pre-work

1.1 Get OnTimeMeasure and Emulab User Accounts

Please register user accounts in the OnTimeMeasure Researcher Web-portal and Emulab Network Testbed

For more details regarding getting an Emulab User Account, please see - https://www.protogeni.net/trac/protogeni/wiki/Tutorial#GetanEmulabAccount

1.2 Upload Emulab SSH Keys

SSH login to users.emulab.net with your Emulab user account and password. Check whether the files id_rsa and id_rsa.pub exist in your .ssh folder. If they do not exist, create a pair using ssh-keygen tool. You can download the files from users.emulab.net using SCP onto your PC.

Next, upload your public key (id_rsa.pub)to Emulab. For this, first visit http://www.emulab.net/ and login, then click "Edit SSH Keys" link as shown in the following UI.

You will get the following UI.

Browse to your public key file, enter password of your Emulab user account. Click the "Add New Keys" button to upload your public key.

For more details on uploading SSH Keys in Emulab, please see - https://www.protogeni.net/trac/protogeni/wiki/Tutorial#UploadingSSHKeys

1.3 Generate SSL Certificate

If you do not have an SSL certificate in Emulab, you can generate one by clicking "Generate SSL Cert" as shown in the UI below.

You will get the following UI.

The PassPhrase you set here will be used to login into Flack

The Emulab Password is the password associated with your Emulab user account.

For more details on generating SSL certificates in Emulab, please see - http://www.protogeni.net/trac/protogeni/wiki/Tutorial#SSLCertificate

If you have problems creating SSH Keys and Certificates in Emulab, you can refer to some general instructions provided at - http://groups.geni.net/geni/wiki/otm-ssh-cert-emulab.

2. Install OnTimeMeasure with Flack

2.1 Login into Flack

Visit - http://protogeni.net/flack

Click "Log in" button. Click "Allow" if the following dialog appears.

In the UI below, click "Select authority" and choose "emulab.net”

Click "Download" button. Your credentials will be downloaded and displayed.

If you get prompt for "You must be logged in on the authority's website", click "yes" to bring up a login page of www.emulab.net in a new tab, then login to the website with your Emulab password.

Return back to Flack tab and press "Download" again. You credentials will be displayed. Enter the PassPhrase you set earlier.

Click "OK" button. If the following dialog appears, click the "Allow" button.

You will get the UI to create slice.

2.2 Create Slice using Flack

Create new slice by entering an unique slice name (i.e. "OTCDemo") to replace the text "Create slice..." in the text-box.

Press enter/return to create the slice.

A slice window will appear with the new slice name.

2.3 Create Sliver using Flack

To create sliver, you can follow either "Create Sliver by Importing RSpec file" in Section 2.3.1 below or "Create Sliver from Scratch" in Section 2.3.2 below.

In either case, at least 3 nodes will be created. One is called root beacon, which creates database tables and configuration files, and generates measurement schedules for node beacons. Root beacon also collects data and provides dashboard visualization. Others are called node beacon, which installs tools that measure network and runs measurements based on a schedule.

2.3.1 Create Sliver by Importing RSpec file

To create sliver by importing RSpec file, download - http://ontime.oar.net/INSTALL/OTCDemo.rspec

Save it to your local file system.

A sample OTCDemo.rspec file is:

<rspec type="request" generated_by="Flack" generated="2012-02-16T18:42:42Z" xsi:schemaLocation="http://www.geni.net/resources/rspec/3 http://www.geni.net/resources/rspec/3/request.xsd" xmlns:flack="http://www.protogeni.net/resources/rspec/ext/flack/1" xmlns:client="http://www.protogeni.net/resources/rspec/ext/client/1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.geni.net/resources/rspec/3">
  <node client_id="ROOT" component_manager_id="urn:publicid:IDN+emulab.net+authority+cm" exclusive="true">
    <sliver_type name="raw-pc">
      <disk_image name="urn:publicid:IDN+emulab.net+image+emulab-ops//FEDORA8-64-OVZ-UPD"/>
    </sliver_type>
    <services>
      <execute command="sudo sh /tmp/ontime_software_install.sh otm_slice_password" shell="sh"/>
      <install install_path="/tmp" url="http://ontime.oar.net/INSTALL/OTCRootInstall.tgz" file_type="tar.gz"/>
    </services>
    <flack:node_info x="138" y="68" unbound="true"/>
  </node>
  <node client_id="NODE1" component_manager_id="urn:publicid:IDN+emulab.net+authority+cm" exclusive="true">
    <sliver_type name="raw-pc">
      <disk_image name="urn:publicid:IDN+emulab.net+image+emulab-ops//UBUNTU10-STD"/>
    </sliver_type>
    <services>
      <execute command="sudo sh /tmp/ontime_software_install.sh" shell="sh"/>
      <install install_path="/tmp" url="http://ontime.oar.net/INSTALL/OTCNodeInstall.tgz" file_type="tar.gz"/>
    </services>
    <flack:node_info x="34" y="162" unbound="true"/>
  </node>
  <node client_id="NODE2" component_manager_id="urn:publicid:IDN+emulab.net+authority+cm" exclusive="true">
    <sliver_type name="raw-pc">
      <disk_image name="urn:publicid:IDN+emulab.net+image+emulab-ops//UBUNTU10-STD"/>
    </sliver_type>
    <services>
      <execute command="sudo sh /tmp/ontime_software_install.sh" shell="sh"/>
      <install install_path="/tmp" url="http://ontime.oar.net/INSTALL/OTCNodeInstall.tgz" file_type="tar.gz"/>
    </services>
    <flack:node_info x="238" y="162" unbound="true"/>
  </node>
</rspec>

The RSpec file specifies three nodes.

One node has name ROOT. It will automatically download http://ontime.oar.net/INSTALL/OTCRootInstall.tgz into /tmp folder and run Root Beacon installation command "sudo sh /tmp/ontime_software_install.sh otm_slice_password".

(Note: You can replace "otm_slice_password" with your own password. The password will be needed to use OnTimeMeasure later in this tutorial. (i.e., you should use the same password for config.yaml file in section 3.1 and database password in section 4.2))

The other two nodes have name NODE1 and NODE2 respectively. They will automatically download http://ontime.oar.net/INSTALL/OTCNodeInstall.tgz into /tmp folder and run Node Beacon installation command "sudo sh /tmp/ontime_software_install.sh".

Click import button, select "import from file"

Choose the file OTCDemo.rspec you downloaded.

The following UI will appear.

Click "Submit" button. Click "Yes" when the following dialog appears.

You can skip the "2.3.2 Create Sliver from Scratch" if you use the above import steps, and jump to the 2.4 Confirmation section

2.3.2 Create Sliver from Scratch

If you do not choose to use the import option, slivers can also be added to the slice by dragging from the aggregate where you want to create the sliver.

NOTE1: To install OnTimeMeasure, make sure you have at least three nodes in your slice.

NOTE2: You can choose nodes in utahemulab.cm if you can't find the needed disk image in section 2.3.2.1 and 2.3.2.2

2.3.2.1 Set Root Beacon

Choose one node as Root Beacon by clicking the "I" icon to edit the node.

A UI will appear to set the node. Set the name(i.e. "ROOT"). Choose "Update to FEDORA8-64-OVZ-STD " for Disk image

Click "Add Install Service" button.

In text field of Install, input

http://ontime.oar.net/INSTALL/OTCRootInstall.tgz

In text field of in, input

/tmp 

Click "Add Exceute Service" button,

In text field of Execute, input

sudo sh /tmp/ontime_software_install.sh otm_slice_password

Note You can replace "otm_slice_password" with your own password. The password will be needed to use OnTimeMeasure later in this tutorial. (i.e., you should use the same password for config.yaml file in section 3.1 and database password in section 4.2)

Click "Apply" button.

2.3.2.2 Set Node Beacons

Choose other nodes as Node Beacons.

Click the "I" icon to edit each node.

Each time a UI appear to set the node, set the name (i.e., "NODE1", you should choose different name for each node) and Choose "Ubuntu 10 32-bit" for Disk image

Click "Add Install Service" button.

In text field of Install, input

http://ontime.oar.net/INSTALL/OTCNodeInstall.tgz

In text field of in, input

/tmp

Click "Add Exceute Service" button,

In text field of Execute, input

sudo sh /tmp/ontime_software_install.sh

Click "Apply" button.

Repeat the above steps till all the nodes are configured.

2.3.2.3 Submit

Click "Submit" button. Click "Yes" when the following dialog appears.

2.4 Confirmation

OnTimeMeasure will begin to install when nodes turn green. You will see the domain name of each nodes. You can use Ping or NSlookup tools in any machine to get IP address of each node.

In this example, the ROOT (Root Beacon) is pc355.emulab.net (155.98.39.155), the NODE1 (Node Beacon 1) is pc337.emulab.net (155.98.39.137). The NODE2 is pc262.emulab.net (155.98.39.62).

Please wait about 5 minutes for the installation of the Root Beacon. You can also jump to 3. Configure, Control, Query OnTimeMeasure using Command-line section and confirm the Root Beacon installation later.

Confirm the installation of Root Beacon (NOTE: use port number 17291 in the web-browser URL, i.e., http://pc355.emulab.net:17291)

Confirm the installation of Node Beacons (NOTE: use port number 17291 in the web-browser URL of each Node Beacon, i.e., http://pc337.emulab.net:17291 and http://pc262.emulab.net:17291).

3. Configure, Control, Query OnTimeMeasure using Command-line

For overview and details of the OnTimeControl tool of OnTimeMeasure, please see - http://groups.geni.net/geni/wiki/OTM-Control

3.1 Config IP address and password

Use SSH to Login to the Root Beacon.

Enter OnTimeControl folder by typing the following command

cd /opt/OnTimeMeasure/OnTimeControl
cp config_example.yaml config.yaml

Change config.yaml with the IP addresses of your Root Beacon and Node Beacons.

A sample configuration is:

root_beacon: 155.98.39.155
db_user: collector
db_pwd: otm_slice_password

nodes:
- name: node1
  IP: 155.98.39.137

- name: node2
  IP: 155.98.39.62

3.2 Check Measurement Service Status

Run command "service_status.py" to verify whether all the components of the measurement service are functioning correctly.

python service_status.py

You will find the status of collector is "Initializing...". It will changed to "OK" when you later add measurement tasks and start the measurement service.

3.3 Manage Measurement Tasks

"task_manage.py" manages the measurement tasks (i.e., requests). Edit measurement.yaml to configure tasks. For details, please reference OnTime Control User Manual

NOTE: You may need to modify the node names if you are not using node1, node2, etc. as node names

Run the "task_manage.py" with parameter "-c <measurement_config_file.yaml>".

python task_manage.py -c measurement.yaml

3.4 Measurement Service Control

"service_control.py" controls the measurement service. Each time you change the measurement.yaml file, you need to restart the measurement service.

python service_control.py restart

3.5 Check Measurement Service Status

Run command "service_status.py" to verify whether all the components of the measurement service are functioning correctly.

python service_status.py

You will find the status of collector is changed to "OK".

3.6 Query Measurement Results

You may need to wait a while for measurement data to start appearing in the Root Beacon database for query. The wait time depends on how you configure the measurement schedules in the measurement.yaml file. You will get some measurement results within one minutes, but some measurement task may need half hour to be scheduled to avoid the measurement conflict.

You can access Root Beacon through port 17280 in your web-browser (i.e., http://pc355.emulab.net:17280/) to check all the measurement data generated already in measurement dashboard.

To see a video on how to navigate Graphite measurement dashboard in OnTimeMeasure, please see - http://ontime.oar.net/demo/dashboard.htm; More details on the Graphite tool can be found at - http://graphite.wikidot.com

To query using OnTimeControl command-line tool, enter the following command (you may need to change the start-and-end dates in the query option).

python query.py -f "2012-01-24 18:00:00" -t "2012-01-27 20:00:00" -s node2 -d node1 -m Throughput -r TimeSeries

Measurement results are automatically downloaded into your local folder.

In this example, two file 20120126164952_IperfTcp_Throughput.txt and 20120126164952_IperfTcp_Throughput.png were created in "results" subfolder to display the result in text and graph plots. The result can also be accessed from browser with the URL listed in the output.

4. Configure, Control, Query OnTimeMeasure using Web-portal

You can also use the web-portal at http://ontime.oar.net to setup resources (i.e., provide RSpec information) and control Node/Root Beacons within slice in an interactive manner.

For videos about how to navigate the Researcher Web-portal of OnTimeMeasure, please see - http://ontime.oar.net/demo

4.1 Researcher Web-portal Login

4.2 Configuration

Click "Setup Resources" button. NOTE: The database password you need to enter is same as the otm_slice_password.

Click the "Specify Request" button, you will be directed to a "Measurement Request Specification" page. The measurement tasks you set using OnTimeControl before will be displayed here automatically. You can modify it if necessary.

Click "Review Request" button, you will be directed to "Measurement Request Submission" page.

Click "Submit Request" button.

4.3 Control

Next, you will be directed to the "Service Control" page. If you have modified measurement tasks in previous page, you need to click "Stop" button to stop measurement collection in your slice. Next, click "Start" button to restart measurement collection in your slice.

4.4 Query

Click the "Query Data" button. You can select query options.

Click the "Submit Query" button, you will see the query result.

Click the "View Result" link, you will be prompt to download query result.

5 Integrate Custom Metric of Experiment with OnTimeMeasure

For overview and details of the custom metric integration feature of OnTimeMeasure, please see - http://groups.geni.net/geni/wiki/OTM-CustomMetric

For videos on case studies to integrate custom metrics within an experiment, please see links for the OSU and Purdue University case studies at - http://ontime.oar.net/demo

5.1 Compose custom file for custom metric specifications and a parser file for the custom metric

For demo, you can download a sample from http://ontime.oar.net/INSTALL/metric/CPU.tgz by typing the following command

cd /opt/OnTimeMeasure/OnTimeControl
wget http://ontime.oar.net/INSTALL/metric/CPU.tgz
tar xzf CPU.tgz

You will get custom metric specification file CPUSpecs.yaml and parser file CPUParser.pm

The sample CPUSpecs.yaml file:

Metrics:
  Mycpu:
     description: CPU Load
     tool: Mycpuinfo
     metrictype: host
Tools:
  Mycpuinfo:
    command:  top -b -n 3 | grep Cpu
    runat: <SRC>
    dbtable: 
          - field: [MycpuinfoDt,HostAddress,mycpu]
            type:  [timestamp, varchar(25), double]
    graphite: 
         - send: [mycpu,HostAddress.mycpu]
    time: 10
    duration: 2
    conflict: []

5.2 Add custom metric specifications file and parser file into OnTimeMeasure framework

Run command "add_metric.py" to add the custom metric specifications file and parser file into OnTimeMeasure framework

python add_metric.py  -c CPUSpecs.yaml -p CPUParser.pm

5.3 Configure Tasks of Custom Metric Measurement

Change the measurement.yaml file

- source: node1
  metric: [MemFree]
- source: node2
  metric: [MemFree]

to

- source: node1
  metric: [MemFree,Mycpu]
- source: node2
  metric: [MemFree,Mycpu]

NOTE: Mycpu is the metric name configured in CPUSpecs.yaml

5.4 Manage Measurement Tasks with Custom Metric Measurement and Restart the Measurement

Run command "task_manage.py" to manage the measurement tasks with custom metric (i.e., requests), which are configured in the measurement.yaml file.

python task_manage.py -c measurement.yaml
python service_control.py restart

6. View OnTimeMeasure Measurement Results in INSTOOLS web-portal

For overview and details of integration the OnTimeMeasure with INSTOOLS , please see - http://groups.geni.net/geni/wiki/OTM-InsToolsDemo

6.1 Install INSTOOLS

In Flack, click Plugins tab, you will get INSTOOLS UI as follows.

Click "Instrumentize" button .

You will be prompted to visit the INSTOOLS portal.

Click "Yes"

Click on the MC icon,and click "Main Page" button. You will be directed to the "Live View" page for the measurement controller of INSTOOLS.

6.2 Create Page to display measurement result of OnTimeMeasure

Click the Create Content/Page link. Enter the title and body.

In the body, enter the following text (NOTE: You need to change the pc355.emulab.net to host name of your Root Beacon ):

<img src= "http://pc355.emulab.net:17280/render?height=300&width=500&&from=-3d&until=now&target=Graphite.node1_node2.RoundTripDelay&uniq='+new Date().getTime();" align ="middle" alt = "round-trip delay graph"/>
<!-- This is for link named node1_node2 in graphite and returns Round-trip Delay for last 3 days, for link node2 to node1, change "node1_node2" -->

Click "Menu settings" button, enter Menu link title (e.g., Round-Trip Delay). Choose a Parent item (i.e., Graph under Node1)

Scroll down the page, click "Input format" link, choose "Full HTML".

Click the "Save" button.

6.3 View Results

You will get the measurement results by clicking the menu link title under the parent item you set.

7. Install and Use OnTimeMeasure with Gush

For overview and details of the Gush integration within OnTimeMeasure, please see - http://groups.geni.net/geni/wiki/OTM-Gush

Till now, we have slice OTCDemo with Root Beacon (155.98.39.155), Node Beacon 1 (155.98.39.137) and Node Beacon 2 (155.98.39.62) created by Flack. In this section, we will use Gush to create a new slice OTMGushDemo with Node Beacon 3 and will configure it to run measurements to Node Beacons 1 and 2 in the OTCDemo slice.

7.1 Install and Configure GUSH

7.1.1 Install Gush

For details on Gush installation procedure, see - http://gush.cs.williams.edu/trac/gush/wiki/UsingGush

Follow the below steps to install Gush in Ubuntu 10.04 Desktop.

sudo  apt-get update
sudo apt-get install  -y  openssh-server subversion keychain g++-4.1 autoconf2.59 makedepend emacs23
sudo ln -s /usr/bin/g++-4.1 /usr/bin/g++
sudo apt-get install  -y  curl libcurl3 libcurl4-openssl-dev libboost-dev xutils-dev flex bison libreadline-dev
sudo apt-get install -y python-m2crypto python-dateutil python-pyopenssl libxmlsec1 xmlsec1 libxmlsec1-openssl libxmlsec1-dev
wget http://gush.cs.williams.edu/xmlrpc-c-1.16.34.tgz
tar xzf xmlrpc-c-1.16.34.tgz
cd xmlrpc-c-1.16.34
./configure
make
sudo make install
 
cd ..
wget  http://gush.cs.williams.edu/gush-bin-32bit.tgz
tar xzf gush-bin-32bit.tgz

7.1.2 Create Clear Text Certificate

In .ssl folder, find your certificate with passphrase. i.e., file encrypted.pem Generate clear text certificate i.e., file keyout.pem by invoking

openssl rsa -in encrypted.pem -out keyout.pem
openssl x509 -in encrypted.pem >> keyout.pem

You may be prompt to enter pass phrase you set in section 1.4

The file keyout.pem will be used in the configuration of omni_config file

7.1.3 Configure Gush

Modify omni_config file below.

A sample config file

[omni]
default_cf = pg
users = xyx

[pg]
type=pg
ch = https://www.emulab.net:443/protogeni/xmlrpc/ch
sa = https://www.emulab.net:443/protogeni/xmlrpc/sa
cert = ~/.ssl/keyout.pem
key = ~/.ssl/keyout.pem


[xyx]
urn = urn:publicid:IDN+emulab.net+user+xyx
keys=~/.ssh/id_rsa.pub

copy it to .gcf folder.

Modify gush.prefs as follows:

<preferences>
    <pref key="ClientPath">/opt/OnTimeMeasure/OnTimeControl/</pref>
    <pref key="ClientPrefix">./</pref>
    <pref key="UsePlanetLab">true</pref>
</preferences>

7.2 Create a RSpec File

A sample file OTMGushDemo.rspec is given below. You can use it directly.

<?xml version="1.0" encoding="UTF-8"?>
<rspec type="request" xsi:schemaLocation="http://www.protogeni.net/resources/rspec/2
                                          http://www.protogeni.net/resources/rspec/2/request.xsd"
       xmlns:flack="http://www.protogeni.net/resources/rspec/ext/flack/1"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns="http://www.protogeni.net/resources/rspec/2">
  
  <node client_id="node3" component_manager_id="urn:publicid:IDN+emulab.net+authority+cm" exclusive="true">
    <sliver_type name="raw-pc">
    <disk_image name="urn:publicid:IDN+emulab.net+image+emulab-ops//UBUNTU10-STD" />
    </sliver_type>
    <services>
      <install url="http://ontime.oar.net/INSTALL/OTCNodeInstall.tgz" install_path="/tmp"/>
      <execute shell="sh" command="sudo sh /tmp/ontime_software_install.sh"/>
   </services>
  </node>
 
 
</rspec>

7.3 Create Slice and Sliver

In helper-scripts folder under Gush installation folder, invoke the following command to create a new slice and sliver

./handle-geni.py -f pg -a https://www.emulab.net/protogeni/xmlrpc/am createslice OTMGushDemo

./handle-geni.py -n -f pg -a https://www.emulab.net/protogeni/xmlrpc/am createsliver OTMGushDemo OTMGushDemo.rspec

The output

root:~/gush2/helper-scripts> ./handle-geni.py -f pg -a https://www.emulab.net/protogeni/xmlrpc/am createslice OTMGushDemo
INFO:omni:Loading config file /users/xyx/.gcf/omni_config
INFO:omni:Using control framework pg
Enter PEM pass phrase:
Enter PEM pass phrase:
Enter PEM pass phrase:
Created slice with Name OTMGushDemo, URN urn:publicid:IDN+emulab.net+slice+OTMGushDemo

pc230:~/gush2/helper-scripts> ./handle-geni.py -n -f pg -a https://www.emulab.net/protogeni/xmlrpc/am createsliver OTMGushDemo OTMGushDemo.rspec
INFO:omni:Loading config file /users/xyx/.gcf/omni_config
INFO:omni:Using control framework pg
Asked https://www.emulab.net/protogeni/xmlrpc/am to reserve resources. Result:
<?xml version="1.0" ?><rspec type="manifest" xmlns="http://www.protogeni.net/resources/rspec/2" xmlns:flack="http://www.protogeni.net/resources/rspec/ext/flack/1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.protogeni.net/resources/rspec/2                                           http://www.protogeni.net/resources/rspec/2/manifest.xsd">
    <node client_id="node3" component_id="urn:publicid:IDN+emulab.net+node+pc513" component_manager_id="urn:publicid:IDN+emulab.net+authority+cm" exclusive="true" sliver_id="urn:publicid:IDN+emulab.net+sliver+72081">
        <sliver_type name="raw-pc">
          <disk_image name="urn:publicid:IDN+emulab.net+image+emulab-ops//UBUNTU10-STD"/>
        </sliver_type>
        <services>
            <install install_path="/tmp" url="http://ontime.oar.net/INSTALL/OTCNodeInstall.tgz"/>
            <execute command="sudo sh /tmp/ontime_software_install.sh" shell="sh"/>
         <login authentication="ssh-keys" hostname="pc513.emulab.net" port="22" username="xyx"/>    </services>
      <rs:vnode name="pc513" xmlns:rs="http://www.protogeni.net/resources/rspec/ext/emulab/1"/>    <host name="node3.OTMGushDemo.emulab-net.emulab.net"/>  </node>

</rspec>

From the output, you can get the hostname of new created node. Ping the node to get IP address

Node3: pc513.emulab.net(155.98.38.113)

This would be used in Section 7.6 "Use Gush to Configure !OnTimeMeasure" below.

7.4 Edit directory.xml file of Gush

A sample directory.xml file is provided with Gush.

Modify your slice name appropriately in the directory.xml file.

<?xml version="1.0" encoding="UTF-8"?>
<gush>
    <resource_manager type="geni-pg">
      <port_map slice="OTMGushDemo" port="61600"/>
    </resource_manager>
    <resource_manager type="geni-pg">
      <port_map slice="OTCDemo" port="61601"/>
    </resource_manager>

</gush>

7.5 Start Gush

In gush folder, invoke the following command to start gush and show nodes information.

./gush -P 15888
info nodes 

The output

pc230:~/gush2> ./gush -P 15888
gush> Gush has learned about the slice OTMGushDemo.
Gush has learned about the slice OTCDemo.
Updated information on the slice OTMGushDemo is available.
Updated information on the slice OTCDemo is available.


gush> info nodes
There are 4 known nodes:
[ P         ] xyx@pc513.emulab.net:61600(pref=0) (Disconnected.)
[ P         ] xyx@pc355.emulab.net:61601(pref=0) (Disconnected.)
[ P         ] xyx@pc337.emulab.net:61601(pref=0) (Disconnected.)
[ P         ] xyx@pc262.emulab.net:61601(pref=0) (Disconnected.)

7.6 Use Gush to Configure OnTimeMeasure

In section 7.3, we got

Node Beacon 3 - pc513.emulab.net(155.98.38.113) 

Now we will configure the Node Beacon 3 (155.98.38.113) to run measurements with other Node Beacons 1 and 2 in the in the OTCDemo slice.

Invoke the following Gush command to configure OnTimeMeasure.

(You need to change the IP address according to your related slice configuration in the following command)

connect pc355.emulab.net:61601
shell "sed -i 's/nodes:/nodes:\n- name: node3\n  IP: 155.98.38.113\n\n/g'   config.yaml"
shell "echo '- source: node3' >> measurement.yaml"
shell "echo '  metric: [MemFree]' >> measurement.yaml"
shell "python task_manage.py -c measurement.yaml"

The output:

gush> connect pc355.emulab.net:61601
Connecting to host xyx@pc355.emulab.net:61601.
gush> xyx@root.otcdemo.emulab-net.emulab.net:61601 has joined the mesh.


gush> shell "sed -i 's/nodes:/nodes:\n- name: node3\n  IP: 155.98.38.113\n\n/g'   config.yaml"
gush> shell "python task_manage.py -c measurement.yaml"
gush> xyx@root.otcdemo.emulab-net.emulab.net:61601,31357: configuring http://155.98.39.137:17248 OK
configuring http://155.98.39.62:17248 OK
configuring http://155.98.38.113:17248 OK
Configuration finished
All old tasks cleared
node1
node2
node2
node1
node1
node1
node2
node2
node3
node3
11 new tasks added
You have to restart the measurement service to put them into effort

7.7 Use Gush to Control and Query OnTimeMeasure

Invoke the following Gush command:

shell "python service_control.py restart"
shell "python service_status.py"
shell "python query.py -f '2012-01-30 18:00:00' -t '2012-01-31 20:00:00' -s node2 -d node1 -m RoundTripTime -r TimeSeries"

You may change the query start-and-end times to obtain results.

The output:

gush> shell "python service_control.py restart"
gush> xyx@root.otcdemo.emulab-net.emulab.net:61601,31379: stopped
running


gush> shell "python service_status.py"
gush> xyx@root.otcdemo.emulab-net.emulab.net:61601,31398: {'root_scheduler': 'OK', 'graphite': 'OK', 'file_server': 'OK', 'collector': 'OK'}


gush> shell "python query.py -f '2012-01-30 18:00:00' -t '2012-02-09 20:00:00' -s node2 -d node1 -m Throughput -r TimeSeries"


gush> xyx@root.otcdemo.emulab-net.emulab.net:61601,23395: http://155.98.38.103:17290/20120131173719_IperfTcp_Throughput.txt
Downloaded to local folder: /opt/OnTimeMeasure/OnTimeControl/results/20120131173719_IperfTcp_Throughput.txt

http://155.98.38.103:17280/render?height=300&width=500&from=18%3A00%2001/30/2012&until=20%3A00%2001/31/2012&target=Graphite.node2_node1.Throughput
Downloaded to local folder: /opt/OnTimeMeasure/OnTimeControl/results/20120131173719_IperfTcp_Throughput.png

You can access the query result in a web-browser or on the local file system of the Root Beacon.

If you want to query the new added measurement task (i.e., MemFree in Node 3), you need wait about 5 minutes and invoke the following Gush command: shell "python query.py -f '2012-01-30 18:00:00' -t '2012-02-09 20:00:00' -s node3 -d node3 -m MemFree -r TimeSeries" }}}

Note: You can also access Root Beacon through port 17280 in your web-browser (i.e., http://pc355.emulab.net:17280/) to check the new measurement data generated already in measurement dashboard.

The output

gush> shell "python query.py -f '2012-01-30 18:00:00' -t '2012-02-09 20:00:00' -s node3 -d node3 -m MemFree -r TimeSeries"
gush> xyx@root.otc2.emulab-net.emulab.net:61600,31595: http://155.98.39.36:17290/20120216201621_MemFree.txt
Downloaded to local folder: /opt/OnTimeMeasure/OnTimeControl/results/20120216201621_MemFree.txt


http://155.98.39.36:17280/render?height=300&width=500&from=18%3A00%2001/30/2012&until=20%3A00%2003/09/2012&target=Graphite.node3_node3.MemFree
Downloaded to local folder: /opt/OnTimeMeasure/OnTimeControl/results/20120216201621_MemFree.png

7.8 Further Operation

You can further follow the instructions in Sections 3-to-6 above to see measurement results for the newly added Node Beacon 3. NOTE:For Section 6 part, you need to manually install INSTOOLS as described at - http://groups.geni.net/geni/wiki/InstrumentationTools

8. Use of OMNI to add PlanetLab Nodes for OnTimeMeasure slice measurements

8.1 Install OMNI

For details on OMNI installation procedure, see - http://trac.gpolab.bbn.com/gcf/wiki/QuickStart

For details on how to use OMNI , see - http://groups.geni.net/geni/wiki/HowToUseOmni

Follow the instruction in http://groups.geni.net/geni/wiki/HowToUseOmni to install and configure OMNI

Follow the below steps to install OMNI in Ubuntu Desktop.

sudo apt-get install python-m2crypto python-dateutil \
                     python-pyopenssl libxmlsec1 xmlsec1 \
                     libxmlsec1-openssl libxmlsec1-dev  python-lxml

You can modify the default omni_config file with the same information of omni_config file used in Section 7.1.3 "Configure Gush" above.

8.2 Create a RSpec File

Use the following RSpec file to create a node in PlanetLab.

File OTMOmniDemo.rspec

<?xml version="1.0" ?>
<!-- Resources at AM:
        URN: unspecified_AM_URN
        URL: http://www.planet-lab.org:12346
 -->
<rspec type="request"
       xmlns="http://www.protogeni.net/resources/rspec/2"
       xmlns:emulab="http://www.protogeni.net/resources/rspec/ext/emulab/1"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.protogeni.net/resources/rspec/2
       http://www.protogeni.net/resources/rspec/2/request.xsd
       http://www.protogeni.net/resources/rspec/ext/emulab/1
       http://www.protogeni.net/resources/rspec/ext/emulab/1/ptop_extension.xsd">
  <node
       component_id="urn:publicid:IDN+plc:williams+node+planetlab3.williams.edu"
    component_manager_id="urn:publicid:IDN+plc+authority+am"
    component_name="planetlab3.williams.edu" exclusive="false"
    client_id="williamspl3">
      <hardware_type name="plab-pc"/>
      <hardware_type name="pc"/>
      <location country="us" latitude="43.0844" longitude="-77.6799"/>
      <sliver_type name="plab-vnode" />
  </node>

</rspec>

8.3 Create PlanetLab Sliver

Invoke the following command to create PlanetLab sliver in existing slice.

src/omni.py --api-version 2  -a http://www.planet-lab.org:12346 -o createsliver OTCDemo  OTMOmniDemo.rspec

The output:

pc230:~/omni/gcf-1.5.1> src/omni.py --api-version 2 -a http://www.planet-lab.org:12346 -o createsliver OTCDemo OTMOmniDemo.rspec
INFO:omni:Loading config file /users/xyx/.gcf/omni_config
INFO:omni:Using control framework pg
INFO:omni:Slice urn:publicid:IDN+emulab.net+slice+OTCDemo expires within 1 day on 2012-02-08 05:06:13 UTC
INFO:omni:Creating sliver(s) from rspec file OTMOmniDemo.rspec for slice urn:publicid:IDN+emulab.net+slice+OTCDemo
INFO:omni:Writing result of createsliver for slice: OTCDemo at AM: http://www.planet-lab.org:12346 to file OTCDemo-manifest-rspec-www-planet-lab-org.xml
INFO:omni:Writing to 'OTCDemo-manifest-rspec-www-planet-lab-org.xml'
INFO:omni: ------------------------------------------------------------
INFO:omni: Completed createsliver:

  Options as run:
                aggregate: http://www.planet-lab.org:12346
                api_version: 2
                framework: pg
                native: True
                output: True

  Args: createsliver OTCDemo OTMOmniDemo.rspec

  Result Summary: Slice urn:publicid:IDN+emulab.net+slice+OTCDemo expires within 1 day(s) on 2012-02-08 05:06:13 UTC
Reserved resources on http://www.planet-lab.org:12346.
   Saved createsliver results to OTCDemo-manifest-rspec-www-planet-lab-org.xml. 
INFO:omni: ============================================================

8.4 Get Node Information

Invoke the following command to check the sliver status

src/omni.py --api-version 2 -a http://www.planet-lab.org:12346 -o sliverstatus OTCDemo

The output:

root:~/omni/gcf-1.5.1> src/omni.py --api-version 2 -a http://www.planet-lab.org:12346 -o sliverstatus OTCDemo
INFO:omni:Loading config file /users/xyx/.gcf/omni_config
INFO:omni:Using control framework pg
INFO:omni:Slice urn:publicid:IDN+emulab.net+slice+OTCDemo expires within 1 day on 2012-02-08 05:06:13 UTC
INFO:omni:Status of Slice urn:publicid:IDN+emulab.net+slice+OTCDemo:
INFO:omni:Writing to 'OTCDemo-sliverstatus-www-planet-lab-org.json'
INFO:omni:Sliver status for Slice urn:publicid:IDN+emulab.net+slice+OTCDemo at AM URL http://www.planet-lab.org:12346
INFO:omni: ------------------------------------------------------------
INFO:omni: Completed sliverstatus:

  Options as run:
                aggregate: http://www.planet-lab.org:12346
                api_version: 2
                framework: pg
                native: True
                output: True

  Args: sliverstatus OTCDemo

  Result Summary: Slice urn:publicid:IDN+emulab.net+slice+OTCDemo expires within 1 day(s) on 2012-02-08 05:06:13 UTC
Saved sliverstatus on OTCDemo at AM http://www.planet-lab.org:12346 to file OTCDemo-sliverstatus-www-planet-lab-org.json.
Returned status of slivers on 1 of 1 possible aggregates.
INFO:omni: ============================================================

Check the output file OTCDemo-sliverstatus-www-planet-lab-org.json

{'geni_resources': [{'geni_error': '',
                     'geni_status': 'ready',
                     'geni_urn': 'urn:publicid:IDN+emulab.net+OTCDemo:23388:1086
0:0',
                     'pl_boot_state': 'boot',
                     'pl_hostname': 'planetlab3.williams.edu',
                     'pl_last_contact': '2012-02-07T18:06:22Z'}],
 'geni_status': 'ready',
 'geni_urn': 'urn:publicid:IDN+emulab.net+OTCDemo',
 'pl_expires': '2012-02-20T12:38:46Z',
 'pl_login': 'emulabnet_OTCDemo'}

In the file above, you can see whether your PlanetLab node reservation was successful, and also see the login name and host name of the reserved node (emulabnet_OTCDemo and planetlab3.williams.edu, in this case).

8.5 Install NodeBeacon

Login into the reserved node by typing the following command(Planetlab may need 30 minutes make the sliver ready before you login into the node).

ssh emulabnet_OTCDemo@planetlab3.williams.edu

NOTE: You need to modify login name and host name according to information you got in section 8.4)

Invoke the following command to install the node beacon.

wget http://ontime.oar.net/download/OnTimeMeasure_latest.php
tar xzf *.gz
tar xzf NodeBeacon.tar.gz
cd NodeBeacon/InstallScript
./ontime_software_install.sh

8.6 Further Operation

You can further follow the instructions in Sections 3-to-7 above to see measurement results for the newly added PlanetLab node. NOTE: For Section 6 part, you need to manually install INSTOOLS as described at - http://groups.geni.net/geni/wiki/InstrumentationTools

9. General Information

If you want to have a quick start of trying out the basic features of OnTimeMeasure, please refer to - Tutorial on using OnTimeMeasure with Emulab Java UI

You can also refer to - http://groups.geni.net/geni/wiki/OnTime-Install and http://groups.geni.net/geni/wiki/OTM-PlanetLabInstall to install OnTimeMeasure manually on ProtoGENI and PlanetLab aggregates.

Attachments (50)