Capture documentation and the process required to make a custom VM images available in InstaGENI

[lnevers@bbn.com]

This ticket will capture the process followed to build a VM custom image for OpenVZ containers. Using the existing instructions, the following steps will be completed:

1. Create an Experiment with an existing image
2. Login to node and customize
3. Create a new Image Descriptor and snapshot
4. Wait for the email saying the image creation is done
5. Verify new customized image
6. Make image available

Background: The following email exchange kicked off this effort:

On 8/30/12 11:50 AM, Robert Ricci wrote:

2) We have support for making disk images of VMs (OpenVZ containers) - from a user perspective, it looks basically the same as making an image from a bare metal node. We have our first guinea pig^HH^HH^HH^HH^HH user trying it out right now.

On 8/30/12 12:44 PM, Luisa Nevers wrote:

If you need a 2nd guinea pig... just point me to the instructions.

On 8/30/12 12:54 PM, Robert Ricci wrote:

There are no instructions it looks the same as doing it for a bare metal node.

On 8/30/12 1:07 PM, Luisa Nevers wrote:

So the existing instructions now needd to be modified to remove all instances on "physical nodes"? Sounds good, I will use the existing instructions.

[lnevers@bbn.com]

Starting with instructions found in the ​Custom OS Images section of the Emulab Tutotirial.

In ​Creating your Custom Image section, there is no mention of the Image versions that are supported for OpenVZ vs dedicated nodes Custom OS. I can make a custom Ubuntu image for dedicated nodes, but for VMs I need to have a fedora image, which is a restriction for OpenVZ that still applies. Right?

[lnevers@bbn.com]

Step 2 in "Creating you Custom Image" has pointers to a knowledge base item for custom images at ​https://www.emulab.net/kb-show.php3?xref_tag=linux_kernel, which defines how to build "Emulab standard Fedora Core 4 or 6 images (i.e., FC4-STD or FC6-STD)".

Using the "Experiment Creation GUI" (​https://www.emulab.net/clientui.php3) to define the one node experiment, selecting the "Node Properties" only shows one standard OS, FC4-STD. Should the "Experiment Creation GUI" include FC6-STD?

[lnevers@bbn.com]

1. Create an Experiment with an existing image - Completed

Manually defined an NS file and started an experiment. NS file content:

set ns [new Simulator]
source tb_compat.tcl
set node0 [$ns node]
tb-set-node-os $node0 FC6-STD 
$ns rtproto Static
$ns run

2. Login to node and customize - Completed

3. Create a new Image Descriptor and snapshot - Completed

Submitted a new Image Descriptor named "LNFC6", according to interface, this should take 10 or more minutes and an email should be sent when completed.

4. Wait for the email saying the image creation is done - NO EMAIL WAS RECEIVED

Waited for email up to 60 minutes, none received, checked the list of available OS and found my image listed.

5. Verify new customized image - Completed

Created a new experiment with the newly created LNFC6 to verify it. Used the following NS file:

set ns [new Simulator]
source tb_compat.tcl
set node0 [$ns node]
tb-set-node-os $node0 LNFC6 
$ns rtproto Static
$ns run

Was able to use image and verify that my modifications were preserved.

6. Make image available - Not Completed.

To make the image available to other experimenters via the AM API the image creator must send a request to the site administrator, who will then forward the request to ​instageni-ops@flux.utah.edu.

Two issues mentioned in ​http://groups.geni.net/instageni/ticket/20#comment:4 remain.

• The tutorial custom image instructions did not give any indication that the image could be shared with others. How will the experimenter know? Should be mentioned in the tutorial.
• How will the site administrators know that they need to forward the request to instageni-ops?

[lnevers@bbn.com]

ARGH, executed experiment at emulab rather than "​https://boss.utah.geniracks.net".

Re-executing the procedure again at the InstaGENI rack but ran into problems when creating the experiment using the images suggested from the knowlege base (FC4-STD, FC6-STD). Each of the FC4-STD, FC6-STD images result in "Invalid osid" when an experiment is run: {{{{

* /usr/testbed/lib/ns2ir/parse.tcl:

[tb-set-node-os] Invalid osid FC6-STD.

* ERROR: parse-ns: Parsing failed (error code 256)! * ERROR: batchexp: NS Parse failed! Cleaning up and exiting with status 1 ... Removing experiment directories ...

}}}

Links (ex. new image descriptor form) and information (available starting images) in the "Custom Os Images" instructions are specific to www.emulab.net and not to boss.utah.geniracks.net.

[lnevers@bbn.com]

Created initial experiment with from the list of available images (FEDORA10-STD and FEDORA15-STD). NS file used FEDORA15-STD for the initial experiment definition.

Received emails for starting the new experiment and for the successful custom image creation.

[lnevers@bbn.com]

Successfully verified the new custom image "LNFEDORA15", sending email to ​instageni-ops@flux.utah.edu to request that this image be made available via AM API.

[lnevers@bbn.com]

Reviewings the ​https://users.emulab.net/trac/protogeni/wiki/ImageHowTo page and have run into an error, the createimage.py fails:

$ ./createimage.py CustomFedora15 urn:publicid:IDN+utah.geniracks.net+sliver+8950
  File "./createimage.py", line 40
    global     = 1
               ^
SyntaxError: invalid syntax

Feedback on documentation:

In the "Creating a New Image" section, it would be helpful to have a link to the location of the test scripts which contain the "createimage.py" script.

I searched the PG wiki for "createimage.py and only one match was found, it was the page ​ImageHowTo, which is the page being reviewed. When done with testing, there should be an example added to the ProtoGENI Test Script page that describes the "createimage.py" script.

[lnevers@bbn.com]

On 9/5/12 12:12 PM, Leigh Stoller wrote:

global = 1 ^{SyntaxError?: invalid syntax}

Sorry, I missed that file in the commit. Should be installed now.

Feedback on documentation:

In the "Creating a New Image" section, it would be helpful to have a link to the location of the test scripts which contain the "createimage.py" script.

Okay, will do.

the page [​https://users.emulab.net/trac/protogeni/wiki/ImageHowTo ImageHowTo?], which is the page being reviewed. When done with testing, there should be an example added to the ProtoGENI Test Script page that describes the "createimage.py" script.

Okay.

Lbs

Picked up the new version of the tools, and re-ran command. Which now fails as follows:

$ ./createimage.py CustomFedora15 urn:publicid:IDN+utah.geniracks.net+sliver+8950 
Got my SA credential
No such slice registered here: Slice does not exist

I am using the sliver URN for the host to duplicate from the sliver manifest:

<node client_id="host1" component_id="urn:publicid:IDN+utah.geniracks.net+node+pc3" 
component_manager_id="urn:publicid:IDN+utah.geniracks.net+authority+cm" 
exclusive="false" sliver_id="urn:publicid:IDN+utah.geniracks.net+sliver+8950">        

This does not work, what should I be using for a sliver urn?

[lnevers@bbn.com]

On 9/5/12 1:13 PM, Leigh Stoller wrote:

The test scripts all require that a slice urn be provided if the slice is not the test script default. I will have to make some changes to the script (or you can look inside and make the obvious change).

All my tests have been with slices created using the other test scripts in that directory.

Oh, I see. You created this slice with omni or Flack using your GPO identity? In that case you have to do this from your GPO ops node since that is where your ssl certificate and slice credential are hosted.

So on *your* ops, run the createimage.py script as before, but add "-n lncustom" to the command line:

yourops> createimage.py -n lncustom ...

You will need to grab a copy of createimage.py of course, since the current version in not installed on your ops yet.

A TODO item for OMNI is to add support for this. Not sure who we ask for that though.

Lbs

On 9/5/12 1:13 PM, Leigh Stoller wrote:

The test scripts all require that a slice urn be provided if the slice is not the test script default. I will have to make some changes to the script (or you can look inside and make the obvious change).

All my tests have been with slices created using the other test scripts in that directory.

Oh, I see. You created this slice with omni or Flack using your GPO identity? In that case you have to do this from your GPO ops node since that is where your ssl certificate and slice credential are hosted.

So on *your* ops, run the createimage.py script as before, but add "-n lncustom" to the command line:

yourops> createimage.py -n lncustom ...

You will need to grab a copy of createimage.py of course, since the current version in not installed on your ops yet.

A TODO item for OMNI is to add support for this. Not sure who we ask for that though.

Lbs

If the credentials used to create the slice, impact where the createimage.py script can be run than it should be documented.

If I use my PG Utah credentials, do I have to run the createimage.py from ops.emulab.com?

[lnevers@bbn.com]

1. Created a new slice "IGcustom" using my PG Utah credentials.
2. Customized sliver.
3. On ops.utah.geniracks.net downloaded latest tools in my home directory.
4. Ran createimage.py script, but it still fails:

 > ./createimage.py -n IGcustom CustomFedora15  urn:publicid:IDN+utah.geniracks.net+sliver+8991
 Got my SA credential
 Asking for slice credential for IGcustom
 Got the slice credential
 Creating the Image ...
 Slice does not exist: Could not create image

[lnevers@bbn.com]

Forgot to mention that i also tried adding the suggested "Also add -m ​https://boss.utah.geniracks.net:12369/protogeni/xmlrpc/cm", but when I did the argument was not recognized:

> ./createimage.py -n IGcustom CustomFedora15 urn:publicid:IDN+utah.geniracks.net+sliver+8991 -m https://boss.utah.geniracks.net:12369/protogeni/xmlrpc/cm
usage: ./createimage.py [option...]
Options:
    -a file, --admincredentials=file    read admin credentials from file
    -c file, --credentials=file         read self-credentials from file
                                            [default: query from SA]
    -d, --debug                         be verbose about XML methods invoked
    -f file, --certificate=file         read SSL certificate from file
                                            [default: ~/.ssl/encrypted.pem]
    -h, --help                          show options and usage
    -l uri, --sa=uri                    specify uri of slice authority
                                            [default: local]
    -m uri, --cm=uri                    specify uri of component manager
                                            [default: local]
    -n name, --slicename=name           specify human-readable name of slice
                                            [default: mytestslice]
    -p file, --passphrase=file          read passphrase from file
                                            [default: ~/.ssl/password]
    -r file, --read-commands=file       specify additional configuration file
    -s file, --slicecredentials=file    read slice credentials from file
                                            [default: query from SA]
> 

[lnevers@bbn.com]

Replying to lnevers@bbn.com:

./createimage.py -n IGcustom CustomFedora15 urn:publicid:IDN+utah.geniracks.net+sliver+8991 -m ​https://boss.utah.geniracks.net:12369/protogeni/xmlrpc/cm

It seem argument placement matters:

> ./createimage.py -n IGcustom -m https://boss.utah.geniracks.net:12369/protogeni/xmlrpc/cm CustomFedora15 urn:publicid:IDN+utah.geniracks.net+sliver+8991
Got my SA credential
Asking for slice credential for IGcustom
Got the slice credential
Creating the Image ...
['urn:publicid:IDN+utah.geniracks.net+image+emulab-net:CustomFedora15', 'https://boss.utah.geniracks.net/image_metadata.php?uuid=b95afc42-f787-11e1-8336-00009b6224df']

So, in summary here is the outline of items that made this work for me:

1. Use PG credentials
2. Run command on ops.utah.geniracks.net
3. Specify sliver (-n slicename)
4. Specify component manager (-m uri)
5. Required image snapshot name is not obvious from command help.
6. Requires sliver URN for the host to be duplicated is not obvious from command help.

The above should should be in the documentation.

[lnevers@bbn.com]

Successfully used the image snapshot created by the createimage.py script in a request RSpec that included both:

• 'urn:publicid:IDN+utah.geniracks.net+image+emulab-net:CustomFedora15'
• '​https://boss.utah.geniracks.net/image_metadata.php?uuid=b95afc42-f787-11e1-8336-00009b6224df'

Used both disk_image types (urn & url) with PG Utah and PG GPO credentials on InstaGENI rack VMs.

RSpec node definitions used:

<node client_id="host1" exclusive="false">
    <sliver_type name="emulab-openvz" />
      <disk_image name="urn:publicid:IDN+utah.geniracks.net+image+emulab-net:CustomFedora15"/> 
      <interface client_id="host1:if0" />
      <interface client_id="host1:if1" />
  </node>

  <node client_id="host2" exclusive="false">
    <sliver_type name="emulab-openvz" />
      <disk_image url="https://boss.utah.geniracks.net/image_metadata.php?uuid=b95afc42-f787-11e1-8336-00009b6224df"/>
      <interface client_id="host2:if0" />
      <interface client_id="host2:if1" />
  </node>

  <link client_id="host1-and-host2-0">
    <interface_ref client_id="host1:if0"/>
    <interface_ref client_id="host2:if0"/>
  </link>

[lnevers@bbn.com]

Testing is completed for VM custom images, the following is a summary of documentation items that are requested for the "Creating a New Image" section:

1. Document that credentials must be from aggregate's authority. (Unlike AM API). As an example describe that PG credentials are used for InstaGENI Utah snapshot.
2. Run command on the aggregate's ops machine. If example is provided, may show that ops.utah.geniracks.net is used for InstaGENI Utah aggregate.
3. Specify slice name to overide defaults (-n slicename)
4. Specify component manager (-m uri)
5. Describe all options that are required, "custom image name" and "sliver URN" for the host to be duplicated.

An example command with all of the above would be helpful:

ops> ./createimage.py -n slicename -m https://boss.utah.geniracks.net:12369/protogeni/xmlrpc/cm 
CustomImageName urn:publicid:IDN+utah.geniracks.net+sliver+8991

[lnevers@bbn.com]

I re-executed the procedures to create a snapshot from a dedicated node (raw pc). Followed the same procedure, but used only raw pcs:

1. Created a sliver with one raw pc, no image specified, default was used.

2. Customized PC by installing a bunch of software packages. (sudo yum install iperf yum-utils sysstat iostat mpstat iptraf ntop vnstat -y)

3. Creates a snapshot successfully:

   > ./createimage.py -n rawcust -m https://boss.utah.geniracks.net:12369/protogeni/xmlrpc/cm CustomF15rawpc urn:publicid:IDN+utah.geniracks.net+sliver+9024
   Got my SA credential
   Asking for slice credential for rawcust
   Got the slice credential
   Creating the Image ...
   ['urn:publicid:IDN+utah.geniracks.net+image+emulab-net:CustomF15rawpc', 'https://boss.utah.geniracks.net/image_metadata.php?uuid=847c50f1-f796-11e1-8336-00009b6224df']
   

4. Created sliver to verify that the URN and URL could be used to request the customized raw-pc image, but sliver created with the URN and URL for the customized image above do not have any of the packages that were installed as part of the customization in step 2.

Am I doing something wrong? I assume I can follow the same procedure, as long as I use raw-pc images to create snapshots for raw-pcs.

[lnevers@bbn.com]

On 9/5/12 5:27 PM, Leigh Stoller wrote:

The logs show this:

<disk_image

name="​https://boss.utah.geniracks.net/image_metadata.php?uuid=847c50f1-f796-11e1-8336-00009b6224df"/>

which is incorrect; when using a url it has to be specified with a url tag:

<disk_image

url="​https://boss.utah.geniracks.net/image_metadata.php?uuid=847c50f1-f796-11e1-8336-00009b6224df"/>

I will put in some sanity checks to catch this.

I did not see a request that used the URN format ...

Lbs

Sorry I used the wrong tag in the disk_image.

I have modified the rspec to use the "url" tag and created a sliver named "verifyraw2" using the custom image url, but I still do not see the customization in the image that is loaded on the node.

I will now re-try the sliver with the "urn" tag next.

[lnevers@bbn.com]

Just create a sliver named "verifyurn" using the "urn" tag for the custom image, but still did not find the packages that should be in the custom image.

[lnevers@bbn.com]

On 9/6/12 10:37 AM, Leigh Stoller wrote:

Hi. The disk_image tag goes *inside* sliver_type. Like:

<node client_id="mynode" exclusive="true">

<sliver_type name="raw-pc">

<disk_image name="urn:publicid:IDN+foo.net+image+foo-net:booboo"/>

</sliver_type>

</node>

Lbs

You are right Leigh!

I forgot to run rspeclint!!! I re-ran a verification sliver and I do get all the customizations. SO there is no issue for raw-pcs custom images.

Are aggregates ever going to reject an invalid rpsec?!? ... which I keep writing :-)

[lnevers@bbn.com]

The ​https://users.emulab.net/trac/protogeni/wiki/ImageHowTo describes how to create a custom image, also the custom image feature is available in flack, and I got my wish (rspeclint is run on every request). Closing this ticket.