Opened 8 years ago

Closed 8 years ago

#20 closed (fixed)

Capture documentation and the process required to make a custom images available in InstaGENI

Reported by: lnevers@bbn.com Owned by: somebody
Priority: minor Milestone: IG-EXP-2
Component: Experiment Version: SPIRAL4
Keywords: Cc:
Dependencies:

Description

This ticket track the activities and documentation required to upload a custom image to an InstaGENI rack. The process start with an email request to geni-dev-utah@flux.utah.edu. Following is the initial email request:

From: Luisa Nevers <lnevers@bbn.com>
Date: 5/14/12 4:40 PM
To: geni-dev-utah@flux.utah.edu, instageni-design@geni.net

Hello,

I am executing the InstaGENI Acceptance tests and need to upload
two Ubuntu images to be used in experiments on the InstaGENI rack at Utah.

Please let me know if there are any special requirements  that I should
consider in creating the Ubuntu images.  I plan to run one Ubuntu image with
"emulab-openvz" and one Ubuntu with "raw-pc" type.

Also, if any documentation already exists that I should reference, please let me know.

Luisa

Attachments (7)

install-ubuntu1204-log.txt (201.0 KB) - added by lnevers@bbn.com 8 years ago.
BeginNSExperiment.jpg (278.4 KB) - added by lnevers@bbn.com 8 years ago.
customUbuntu1204.jpg (410.3 KB) - added by lnevers@bbn.com 8 years ago.
customUbuntu1204CreateDescriptor.jpg (287.0 KB) - added by lnevers@bbn.com 8 years ago.
nodesetup.jpg (382.0 KB) - added by lnevers@bbn.com 8 years ago.
runCustomImage.jpg (222.2 KB) - added by lnevers@bbn.com 8 years ago.
testCustomImage.jpg (188.0 KB) - added by lnevers@bbn.com 8 years ago.

Download all attachments as: .zip

Change History (12)

comment:1 Changed 8 years ago by lnevers@bbn.com

Capturing responses by Leigh

On 5/16/12 10:28 AM, Leigh Stoller wrote:

Please let me know if there are any special requirements that I should consider in creating the Ubuntu images. I plan to run one Ubuntu image with "emulab-openvz" and one Ubuntu with "raw-pc" type.

Hi. See my response in the other ticket; you do not get to pick the OS when you ask for an openvz node.

Lbs

On 5/16/12 10:39 AM, Leigh Stoller wrote:

I am executing the InstaGENI Acceptance tests and need to upload two Ubuntu images to be used in experiments on the InstaGENI rack at Utah.

Also, if any documentation already exists that I should reference, please let me know.

So the first question is where the images were generated. In general, images created on one testbed are not guaranteed to run on another cause of hardware differences.

The approach for creating images to use on the racks, is to use the Emulab interface on the rack. See this tutorial section on how to create an image.

https://users.emulab.net/trac/emulab/wiki/Tutorial#CustomOSImages

In short, you would log into your Emulab account, allocate a node, load your image, test your image, take a snapshot, etc. Others at BBN are familiar with this process so they can help you out.

Once it works, the procedure for getting it "exported" to the Protogeni interface, is to send us mail and tell us about it. We will move the image into place, set a couple of bits in the DB state, and presto, you can use ask for it in your rspec.

Lbs

Thank you, now reading https://users.emulab.net/trac/emulab/wiki/Tutorial#CustomOSImages

Changed 8 years ago by lnevers@bbn.com

Attachment: install-ubuntu1204-log.txt added

comment:2 in reply to:  1 Changed 8 years ago by lnevers@bbn.com

Capturing Update for the Custom Image creation:

On 5/16/12 10:39 AM, Leigh Stoller wrote:

The approach for creating images to use on the racks, is to use the Emulab interface on the rack. See this tutorial section on how to create an image.

https://users.emulab.net/trac/emulab/wiki/Tutorial#CustomOSImages

In short, you would log into your Emulab account, allocate a node, load your image, test your image, take a snapshot, etc. Others at BBN are familiar with this process so they can help you out.

I followed the instruction given in the tutorial and created a one node experiment which was defined with "tb-set-node-os $node0 UBUNTU11-64-STD" as a starting OS. Experiment was started and I was able to login to the node without problem.

My goal custom image is Ubuntu 12.04, so I upgraded the Ubuntu 11.04 node to Ubuntu 12.04, but upon reboot the node did not come up. Logs for the assigned node (pc250.emulab.net) show the following:

PXE version 2.1, real mode entry point @9c3e:0106 (!PXE)
BIOS 640kB/2095872kB available memory

FreeBSD/i386 BTX-based Emulab PXE boot loader, Revision 1.17
(root@node.freebsd73-node.testbed.emulab.net, Fri Jun 10 16:38:21 MDT 2011)
pxe_open: using PXE DHCP info
pxe_open: ipaddr: 155.98.39.50/255.255.252.0
pxe_open: server addr: 155.98.38.163
pxe_open: server path: /tftpboot
pxe_open: gateway ip:  155.98.36.1
DHCP Info: IP: 155.98.39.50, Server: 155.98.38.163, Gateway: 155.98.36.1
Type a key for interactive mode (quick, quick!)
|Loading disk 0x80 MBR and booting system partition 2
/-\

The upgrade is captured in the attached file named install-ubuntu1204-log.txt.

Determine what to do next....

comment:3 Changed 8 years ago by lnevers@bbn.com

InstaGENI Custom Linux Image

The following procedure was followed to customize and Ubuntu image for use in the InstaGENI Utah rack.

Initial email requests was sent to the geni-dev-utah@flux.utah.edu list. I was pointed to the Tutorial page in the Linux Custom OS Image section. Also found the Windows Custom Images page, which will be captured in separate procedure.

Here are the high level steps for creating a custom Ubuntu image for an InstaGENI rack:

  1. Create a NS experiment with one of the existing image.
  2. Once the experiment is running login to the node and customize.
  3. Create a new Image Descriptor and snapshot
  4. Wait for the email saying the image creation is done.
  5. Verify new customized image.
  6. Make image available

This scenario generated a custom Ubuntu 12.04 image based on an available Ubuntu 11 image.

Step 1. Create an Experiment with an existing image

The first step in the Custom Images section requires defining a Network Simulator (NS) file with a Java GUI found at https://boss.utah.geniracks.net/clientui.php3. Used this client to define a one nodes NS files as suggested, or one can just simply use the following file:

    set ns [new Simulator]
    source tb_compat.tcl
    set node0 [$ns node]
    tb-set-node-os $node0 UBUNTU11-64-STD
    $ns rtpproto Static
    $ns run

Referred to the current list of available OS Image Descriptors to choose starting OS tb-set-node-os UBUNTU11-64-STD.

Once the ns file is defined, you can create an experiment. Login to https://boss.utah.geniracks.net/ and choose "Experimentation"-> "Begin an Experiment". On the "Begin a Testbed Experiment" page name your experiment, and load the NS file shown above, make sure to run the "Syntax Check" and modify "Linktest Option" to skip the Link test.

(see attached capture of the form named BeginNSExperiment.jpg)

Once the form is filled, you can now press "Submit" which brings up a new pages showing how the experiment is being set up and which node is used.

(see attached capture of the page named nodesetup.jpg)

Step 2. Login to node and customize

Once the experiment is set up, you may login and customize your node.

lnevers@sendaria:~/gcf-1.6.1$ ssh pc250.emulab.net
Warning: Permanently added 'pc250.emulab.net,155.98.39.50' (RSA) to the list of known hosts.
Welcome to Ubuntu 11.04 (GNU/Linux 2.6.38.7-1.0emulab x86_64)

 * Documentation:  https://help.ubuntu.com/
/users/lnevers% cat /etc/issue
Ubuntu 11.04 \n \l
/users/lnevers% 

The customization in this procedure executed an upgrade from Ubuntu 11.04 to Ubuntu 12.04.

Step 3. Create a new Image Descriptor and snapshot

Once the customization in step 2 is completed, use the Create new Image Descriptor page to create the Image Descriptor which will be associated with your new custom image.

(see attached capture of the page named customUbuntu1204.jpg)

Note: "*Which DOS Partition[1]: (DOS partitions are numbered 1-4)" value used was 2 for the Ubuntu custom image.

When the request to create a new image descriptor is submitted, user told that email notification will be sent upon completion.

(see attached capture of the page named customUbuntu1204CreateDescriptor.jpg)

Step 4. Wait for the email saying the image creation is done

The following email notification was generated from the successful image creation:

Subject: UTAHGENIRACK: Image Creation on pc2 Completed: gpo/LNUBUNTU1204
From: Luisa Nevers <lnevers@bbn.com>
Date: 5/21/12 6:05 PM
To: Luisa Nevers <lnevers@bbn.com>

Image creation on pc2 has completed. As you requested, the
image has been written to /proj/gpo/images/LNUBUNTU1204.ndz.
You may now os_load this image on other nodes in your experiment.

--------- /tmp/logfile.Itj0ZZ --------
reboot (pc2): Attempting to reboot ...
reboot (pc2): Successful!
reboot: Done. There were 0 failures.
   pc2 
Waiting for nodes to come up.
All nodes are up.
pc2: started image capture, waiting up to 72 minutes
pc2: still waiting ... it has been 2 minutes. Current image size: 504365056 bytes.
pc2: still waiting ... it has been 4 minutes. Current image size: 1008730112 bytes.
pc2: image capture has completed: status='0'
All nodes have completed their command.
reboot (pc2): Attempting to reboot ...
Connection to pc2.utah.geniracks.net closed by remote host.
reboot (pc2): Successful!
reboot: Done. There were 0 failures.
   pc2 
/proj/gpo/images/LNUBUNTU1204.ndz: 1194 chunks, 2082 regions, 41970 hashregions, 2678062080 data bytes
2678062080 bytes: inflate cycles: 50950875517
Swapout signature file created
Image creation succeeded.
Image written to /proj/gpo/images/LNUBUNTU1204.ndz.

------------------ Prepare Output ----------------
Legacy library ctime.pl will be removed from the Perl core distribution in the next major release. Please install the separate libperl4-corelibs-perl package. It is being used at /usr/local/etc/emulab/prepare, line 9.
Cleaning node; removing configuration files
Running /usr/local/etc/emulab/rc/rc.config to clean up ...
Checking manifest...
Running 'diff -q -u /etc/emulab/gshadow /etc/emulab/gshadow.new'
Files /etc/emulab/gshadow and /etc/emulab/gshadow.new differ
Files /etc/emulab/gshadow.new and /etc/emulab/gshadow differ, but we can't show the changes, and I was told not to update the master files!
Running 'diff -q -u /etc/emulab/shadow /etc/emulab/shadow.new'
Files /etc/emulab/shadow and /etc/emulab/shadow.new differ
Files /etc/emulab/shadow.new and /etc/emulab/shadow differ, but we can't show the changes, and I was told not to update the master files!
Running 'diff -u /etc/emulab/group /etc/emulab/group.new'
--- /etc/emulab/group	2011-08-25 11:59:34.000000000 -0600
+++ /etc/emulab/group.new	2012-05-21 15:54:31.000000000 -0600
@@ -51,3 +51,6 @@
 ssl-cert:x:111:
 postfix:x:112:
 postdrop:x:113:
+messagebus:x:114:
+utempter:x:116:
+_cvsadmin:x:115:
Files /etc/emulab/group.new and /etc/emulab/group differ, but I was told not to update the master files!.
Running 'diff -u /etc/emulab/passwd /etc/emulab/passwd.new'
--- /etc/emulab/passwd	2011-08-25 11:59:00.000000000 -0600
+++ /etc/emulab/passwd.new	2012-05-21 15:54:31.000000000 -0600
@@ -23,3 +23,4 @@
 statd:x:103:65534::/var/lib/nfs:/bin/false
 ntp:x:104:110::/home/ntp:/bin/false
 postfix:x:105:112::/var/spool/postfix:/bin/false
+messagebus:x:106:114::/var/run/dbus:/bin/false
Files /etc/emulab/passwd.new and /etc/emulab/passwd differ, but I was told not to update the master files!.
Resetting passwd and group files
/bin/cp: cannot stat `/etc/emulab/hosts': No such file or directory
*** /usr/local/etc/emulab/rc/rc.hostnames:
    Could not copy default /etc/hosts into place!
  Failed running rc.hostnames (512)! at /usr/local/etc/emulab/libsetup.pm line 1085.
Removing old DB files ...
Removing old /etc/dumpdates file ...
Creating stub /etc/dumpdates file ...
Cleaning logfiles ...
Removing accounting files ...
Removing root's history ...
Removing root's mailfile ...
Resetting NTP drift ...
Clearing out /var/run ...
Clearing out /tmp ...
Cleaning out /local/logs ...
Removing SFS files ...
Clearing out directories in /var ...
Cleaning up /var/tmp ....
Clearing out directories in /var/emulab ...
Clearing out old Emulab scripts and binaries in /usr/local/etc/emulab ...
Removing backup files in /etc
Removing assorted unix-domain socket files
prepare ran successfully!

Step 5. Verify new customized image

Once the image is available, verify that it can be used by creating a new one node experiment that uses the image. For this scenario the following ns file was generated to test the new custom image:

set ns [new Simulator]
source tb_compat.tcl
set node0 [$ns node]
tb-set-node-os $node0 LNUBUNTU1204
$ns rtproto Static
$ns run

Begin a new experiment by choosing "Experimentation"-> "Begin an Experiment". On the "Begin a Testbed Experiment" page name your experiment, and load the NS file shown above, run the "Syntax Check" and modify "Linktest Option" to skip the Link test.

(see attached capture of form named testCustomImage.jpg)

Once you click "Submit", the experiment will be started.

(see attached capture of the image named runCustomImage.jpg)

Step 6. Make image available

To make the image available to other experimenters via the AM API send a request to the site administrator, who will forward the request to instageni-ops@flux.utah.edu.

Changed 8 years ago by lnevers@bbn.com

Attachment: BeginNSExperiment.jpg added

Changed 8 years ago by lnevers@bbn.com

Attachment: customUbuntu1204.jpg added

Changed 8 years ago by lnevers@bbn.com

Changed 8 years ago by lnevers@bbn.com

Attachment: nodesetup.jpg added

Changed 8 years ago by lnevers@bbn.com

Attachment: runCustomImage.jpg added

Changed 8 years ago by lnevers@bbn.com

Attachment: testCustomImage.jpg added

comment:4 Changed 8 years ago by lnevers@bbn.com

The custom Ubuntu image has been used in several experiments.

Only documentation item remains for defining site admin responsibilities, as stated in email:

On 5/23/12 11:08 AM, Leigh Stoller wrote:

  1. The tutorial custom image instructions did not give any indication that

the image could be shared with others. How will the experimenter know? Should this be mentioned in the tutorial?

  1. How will the site admins know that they need to forward the request to

instageni-ops?

I've added a todo item to document this ...

Lbs

comment:5 Changed 8 years ago by ricci@cs.utah.edu

Resolution: fixed
Status: newclosed

We implemented an image-creation call as an extension to the GENI AM API. Leigh's mail about this was:

I added a new method to the CM (and AM) API, which allows a geni user to
create a sliver, customize the node, and then take a snapshot (possibly
creating a new image descriptor) without having to use the Emulab web
interface. The API looks like:

     string CreateImage(slice_urn, imagename, sliver_urn, credentials[]);

Imagename is just an identifier like "myfedora" or whatever. The slice must
be unlocked and the sliver in the ready state.

A new image descriptor will be created if it does not exist, and then the
node will be rebooted and a snapshot taken. When complete, the node is
rebooted again.

The return value is the urn of the image. For example:

    urn:publicid:IDN+emulab.net+image+pgeni-gpolab-bbn-com:myfedora

which is how you request the new image in your rspec.

Once the operation starts, the slice is locked until the backend finishes.
This is something that I might revisit later, but this was the easiest
approach that ensures consistency. It means you cannot ask for sliverstatus
either (another issue to address someday) for a while.

The image is written to a file on the NFS server that is accessible
from the node. Look in /proj/$pid/images.

Note that these images are not going to show up in discover since they are
private. You can still request them using the URN above, but unless the
image is "exported" by an admin, it is hidden and only available to slices
created belonging to the same SA.

Note: See TracTickets for help on using tickets.