= IG-EXP-1: Bare Metal Support Acceptance Test = This page captures status for the test case IG-EXP-1, which verified support for bare metal nodes. For overall status see the [wiki:GENIRacksHome/InstageniRacks/AcceptanceTestStatus InstaGENI Acceptance Test Status] page. ''Last update: 2013/02/06'' = Test Status = This section captures the status for each step in the acceptance test plan. || '''Step''' || '''State''' || ''' Ticket ''' || '''Notes''' || || Step 1 ||[[Color(green,Pass)]] || || || || Step 2 ||[[Color(red,Fail)]] || instaticket:13 ||Windows not supported. || || Step 3 ||[[Color(red,Fail)]] || ||Windows not supported. || || Step 4 ||[[Color(green,Pass)]] || || || || Step 5 ||[[Color(green,Pass)]] || || || || Step 6 ||[[Color(green,Pass)]] || || || || Step 7 || || || || [[BR]] || '''State Legend''' || '''Description''' || || [[Color(green,Pass)]] || Test completed and met all criteria || || [[Color(#98FB98,Pass: most criteria)]] || Test completed and met most criteria. Exceptions documented || || [[Color(red,Fail)]] || Test completed and failed to meet criteria. || || [[Color(yellow,Complete)]] || Test completed but will require re-execution due to expected changes || || [[Color(orange,Blocked)]] || Blocked by ticketed issue(s). || || [[Color(#63B8FF,In Progress)]] || Currently under test. || [[BR]] = Test Plan Steps = A nick_name alias is used for the GPO InstaGENI aggregate manager in the omni_config: {{{ ig-gpo=,http://instageni.gpolab.bbn.com/protogeni/xmlrpc/am/2.0 }}} GPO ProtoGENI user credentials for lnevers@bbn.com were used for Experimenter1. == Step 1. Determine which nodes can be used as exclusive nodes == Able to determine which node are exclusive and if they are available via Omni listresources. !ListResources obtained with the following Omni command: {{{ $ omni.py -a ig-gpo listresources --available -o }}} == Step 2. Obtain 2 licensed recent Microsoft OS images for physical nodes from the site (BBN) == Windows is not supported. Need to get InstaGENI folks to document the non-support of MS Windows and cleanup of windows references ([http://groups.geni.net/instageni/ticket/13 ticket 13]). == Step 3. Reserve and boot 2 physical nodes using Microsoft image == MS Windows is not verified ([http://groups.geni.net/instageni/ticket/13 ticket 13]) According to email from Leigh, this is very time intensive and would require weeks. Also Windows XP instructions existed, but have since been removed. == Step 4. Obtain a recent Linux OS image for physical nodes from the InstaGENI list == Have requested the list of currently supported OS by InstaGENI. Initially only Standard 32-bit Fedora 15 image was available, two images have been added to the list of available OS in the Utah InstaGENI rack ([http://groups.geni.net/instageni/ticket/13 ticket 13]). Tested each images made available to experimenters, by setting up a 2 raw pc with 1 lan experiment for each image and attached Rspec: - FreeBSD 8.2 32-bit version - Standard 64-bit Ubuntu 12.04 image - Standard 32-bit Fedora 15 image (default) Command used to determine images: {{{ $ $ omni.py -a ig-gpo listresources --available -o $ egrep "node component|disk_image|available" rspec-instageni-gpolab-bbn-com-protogeniv2.xml <...> }}} == Step 5. Reserve and boot physical node using each loading a different Linux OS image == The RSpec [http://groups.geni.net/geni/browser/trunk/GENIRacks/InstaGENI/Spiral4/RSpecs/AcceptanceTests/IG-EXP-1/IG-EXP-1-gpo.rspec IG-EXP-1-gpo.rspec] that includes 1 FreeBSD and 1 Ubuntu node was used to create a slice and sliver at the GPO InstaGENI: {{{ $ omni.py createslice IG-EXP-1 INFO:omni:Loading config file /home/lnevers/.gcf/omni_config INFO:omni:Using control framework pg INFO:omni:Created slice with Name IG-EXP-1, URN urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-EXP-1, Expiration 2013-01-25 21:04:16+00:00 INFO:omni: ------------------------------------------------------------ INFO:omni: Completed createslice: Options as run: framework: pg Args: createslice IG-EXP-1 Result Summary: Created slice with Name IG-EXP-1, URN urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-EXP-1, Expiration 2013-01-25 21:04:16+00:00 INFO:omni: ============================================================ $ omni.py createsliver -a ig-gpo IG-EXP-1 IG-EXP-1-gpo.rspec INFO:omni:Loading config file /home/lnevers/.gcf/omni_config INFO:omni:Using control framework pg INFO:omni:Substituting AM nickname ig-gpo with URL http://instageni.gpolab.bbn.com/protogeni/xmlrpc/am/2.0, URN unspecified_AM_URN INFO:omni:Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-EXP-1 expires on 2013-01-25 21:04:16 UTC INFO:omni:Substituting AM nickname ig-gpo with URL http://instageni.gpolab.bbn.com/protogeni/xmlrpc/am/2.0, URN unspecified_AM_URN INFO:omni:Substituting AM nickname ig-gpo with URL http://instageni.gpolab.bbn.com/protogeni/xmlrpc/am/2.0, URN unspecified_AM_URN INFO:omni:Creating sliver(s) from rspec file IG-EXP-1-gpo.rspec for slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-EXP-1 INFO:omni:Got return from CreateSliver for slice IG-EXP-1 at http://instageni.gpolab.bbn.com/protogeni/xmlrpc/am/2.0: INFO:omni: INFO:omni: INFO:omni: ------------------------------------------------------------ INFO:omni: Completed createsliver: Options as run: aggregate: ['ig-gpo'] framework: pg Args: createsliver IG-EXP-1 IG-EXP-1-gpo.rspec Result Summary: Got Reserved resources RSpec from instageni-gpolab-bbn-com-protogeniv2 INFO:omni: ============================================================ }}} Note: A version of the RSpec [http://groups.geni.net/geni/browser/trunk/GENIRacks/InstaGENI/Spiral4/RSpecs/AcceptanceTests/IG-EXP-1/IG-EXP-1-utah.rspec IG-EXP-1-utah.rspec] is available for IG Utah rack. Once sliver is ready, determined the login information for the sliver EG-EXP-1 nodes: {{{ $ readyToLogin.py -a ig-gpo IG-EXP-1 <...> host-ubuntu12's geni_status is: changing (am_status:ready) User lnevers logins to host-ubuntu12 using: xterm -e ssh -i /home/lnevers/.ssh/id_rsa lnevers@pc5.instageni.gpolab.bbn.com & host-bsd's geni_status is: ready (am_status:ready) User lnevers logins to host-bsd using: xterm -e ssh -i /home/lnevers/.ssh/id_rsa lnevers@pc4.instageni.gpolab.bbn.com & }}} Verified dataplane connectivity for each host in each experiment. Checked the FreeBSD host: {{{ $ ssh -i /home/lnevers/.ssh/id_rsa lnevers@pc4.instageni.gpolab.bbn.com [lnevers@host-bsd ~]$ /sbin/ifconfig bce1 bce1: flags=8843 metric 0 mtu 1500 options=c01bb ether 10:60:4b:96:00:d6 inet 10.10.1.2 netmask 0xffffff00 broadcast 10.10.1.255 media: Ethernet 1000baseT status: active [lnevers@host-bsd ~]$ ping 10.10.1.1 PING 10.10.1.1 (10.10.1.1): 56 data bytes 64 bytes from 10.10.1.1: icmp_seq=0 ttl=64 time=0.233 ms 64 bytes from 10.10.1.1: icmp_seq=1 ttl=64 time=0.110 ms }}} Checked the Ubuntu host: {{{ lnevers@localhost:~$ /sbin/ifconfig eth1 eth1 Link encap:Ethernet HWaddr 10:60:4b:9c:47:6a inet addr:10.10.1.1 Bcast:10.10.1.255 Mask:255.255.255.0 inet6 addr: fe80::1260:4bff:fe9c:476a/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:6 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:492 (492.0 B) Interrupt:37 Memory:f2000000-f2012800 lnevers@localhost:~$ ping 10.10.1.2 PING 10.10.1.2 (10.10.1.2) 56(84) bytes of data. 64 bytes from 10.10.1.2: icmp_req=1 ttl=64 time=0.103 ms 64 bytes from 10.10.1.2: icmp_req=2 ttl=64 time=0.091 ms }}} == Step 6. Release physical node resource == Released the sliver compute resources: {{{ $ omni.py deletesliver -a ig-gpo IG-EXP-1 INFO:omni:Loading config file /home/lnevers/.gcf/omni_config INFO:omni:Using control framework pg INFO:omni:Substituting AM nickname ig-gpo with URL http://instageni.gpolab.bbn.com/protogeni/xmlrpc/am/2.0, URN unspecified_AM_URN INFO:omni:Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-EXP-1 expires on 2013-01-25 21:04:16 UTC INFO:omni:Substituting AM nickname ig-gpo with URL http://instageni.gpolab.bbn.com/protogeni/xmlrpc/am/2.0, URN unspecified_AM_URN INFO:omni:Deleted sliver urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-EXP-1 on unspecified_AM_URN at http://instageni.gpolab.bbn.com/protogeni/xmlrpc/am/2.0 INFO:omni: ------------------------------------------------------------ INFO:omni: Completed deletesliver: Options as run: aggregate: ['ig-gpo'] framework: pg Args: deletesliver IG-EXP-1 Result Summary: Deleted sliver urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-EXP-1 on unspecified_AM_URN at http://instageni.gpolab.bbn.com/protogeni/xmlrpc/am/2.0 INFO:omni: ============================================================ }}} == Step 7. Modify Aggregate resource allocation for the rack to add node == This steps validates the ability to adjust the shared node allocation by following the instructions available in the [https://wiki.emulab.net/Emulab/wiki/SetupSharedNodes Setup Shared Nodes] page. The procedure executes the following: a. Checked resource InstaGENI GPO rack to find available dedicated node. b. Add the node to the shared pool as defined in the [https://wiki.emulab.net/Emulab/wiki/SetupSharedNodes Setup Shared Nodes] instructions. c. Verify that the node becomes part of the shared pool and VMs from the node can be used in an experiment. d. Verify that no VMs are in use for the new shared node to verify no VMs are in use. e. Modify shared-nodes experiment to remove the node(if not in use). f. Verify that the node is no longer part of the shared pool. Here are details for each step that is executed: a. Checked resource status in GPO rack: Referenced the [https://boss.instageni.gpolab.bbn.com/nodecontrol_list.php3?showtype=pcs PCs status] page, to determine available node to re-allocate. The following was the status for the nodes: [[Image(IG-EXP-1-pre-alloc.jpg]] The status show 3 nodes in used pc1, pc2, and pc3. The two nodes that are used as by the shared-nodes experiment (exclusive=false) are pc1 and pc2. Will add pc4 to the shared node pool. b. Use instructions in [https://wiki.emulab.net/Emulab/wiki/SetupSharedNodes Setup Shared Nodes] page Displayed the initial NS file for the shared-nodes experiment: {{{ source tb_compat.tcl set ns [new Simulator] set vhost1 [$ns node] tb-set-node-os $vhost1 FEDORA15-OPENVZ-STD tb-set-node-sharingmode $vhost1 "shared_local" set vhost2 [$ns node] tb-set-node-os $vhost2 FEDORA15-OPENVZ-STD tb-set-node-sharingmode $vhost2 "shared_local" tb-fix-node $vhost1 pc1 tb-fix-node $vhost2 pc2 $ns rtproto Static $ns run }}} While in '''red-dot mode", selected the shared-nodes experiment: [[Image(IG-EXP-1-a.jpg)]] On the page above, selected the option "Modify Experiment" link, and used the ''Edit:" feature to modify the original NS file to add pc4: {{{ source tb_compat.tcl set ns [new Simulator] set vhost1 [$ns node] tb-set-node-os $vhost1 FEDORA15-OPENVZ-STD tb-set-node-sharingmode $vhost1 "shared_local" set vhost2 [$ns node] tb-set-node-os $vhost2 FEDORA15-OPENVZ-STD tb-set-node-sharingmode $vhost2 "shared_local" set vhost3 [$ns node] tb-set-node-os $vhost3 FEDORA15-OPENVZ-STD tb-set-node-sharingmode $vhost3 "shared_local" tb-fix-node $vhost1 pc1 tb-fix-node $vhost2 pc2 tb-fix-node $vhost3 pc4 $ns rtproto Static $ns run }}} Submitted the modifications and the following was reported as the allocation was modified: {{{ Doing a preswapmod on the experiment archive ... Backing up old experiment state ... 10:42:28:275547 Running 'tbprerun -e 13 shared-nodes-modify.ns' Beginning pre run for emulab-ops/shared-nodes. 10:42:28:567671 Running parser ... 10:42:28:901234 Parser done! 10:42:31:271349 Setting up static routes (if requested) ... *** WARNING: staticroutes: *** No links or lans in experiment emulab-ops/shared-nodes! Generating topomap ... Verifying parse ... Doing a pre-assign: '/usr/testbed/bin/vtopgen -p emulab-ops shared-nodes' ... Checking for feature LibVtopTest. Using test version of libvtop ... Minimum nodes = 3 Maximum nodes = 3 Writing environment strings ... Setting up additional program agent support ... Setting up additional network agent support ... Writing program agent info ... Pre run finished. 10:42:34:348647 Running 'tbswap modify -reboot -eventsys_restart -noswapout -noreconfig emulab-ops shared-nodes' Beginning swap-modify for emulab-ops/shared-nodes (13). 02/06/2013 10:42:34 TIMESTAMP: 10:42:34:635812 tbswap modify started Checking for feature SyncVlans. Checking with Admission Control ... Backing up physical state... Stopping the event system Checking for feature NewEventScheduler. Mapping to physical reality ... TIMESTAMP: 10:42:36:53970 mapper wrapper started Checking for feature OldMapper. Checking for feature NewAssign. Checking for feature ControlNetVlans. Checking for feature LibVtopTest. Using test version of libvtop ... Starting the new and improved mapper wrapper. Minimum nodes = 3 Maximum nodes = 3 Reserved pnodes = 2 Clearing physical state before updating. Mapper loop 1 ptopargs: '-p emulab-ops -e shared-nodes ' assign command: 'assign -P -f text emulab-ops-shared-nodes-864.ptop emulab-ops-shared-nodes-864.vtop' Reading assign results. [Node: pc1] already reserved in holding reservation. [Node: pc2] already reserved in holding reservation. pc4 pc1 pc2 TIMESTAMP: 10:42:39:703877 mapper wrapper finished Mapped to physical reality! Synchronizing VLANs. getTrunksForVlan: 8: procurve2 mapVlansToSwitches: procurve2 getExperimentTrunksForVlan: 8: procurve2 mapStaleVlansToSwitches: procurve2 procurve2 -> startChildCall(FlipDebug) Experiment vlans: 8 Trunk Ports: pc1:3.1 pc2:3.1 pc2:2.1 pc4:1.1 pc2:1.1 pc4:3.1 pc4:2.1 pc1:2.1 pc1:1.1 Existing vlans: 8 Stale vlans: Existing Trunk Ports: pc1:3.1 pc2:3.1 pc2:1.1 pc1:1.1 Creating dual mode trunking vlan 8: pc4:1.1 pc4:2.1 pc4:3.1 snmpit_stack::findVlans( ) procurve2 -> startChildCall(findVlans) procurve2::findVlans snmpit_stack::reapCall(findVlans) stack::findVlans calling procurve2 VLAN 8 already exists snmpit_stack::findVlan( 8 ) snmpit_stack::findVlans( 8 ) procurve2 -> startChildCall(findVlans) procurve2::findVlans snmpit_stack::reapCall(findVlans) stack::findVlans calling procurve2 snmpit_stack::findVlans( ) procurve2 -> startChildCall(findVlans) procurve2::findVlans snmpit_stack::reapCall(findVlans) stack::findVlans calling procurve2 getTrunksForVlan: 8: procurve2 procurve2 -> startChildCall(vlanNumberExists) Vlan 8 already exists on procurve2 jitdev::setPortVlan( 257 pc4:1.1 pc4:2.1 pc4:3.1 ) procurve2 -> startChildCall(setPortVlan) procurve2::setPortVlan(257)as ifIndexes: 106,107,108 portControl: enable -> (pc4:1.1 pc4:2.1 pc4:3.1) checking row 106 for up ... Port pc4:1.1, row 106 was down Setting pc4:1.1 (r 106) to up...succeeded. checking row 107 for up ... Port pc4:2.1, row 107 was down Setting pc4:2.1 (r 107) to up...succeeded. checking row 108 for up ... Port pc4:3.1, row 108 was down Setting pc4:3.1 (r 108) to up...succeeded. procurve2 -> startChildCall(portControl) portControl: 1000mbit -> (pc4:1.1) checking row 106 for auto-1000Mbits ... Port pc4:1.1, row 106 was auto-1000Mbits procurve2 -> startChildCall(portControl) portControl: full -> (pc4:1.1) checking row 106 for full ... Port pc4:1.1, row 106 was auto-1000Mbits procurve2 -> startChildCall(portControl) portControl: 1000mbit -> (pc4:2.1) checking row 107 for auto-1000Mbits ... Port pc4:2.1, row 107 was auto-1000Mbits procurve2 -> startChildCall(portControl) portControl: full -> (pc4:2.1) checking row 107 for full ... Port pc4:2.1, row 107 was auto-1000Mbits procurve2 -> startChildCall(portControl) portControl: 1000mbit -> (pc4:3.1) checking row 108 for auto-1000Mbits ... Port pc4:3.1, row 108 was auto-1000Mbits procurve2 -> startChildCall(portControl) portControl: full -> (pc4:3.1) checking row 108 for full ... Port pc4:3.1, row 108 was auto-1000Mbits Enabling dual mode trunking on pc1:2.1:8 Enabling dual trunking (tagging) on pc1:2.1 8 procurve2 -> startChildCall(findVlan) procurve2::findVlans Enable trunking: Port is pc1:2.1 native VLAN is 8 procurve2 -> startChildCall(enablePortTrunking2) procurve2::setPortVlan(257)as ifIndexes: 98 portControl: enable -> (pc1:2.1) checking row 98 for up ... Port pc1:2.1, row 98 was down Setting pc1:2.1 (r 98) to up...succeeded. procurve2::updateOneVlan(0,0,1) as ifIndexes: 98 Enabling dual mode trunking on pc2:2.1:8 Enabling dual trunking (tagging) on pc2:2.1 8 procurve2 -> startChildCall(findVlan) procurve2::findVlans Enable trunking: Port is pc2:2.1 native VLAN is 8 procurve2 -> startChildCall(enablePortTrunking2) procurve2::setPortVlan(257)as ifIndexes: 101 portControl: enable -> (pc2:2.1) checking row 101 for up ... Port pc2:2.1, row 101 was down Setting pc2:2.1 (r 101) to up...succeeded. procurve2::updateOneVlan(0,0,1) as ifIndexes: 101 Enabling dual mode trunking on pc4:1.1:8 Enabling dual trunking (tagging) on pc4:1.1 8 procurve2 -> startChildCall(findVlan) procurve2::findVlans Enable trunking: Port is pc4:1.1 native VLAN is 8 procurve2 -> startChildCall(enablePortTrunking2) procurve2::updateOneVlan(0,0,1) as ifIndexes: 106 Enabling dual mode trunking on pc4:2.1:8 Enabling dual trunking (tagging) on pc4:2.1 8 procurve2 -> startChildCall(findVlan) procurve2::findVlans Enable trunking: Port is pc4:2.1 native VLAN is 8 procurve2 -> startChildCall(enablePortTrunking2) procurve2::updateOneVlan(0,0,1) as ifIndexes: 107 Enabling dual mode trunking on pc4:3.1:8 Enabling dual trunking (tagging) on pc4:3.1 8 procurve2 -> startChildCall(findVlan) procurve2::findVlans Enable trunking: Port is pc4:3.1 native VLAN is 8 procurve2 -> startChildCall(enablePortTrunking2) procurve2::updateOneVlan(0,0,1) as ifIndexes: 108 Fetching tarballs and RPMs (if any) ... TIMESTAMP: 10:42:42:105102 tarfiles_setup started TIMESTAMP: 10:42:42:428132 tarfiles_setup finished Updating dynamic blobs. Setting up mountpoints. TIMESTAMP: 10:42:42:486896 mountpoints started TIMESTAMP: 10:42:44:738628 mountpoints finished TIMESTAMP: 10:42:44:740361 named started Setting up named maps. TIMESTAMP: 10:42:45:231283 named finished TIMESTAMP: 10:42:45:232917 gentopofile started Generating ltmap (again) ... TIMESTAMP: 10:42:45:418914 gentopofile finished Resetting OS and rebooting. TIMESTAMP: 10:42:45:421243 launching os_setup Checking for feature OldOsSetup. TIMESTAMP: 10:42:46:11935 os_setup started Checking for feature NewOsload. Adding pc1 to type object pc pc1 - [OS 10016: emulab-ops,FEDORA15-OPENVZ-STD] Adding pc2 to type object pc pc2 - [OS 10016: emulab-ops,FEDORA15-OPENVZ-STD] Adding pc4 to type object pc pc4 - [OS 10016: emulab-ops,FEDORA15-OPENVZ-STD] Asking [pc] for volunteers [pc] returns volunteers: [Node: pc4] [Node: pc1] [Node: pc2] Lighting up nodes in parallel ... Lighting up nodes: [Node: pc4] [Node: pc1] [Node: pc2] pc4 will be reloaded with [Image 10016: emulab-ops,FEDORA15-OPENVZ-STD] Checking for feature NewOsload. osload: Using [Image 10016: emulab-ops,FEDORA15-OPENVZ-STD] for pc4 [Image 10016: emulab-ops,FEDORA15-OPENVZ-STD]: loadpart=2, loadlen=1, \ imagepath=/usr/testbed/images/FEDORA15-OPENVZ-STD.ndz, imagesize=1217396736, defosid=10016, maxloadwait=1354 osload (pc4): Changing default OS to [OS 10016: emulab-ops,FEDORA15-OPENVZ-STD] Setting up reload for pc4 (mode: Frisbee) TIMESTAMP: 10:42:47:373034 osload: database setup done osload: Issuing reboot for pc4 and then waiting ... TIMESTAMP: 10:42:47:376165 osload: event handler enabled reboot: no privs; invoking real nodereboot script! reboot: Running in pipemode (30). reboot (pc4): Attempting to reboot ... reboot (pc4): in PXEWAIT: sending wakeup command. Built 17-Jan-2013 by elabman@boss:/users/elabman/testbed/obj-real/pxe Sending event PXEWAKEUP for node pc4 192.1.242.143: SEND: query bootinfo reboot (pc4): Successful! reboot: Done. There were 0 failures. reboot (pc4): child returned 0 status. Waiting for pc4 to finish reloading Wed Feb 6 10:42:47 EST 2013 TIMESTAMP: 10:42:47:992938 osload: starting reload-done wait osload: eventhandler: pc4 => RELOADSETUP @ 1360165393 osload: eventhandler: pc4 => RELOADING @ 1360165393 got 2 events osload (pc4): still waiting; it has been 1 minute(s) osload (pc4): still waiting; it has been 2 minute(s) osload (pc4): still waiting; it has been 3 minute(s) osload (pc4): still waiting; it has been 4 minute(s) osload (pc4): still waiting; it has been 5 minute(s) osload (pc4): still waiting; it has been 6 minute(s) osload (pc4): still waiting; it has been 7 minute(s) osload (pc4): left reloading mode at Wed Feb 6 10:50:08 EST 2013 osload: Done! There were 0 failures. reload (pc4): child returned 0 status. Waiting for nodes ... pc1 has reported state ISUP Checking for feature OsloadFailNonfatal. pc1 is alive and well pc2 has reported state ISUP Checking for feature OsloadFailNonfatal. pc2 is alive and well *** os_setup: Still waiting for pc4 (SHUTDOWN) - it's been 1 minute(s). *** os_setup: Still waiting for pc4 (SHUTDOWN) - it's been 2 minute(s). *** os_setup: Still waiting for pc4 (BOOTING) - it's been 3 minute(s). pc4 has reported state ISUP Checking for feature OsloadFailNonfatal. pc4 is alive and well Asking [pc] for volunteers TIMESTAMP: 10:53:34:398383 os_setup finished Starting the event system. TIMESTAMP: 10:53:34:416968 eventsys_control started Checking for feature NewEventScheduler. /usr/testbed/sbin/event-sched -d -s localhost -k /proj/emulab-ops/exp/shared-nodes/tbdata/eventkey emulab-ops \ shared-nodes Beginning call to experiment.metadata Error occured: problem setting up certificate could not get experiment metadata Finished with call to experiment.metadata *** eventsys.proxy: Failed to start event system for emulab-ops/shared-nodes: 47835 0! *** WARNING: tbswap: *** Failed to start the event system, must restart manually. Successfully finished swap-modify for emulab-ops/shared-nodes. 10:53:37:300754 TIMESTAMP: 10:53:37:301712 tbswap modify finished (succeeded) Image rendering proceeding in background mode ... Running 'tbreport -b emulab-ops shared-nodes' cp: utimes: /usr/testbed/expinfo/emulab-ops/shared-nodes/13/pstate.DvOhzX: Operation not permitted cp: utimes: /usr/testbed/expinfo/emulab-ops/shared-nodes/13/vstate.12tonR: Operation not permitted cp: utimes: /usr/testbed/expinfo/emulab-ops/shared-nodes/13/vstate.FdQ9Eq: Operation not permitted cp: utimes: /usr/testbed/expinfo/emulab-ops/shared-nodes/13/vstate.Qk8L8S: Operation not permitted cp: utimes: /usr/testbed/expinfo/emulab-ops/shared-nodes/13: Operation not permitted Doing a commit on the previous experiment archive ... Doing a savepoint on the experiment archive ... Swap Success! }}} c. Verified that the node becomes part of the shared pool. Checked the details for the shared-node experiment which shows pc4 as one of the nodes in the shared nodes pool: [[Image(IG-EXP-1-pc4-shared.jpg)]] Additionally set up an experiment which used a VM on shared node pc4: {{{ $ readyToLogin.py -a ig-gpo IG-MON4-exp2 <...> VM's geni_status is: ready (am_status:ready) User lnevers logins to VM using: xterm -e ssh -p 31290 -i /home/lnevers/.ssh/id_rsa lnevers@pc4.instageni.gpolab.bbn.com & PC's geni_status is: ready (am_status:ready) User lnevers logins to PC using: xterm -e ssh -i /home/lnevers/.ssh/id_rsa lnevers@pc5.instageni.gpolab.bbn.com & }}} Successfully logged in the pc4 VM and ran iperf. Deleted sliver upon completion. d. Verify that no VMs are in use for the newly added shared node. Collect list resources for the InstaGENI Rack and also check the pc4 node status page. {{{ }}} Also verified the pc4 node status to make sure that no VMs are in use. e. Modified shared-nodes experiment to remove the newly added shared node. After verifying that no VMs are in used on pc4, selected the "modify experiment" link in the web interface and removed the entries relating to pc4: {{{ set vhost3 [$ns node] tb-set-node-os $vhost3 FEDORA15-OPENVZ-STD tb-set-node-sharingmode $vhost3 "shared_local" tb-fix-node $vhost3 pc4 }}} Submited the changes and the following operations took place: {{{ Doing a preswapmod on the experiment archive ... Backing up old experiment state ... 15:38:33:975849 Running 'tbprerun -e 13 shared-nodes-modify.ns' Beginning pre run for emulab-ops/shared-nodes. 15:38:34:358577 Running parser ... 15:38:34:603168 Parser done! 15:38:36:974095 Setting up static routes (if requested) ... *** WARNING: staticroutes: *** No links or lans in experiment emulab-ops/shared-nodes! Generating topomap ... Verifying parse ... Doing a pre-assign: '/usr/testbed/bin/vtopgen -p emulab-ops shared-nodes' ... Checking for feature LibVtopTest. Using test version of libvtop ... Minimum nodes = 2 Maximum nodes = 2 Writing environment strings ... Setting up additional program agent support ... Setting up additional network agent support ... Writing program agent info ... Pre run finished. 15:38:39:943787 Running 'tbswap modify -eventsys_restart -noswapout -noreconfig emulab-ops shared-nodes' Beginning swap-modify for emulab-ops/shared-nodes (13). 02/06/2013 15:38:40 TIMESTAMP: 15:38:40:231396 tbswap modify started Checking for feature SyncVlans. Checking with Admission Control ... Backing up physical state... Stopping the event system Checking for feature NewEventScheduler. Freeing failed nodes. TIMESTAMP: 15:38:41:60215 nfree started [Interface: pc4:eth1] is still tagged! But we will fix that. [Interface: pc4:eth2] is still tagged! But we will fix that. [Interface: pc4:eth3] is still tagged! But we will fix that. *** Turning off tagging for: pc4:eth1 pc4:eth2 pc4:eth3 Disabling trunking (tagging) on port pc4:1.1 ... Moving [Node: pc4] to [Experiment: emulab-ops/reloadpending] TIMESTAMP: 15:38:42:395301 nfree finished Mapping to physical reality ... TIMESTAMP: 15:38:42:591955 mapper wrapper started Checking for feature OldMapper. Checking for feature NewAssign. Checking for feature ControlNetVlans. Checking for feature LibVtopTest. Using test version of libvtop ... Starting the new and improved mapper wrapper. Minimum nodes = 2 Maximum nodes = 2 Reserved pnodes = 2 Clearing physical state before updating. Mapper loop 1 ptopargs: '-p emulab-ops -e shared-nodes ' assign command: 'assign -P -f text emulab-ops-shared-nodes-43514.ptop emulab-ops-shared-nodes-43514.vtop' Reading assign results. [Node: pc1] already reserved in holding reservation. [Node: pc2] already reserved in holding reservation. pc1 pc2 TIMESTAMP: 15:38:46:203759 mapper wrapper finished Mapped to physical reality! Synchronizing VLANs. getTrunksForVlan: 8: procurve2 mapVlansToSwitches: procurve2 getExperimentTrunksForVlan: 8: procurve2 mapStaleVlansToSwitches: procurve2 procurve2 -> startChildCall(FlipDebug) Experiment vlans: 8 Trunk Ports: pc1:3.1 pc2:3.1 pc2:2.1 pc1:2.1 pc2:1.1 pc1:1.1 Existing vlans: 8 Stale vlans: Existing Trunk Ports: pc2:3.1 pc2:1.1 pc1:1.1 snmpit_stack::findVlan( 8 ) snmpit_stack::findVlans( 8 ) procurve2 -> startChildCall(findVlans) procurve2::findVlans snmpit_stack::reapCall(findVlans) stack::findVlans calling procurve2 snmpit_stack::findVlans( ) procurve2 -> startChildCall(findVlans) procurve2::findVlans snmpit_stack::reapCall(findVlans) stack::findVlans calling procurve2 getExperimentTrunksForVlan: 8: procurve2 Removing stale ports from vlan 8: pc4:3.1 pc4:2.1 pc4:1.1 snmpit_stack::findVlans( 8 ) procurve2 -> startChildCall(findVlans) procurve2::findVlans snmpit_stack::reapCall(findVlans) stack::findVlans calling procurve2 procurve2 -> startChildCall(findVlans) procurve2::findVlans Removing ports on procurve2 from VLAN 8 (257) procurve2 -> startChildCall(removeSomePortsFromVlan) procurve2::updateOneVlan(0,1,257) as ifIndexes: 107 procurve2::updateOneVlan(0,0,1) as ifIndexes: 107 procurve2::updateOneVlan(0,0,257) as ifIndexes: 107 procurve2:set had error number 12 and had error string (inconsistentValue) The value set is not currently \ acceptable procurve2:set ... will try again procurve2:set had error number 12 and had error string (inconsistentValue) The value set is not currently \ acceptable procurve2:set ... will try again procurve2:set .. giving up vlists failed procurve2::updateOneVlan(0,1,1) as ifIndexes: 107 procurve2::updateOneVlan(0,0,257) as ifIndexes: 107 disabling port 106 from vlan 257 disabling port 108 from vlan 257 snmpit_stack::findVlan( 8 ) snmpit_stack::findVlans( 8 ) procurve2 -> startChildCall(findVlans) procurve2::findVlans snmpit_stack::reapCall(findVlans) stack::findVlans calling procurve2 snmpit_stack::findVlans( ) procurve2 -> startChildCall(findVlans) procurve2::findVlans snmpit_stack::reapCall(findVlans) stack::findVlans calling procurve2 getTrunksForVlan: 8: procurve2 RecordVLanModification: [VLan 8:257: emulab-ops/shared-nodes/sharedlan] - pc4:eth3 pc4:eth2 pc4:eth1 RecordVLanModification: [VLan 8:257: emulab-ops/shared-nodes/sharedlan] - pc4:eth3 pc4:eth2 pc1:eth3 pc2:eth3 \ pc1:eth1 pc1:eth2 pc2:eth2 pc4:eth1 pc2:eth1 pc4:eth3 pc4:eth2 pc4:eth1 RecordVLanModification (after): [VLan 8:257: emulab-ops/shared-nodes/sharedlan] - pc4:eth3 pc4:eth2 pc1:eth3 \ pc2:eth3 pc1:eth1 pc1:eth2 pc2:eth2 pc4:eth1 pc2:eth1 Enabling dual mode trunking on pc1:2.1:8 Enabling dual trunking (tagging) on pc1:2.1 8 procurve2 -> startChildCall(findVlan) procurve2::findVlans Enable trunking: Port is pc1:2.1 native VLAN is 8 procurve2 -> startChildCall(enablePortTrunking2) procurve2::setPortVlan(257)as ifIndexes: 98 portControl: enable -> (pc1:2.1) checking row 98 for up ... Port pc1:2.1, row 98 was down Setting pc1:2.1 (r 98) to up...succeeded. procurve2::updateOneVlan(0,0,1) as ifIndexes: 98 Enabling dual mode trunking on pc1:3.1:8 Enabling dual trunking (tagging) on pc1:3.1 8 procurve2 -> startChildCall(findVlan) procurve2::findVlans Enable trunking: Port is pc1:3.1 native VLAN is 8 procurve2 -> startChildCall(enablePortTrunking2) procurve2::setPortVlan(257)as ifIndexes: 99 portControl: enable -> (pc1:3.1) checking row 99 for up ... Port pc1:3.1, row 99 was down Setting pc1:3.1 (r 99) to up...succeeded. procurve2::updateOneVlan(0,0,1) as ifIndexes: 99 Enabling dual mode trunking on pc2:2.1:8 Enabling dual trunking (tagging) on pc2:2.1 8 procurve2 -> startChildCall(findVlan) procurve2::findVlans Enable trunking: Port is pc2:2.1 native VLAN is 8 procurve2 -> startChildCall(enablePortTrunking2) procurve2::setPortVlan(257)as ifIndexes: 101 portControl: enable -> (pc2:2.1) checking row 101 for up ... Port pc2:2.1, row 101 was down Setting pc2:2.1 (r 101) to up...succeeded. procurve2::updateOneVlan(0,0,1) as ifIndexes: 101 Fetching tarballs and RPMs (if any) ... TIMESTAMP: 15:38:51:675558 tarfiles_setup started TIMESTAMP: 15:38:52:2669 tarfiles_setup finished Updating dynamic blobs. Setting up mountpoints. TIMESTAMP: 15:38:52:466517 mountpoints started TIMESTAMP: 15:38:54:828490 mountpoints finished TIMESTAMP: 15:38:54:830269 named started Setting up named maps. TIMESTAMP: 15:38:55:495790 named finished TIMESTAMP: 15:38:55:514904 gentopofile started Generating ltmap (again) ... TIMESTAMP: 15:38:55:702383 gentopofile finished Resetting OS and rebooting. TIMESTAMP: 15:38:55:704739 launching os_setup Checking for feature OldOsSetup. TIMESTAMP: 15:38:56:260025 os_setup started Checking for feature NewOsload. Adding pc1 to type object pc pc1 - [OS 10016: emulab-ops,FEDORA15-OPENVZ-STD] Adding pc2 to type object pc pc2 - [OS 10016: emulab-ops,FEDORA15-OPENVZ-STD] Asking [pc] for volunteers [pc] returns volunteers: [Node: pc1] [Node: pc2] Lighting up nodes in parallel ... Lighting up nodes: [Node: pc1] [Node: pc2] Waiting for nodes ... pc1 has reported state ISUP Checking for feature OsloadFailNonfatal. pc1 is alive and well pc2 has reported state ISUP Checking for feature OsloadFailNonfatal. pc2 is alive and well Asking [pc] for volunteers TIMESTAMP: 15:39:02:363058 os_setup finished Starting the event system. TIMESTAMP: 15:39:02:378643 eventsys_control started Checking for feature NewEventScheduler. /usr/testbed/sbin/event-sched -d -s localhost -k /proj/emulab-ops/exp/shared-nodes/tbdata/eventkey emulab-ops \ shared-nodes Beginning call to experiment.metadata Error occured: problem setting up certificate could not get experiment metadata Finished with call to experiment.metadata *** eventsys.proxy: Failed to start event system for emulab-ops/shared-nodes: 48789 0! *** WARNING: tbswap: *** Failed to start the event system, must restart manually. Successfully finished swap-modify for emulab-ops/shared-nodes. 15:39:05:219620 TIMESTAMP: 15:39:05:220596 tbswap modify finished (succeeded) Image rendering proceeding in background mode ... Running 'tbreport -b emulab-ops shared-nodes' cp: utimes: /usr/testbed/expinfo/emulab-ops/shared-nodes/13/vstate.12tonR: Operation not permitted cp: utimes: /usr/testbed/expinfo/emulab-ops/shared-nodes/13/pstate.DvOhzX: Operation not permitted cp: utimes: /usr/testbed/expinfo/emulab-ops/shared-nodes/13/vstate.Qk8L8S: Operation not permitted cp: utimes: /usr/testbed/expinfo/emulab-ops/shared-nodes/13/vstate.FdQ9Eq: Operation not permitted cp: utimes: /usr/testbed/expinfo/emulab-ops/shared-nodes/13: Operation not permitted Doing a commit on the previous experiment archive ... Doing a savepoint on the experiment archive ... Swap Success! }}} f. Verify that the node is no longer part of the shared pool: Used listresources to show that pc4 was no longer a shared nodes and once again a dedicated node and then set up an experiment that used 2 raw-pcs including pc4. {{{ $ readyToLogin.py pc4pc5 -a ig-gpo ./insta-pc4-pc5.rspec <...> PC5's geni_status is: ready (am_status:ready) User lnevers logins to PC5 using: xterm -e ssh -i /home/lnevers/.ssh/id_rsa lnevers@pc5.instageni.gpolab.bbn.com & PC4's geni_status is: ready (am_status:ready) User lnevers logins to PC4 using: xterm -e ssh -i /home/lnevers/.ssh/id_rsa lnevers@pc4.instageni.gpolab.bbn.com & }}}