13 node linear sliver fails with geni_status: 'failed'

[lnevers@bbn.com]

Since linear topologies work, I attempted to request all available resources at the ExoGENI rack via the BBN SM. The slice name used is "18vmlinear-local", although only 13 VM were found to be available (cannot tell before the request).

1. Tried to request 18 VM linear topology which failed with:

    Request id: 8de94436-170e-44ab-86d5-7f99fe88597c
    Embedding workflow ERROR: 1:Insufficient resources or Unknown domain: 
    http://geni-orca.renci.org/owl/bbnvmsite.rdf#bbnvmsite/Domain/vm:vm:13! 
    Requested: vm:18 
   

2. Then requested 13 nodes in a linear sliver which was successful:

    $ ./src/omni.py -a exobbn createsliver 18vmlinear-local ./exorspec/exo-13vm-linear.rspec
   INFO:omni:Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+18vmlinear-local expires on 2012-06-13 03:50:07 UTC
   INFO:omni:Substituting AM nickname exobbn with URL https://bbn-hn.exogeni.net:11443/orca/xmlrpc, URN unspecified_AM_U
   RN
   INFO:omni:Substituting AM nickname exobbn with URL https://bbn-hn.exogeni.net:11443/orca/xmlrpc, URN unspecified_AM_U
   RN
   INFO:omni:Creating sliver(s) from rspec file ./exorspec/exo-13vm-linear.rspec for slice urn:publicid:IDN+pgeni.gpolab
   .bbn.com+slice+18vmlinear-local
   INFO:omni:Asked https://bbn-hn.exogeni.net:11443/orca/xmlrpc to reserve resources. Result:
   INFO:omni:<?xml version="1.0" ?>
   INFO:omni:<!-- Reserved resources for:
   	Slice: 18vmlinear-local
   	At AM:
   	URL: https://bbn-hn.exogeni.net:11443/orca/xmlrpc
    -->
   INFO:omni:<rspec type="manifest" xmlns="http://www.protogeni.net/resources/rspec/2"
   ... <lines> deleted...
   
   

   Note: Attaching sliver manifest RSpec.

3. The sliver status was checked immediately after the sliver creation and it showed the error:

   INFO:omni:Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+18vmlinear-local expires on 2012-06-13 03:50:07 UTC
   INFO:omni:Substituting AM nickname exobbn with URL https://bbn-hn.exogeni.net:11443/orca/xmlrpc, 
   URN unspecified_AM_URN
   INFO:omni:Status of Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+18vmlinear-local:
   INFO:omni:Sliver status for Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+18vmlinear-local at AM URL https://bbn-
   hn.exogeni.net:11443/orca/xmlrpc
   INFO:omni:{'geni_resources': '',
    'geni_status': 'failed',
    'geni_urn': 'urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+18vmlinear-local'}
   
   

Also attaching the sliver status. Let sliver running.

[ahelsing@bbn.com]

I notice a couple things about this manifest

• many links are missing the vlantag element
• many nodes have empty services tag

I suspect those two things are symptoms of whatever is failing.

[ibaldin@renci.org]

Head node at BBN ran out of file descriptors. We've cleaned up. We do not know what has caused it - there were some old processes hanging around from previous redeployments that have now been terminated and could've been the cause.

[lnevers@bbn.com]

Delete the earlier failed sliver and then proceeded to execute the following:

1. Request 18 VM linear topology, to determine available VMs (13 VMs)
2. Created a sliver with 13 VM linear (slice=Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+allVMlinear)
3. Sliver status returned:

      INFO:omni:Sliver status for Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+allVMlinear 
      at AM URL https://bbn-hn.exogeni.net:11443/orca/xmlrpc
      INFO:omni:{'geni_resources': '',
      'geni_status': 'failed',
      'geni_urn': 'urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+allVMlinear'}
   

Attaching sliver manifest to ticket.

[lnevers@bbn.com]

Deleted the previous failed sliver, then created a 13VMs sliver named "13vmlinear".

The create sliver was successful. Checked the sliver status which was "configuring" for a few minutes and then a failure was reported:

INFO:omni:Sliver status for Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+13vmlinear 
at AM URL https://bbn-hn.exogeni.net:11443/orca/xmlrpc
INFO:omni:{'geni_resources': '',
 'geni_status': 'failed',
 'geni_urn': 'urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+13vmlinear'}

In previous attempt the failure was immediate, while now it takes a few minutes..some progress.

Attaching the sliver manifest.

[ibaldin@renci.org]

Give it another try. You had an old half-built slice hanging around with 5 vms. I cleaned it all up. PLEASE BE SURE TO CLOSE THE SLICE EVEN IF IT FAILS, SINCE OTHERWISE SLIVERS OF IT THAT DID SUCCEED (VLANs or VMs) WILL REMAIN ACTIVE UNTIL THEY EXPIRE.

[lnevers@bbn.com]

I usually delete all slivers after failures, unless I am leaving them up for investigation.

I did set up a 5 node sliver yesterday which resulted in the "Unknown slice URN" error . At the time I did not delete the sliver because because the aggregate did not know the slice urn.

BTW, I just submitted a sliver request to the BBN Rack SM and it has been "configuring for approx 8 minute. I will follow up with the results.

[ibaldin@renci.org]

It is coming up.

[ibaldin@renci.org]

I take that back. One vlan failed again (which was the failure last time too).

[ibaldin@renci.org]

This is a problem with floodlight not starting reliably. Please try again - I put in a temporary fix.

[lnevers@bbn.com]

First deleted the previous sliver, then created a new slice named "allvmlinear". Created a sliver requesting all VMS, which failed as follows:

$ ./src/omni.py -a exobbn createsliver allvmlinear ./exorspec/exo-18vm-linear.rspec 

INFO:omni:Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+allvmlinear expires within 1 day on 2012-06-13 14:45:38 UTC
INFO:omni:Substituting AM nickname exobbn with URL https://bbn-hn.exogeni.net:11443/orca/xmlrpc, URN unspecified_AM_URN
INFO:omni:Substituting AM nickname exobbn with URL https://bbn-hn.exogeni.net:11443/orca/xmlrpc, URN unspecified_AM_URN
INFO:omni:Creating sliver(s) from rspec file ./exorspec/exo-18vm-linear.rspec for 
slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+allvmlinear
INFO:omni:Asked https://bbn-hn.exogeni.net:11443/orca/xmlrpc to reserve resources. Result:
INFO:omni:<!-- Reserved resources for:
	Slice: allvmlinear
	At AM:
	URL: https://bbn-hn.exogeni.net:11443/orca/xmlrpc
 -->
INFO:omni: ------------------------------------------------------------
INFO:omni: Completed createsliver:

  Options as run:
		aggregate: exobbn
		framework: pgeni
		native: True

  Args: createsliver allvmlinear ./exorspec/exo-18vm-linear.rspec

  Result Summary: Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+allvmlinear 
expires within 1 day(s) on 2012-06-13 14:45:38 UTC
Asked https://bbn-hn.exogeni.net:11443/orca/xmlrpc to reserve resources. No manifest 
Rspec returned. Credendial Exception: javax.security.auth.login.CredentialException: 
No credential was found with appropriate privileges. 

[ibaldin@renci.org]

This is the one I said earlier - it failed because of one VLAN not coming up. I closed all reservations in this slice, please try again.

[ibaldin@renci.org]

everything except one vm is up. openstack is not letting it come up for some reason. 30 minutes after the start the timeout will kick in and orca will kill this vm and try it again.

[lnevers@bbn.com]

Thanks Ilia, I had been watching the sliver status for the past 16 minutes. Should I let it run until the timeout?

[lnevers@bbn.com]

I am able to login and use the 17 VMs in the slive manifest. I assume the 18th will be added when/if successfully started.

[lnevers@bbn.com]

Here are the results:

• Sliver created at 14:47
• geni_status remained "configuring" until 15:17
• geni_status became 'ready' at 15:18:13
• geni_status became 'failed' at 15:18:35

After the geni_status 'failed' is reported, I can still use the 17 VMs.

What would you like for me to do next, Ilia?

[ibaldin@renci.org]

OpenStack? failed to bring up the last vm - I believe it is a transient OpenStack? failure (we have seen it in the past). We can't do anything about them other than wait for a newer version of OpenStack?. Please delete and try again.

[ibaldin@renci.org]

The slice status is determined as a conjunction of all reservation statuses. If any of reservations fail, the slice is flagged as failed.

[ibaldin@renci.org]

Looks like it worked this time.

As I said - transient OpenStack? behavior is a problem and the only solution is to try a few times (if you are sure the image is good).

[lnevers@bbn.com]

YES!!!! I have a sliver with 18 VMs in a linear topology from the local rack SM.

I will re-run this test a few more times and then update the ticket (or close if no issue).

[ibaldin@renci.org]

Just to summarize:

1) Original failure caused by head node running out of file descriptors. Not clear what caused it, we will keep on watching for it. There were traces of old processes that should have been terminated during last redeploy that could've been the cause.

2) There were vlan failiures caused by unreliable floodlight startup. I deployed a small fix (increased timeout) and committed a change to orca code

3) There was transient OpenStack? vm failure. The only fix is trying again (and waiting for a better OpenStack? version).

[lnevers@bbn.com]

Here are results for 3 attempts, where each sliver requested 18 VMs from BBN rack SM:

• Created sliver lnexo1, sliver ready in 3 minutes, able to login to each host - OK
• Deleted sliver lnexo1.
• Created sliver lnexo2, sliver failed in 15 minutes - FAIL
• Deleted sliver lnexo2.
• Created sliver lnexo3, sliver was in "configuring" state for over 36 minutes, deleted the sliver. Should this have timeout out in 30 minutes?

[ibaldin@renci.org]

I can't tell now. I don't see any new failures in the logs, so I'm not sure what happened. This will have to wait until the rack is back tomorrow.

[lnevers@bbn.com]

Re-ran this test by requesting a 13 node linear topology to the BBN SM which generated this failure:

Asked https://bbn-hn.exogeni.net:11443/orca/xmlrpc to reserve resources. No manifest Rspec returned. 
Error encountered converting manifest to RSpec: ERROR encountered while converting the manifest 
due to null pointer, inspect the logs 

Note: This converter problem had been fixed on Tuesday, but now shows up for this sliver and not others.

After the sliver named 13linear is created there are inconsistencies.

The sliverstatus shows 'geni_status'='ready', and the 13 VMs allocated

$ egrep VM ./13linear-sliverstatus-bbn-hn-exogeni-net-11443-orca.json 
'geni_urn': 'urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+196dab99-e4dc-41d0-8a1f-64048367037e#VM-1',
'geni_urn': 'urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+196dab99-e4dc-41d0-8a1f-64048367037e#VM-3',
'geni_urn': 'urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+196dab99-e4dc-41d0-8a1f-64048367037e#VM-2',
'geni_urn': 'urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+196dab99-e4dc-41d0-8a1f-64048367037e#VM-8',
'geni_urn': 'urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+196dab99-e4dc-41d0-8a1f-64048367037e#VM-9',
'geni_urn': 'urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+196dab99-e4dc-41d0-8a1f-64048367037e#VM-4',
'geni_urn': 'urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+196dab99-e4dc-41d0-8a1f-64048367037e#VM-5',
'geni_urn': 'urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+196dab99-e4dc-41d0-8a1f-64048367037e#VM-6',
'geni_urn': 'urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+196dab99-e4dc-41d0-8a1f-64048367037e#VM-7',
'geni_urn': 'urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+196dab99-e4dc-41d0-8a1f-64048367037e#VM-10',
'geni_urn': 'urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+196dab99-e4dc-41d0-8a1f-64048367037e#VM-11',
'geni_urn': 'urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+196dab99-e4dc-41d0-8a1f-64048367037e#VM-12',
'geni_urn': 'urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+196dab99-e4dc-41d0-8a1f-64048367037e#VM-13',

But the listresources only show 11 hosts (note the listresources was collected 10 minutes after the geni_status was ready) :

$ egrep hostname 13vmlinear-rspec-bbn-hn-exogeni-net-11443-orca.xml    
<login authentication="ssh-keys" hostname="192.1.242.28" port="22" username="root"/>      
<login authentication="ssh-keys" hostname="192.1.242.44" port="22" username="root"/>      
<login authentication="ssh-keys" hostname="192.1.242.46" port="22" username="root"/>      
<login authentication="ssh-keys" hostname="192.1.242.38" port="22" username="root"/>      
<login authentication="ssh-keys" hostname="192.1.242.29" port="22" username="root"/>      
<login authentication="ssh-keys" hostname="192.1.242.45" port="22" username="root"/>      
<login authentication="ssh-keys" hostname="192.1.242.37" port="22" username="root"/>      
<login authentication="ssh-keys" hostname="192.1.242.22" port="22" username="root"/>      
<login authentication="ssh-keys" hostname="192.1.242.26" port="22" username="root"/>      
<login authentication="ssh-keys" hostname="192.1.242.42" port="22" username="root"/>      
<login authentication="ssh-keys" hostname="192.1.242.27" port="22" username="root"/>      

Attaching the Request RSpec, the listresources (manifest) RSpec, and the sliverstatus output.

[lnevers@bbn.com]

I have tried an 18 node linear topology sliver from the BBN SM and it fails. The failed sliver status and urn:

 'geni_status': 'failed',
 'geni_urn': 'urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+lnexo'}

I realize this original ticket was written for a 13 VM linear topology sliver, and the current issue may be unrelated. If needed, I can write a new ticket, please let me know.

Also, I am including a capture of email exchanges outside the ticket about the original failure in this ticket:

On 7/5/12 4:32 PM, Ilia Baldine wrote:

Don't kill this slice please

On 7/5/12 4:49 PM, Ilia Baldine wrote:

Try again on the same slice. I've updated the converter.

On 7/5/12 5:13 PM, Ilia Baldine wrote:

I see a slice (13-node) created at 5:05 that is active. Anything else?

On Jul 5, 2012, at 5:17 PM, Luisa Nevers wrote:

The request just returned a failure:

Result Summary: Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+13linear expires on 2012-07-06 21:05:53 UTC Asked ​https://bbn-hn.exogeni.net:11443/orca/xmlrpc to reserve resources. No manifest Rspec returned. Error encountered converting manifest to RSpec: Unable to contact/use any of specified NDL-RSPEC converters at

​http://geni.renci.org:12080/ndl-conversion/,http://bbn-hn.exogeni.net:14080/ndl-conversion/

On 7/5/12 5:18 PM, Ilia Baldine wrote:

They both seem to have run out of memory. Try requesting manifest now. I restarted them. I may need to tweak the amount of virtual memory Java uses for conversion.

On 7/5/12 5:20 PM, Ilia Baldine wrote:

I was talking about the converters. The Java containers running them have run out of memory.

On 7/5/12 5:30 PM, Ilia Baldine wrote:

Too many moving pieces. I will tweak the converter settings tonight to increase the memory available to them to make sure this doesn't happen.

On Jul 5, 2012, at 5:29 PM, Luisa Nevers wrote:

It now works, I request 13 and get 13 in the listresources.

On 7/5/12 5:47 PM, Ilia Baldine wrote:

I've added parameters to tomcat start scripts on both converters that should help with this. I've also restarted them.

[lnevers@bbn.com]

Replying to lnevers@bbn.com:

I have tried an 18 node linear topology sliver from the BBN SM and it fails. The failed sliver status and urn:

 'geni_status': 'failed',
 'geni_urn': 'urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+lnexo'}

I had not noticed the following error because the output had scrolled of the screen:

                    {'geni_error': 'Reservation 7b3f4c19-46a8-4705-aa08-ab9fac5625c2 (Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+lnexo) is in state [Failed,None], err=resources failed to join: Error during join for unit: 3E2D1B98 [1]: unable to create instance: exit code 1, \n',
                     'geni_status': 'Failed',

[ibaldin@renci.org]

This is a transient OpenStack? error in one of the VMs. All I can say is 'try again please'. We hope the newer version of OpenStack? proves more reliable.

[ibaldin@renci.org]

See ​https://geni-orca.renci.org/trac/wiki/orca-errors

[lnevers@bbn.com]

Ilia, I took a quick look and at the new page and have some feedback:

1. The text "Errors from the first category" and "Errors from the second category" should mention the category name (transient/persistent), as I scrolled down to read the second section, I no longer recalled the order they were listed in at the top of the page.

2. There is no mapping between the types of errors (transient/persistent) and the phases (embedding/allocation/provisioning). The page tells me about the phases that occur while creating a slice, but does not relate the phases (embedding/allocation/provisioning) to the types of errors (transient/persistent).

3. It would be helpful to show an example for the persistent errors. We know what the errors look like and it makes it easier for the experimenter searching for help when they run into the problem.

[lnevers@bbn.com]

2nd attempt, same failure:

{'geni_error': 'Reservation b27bb316-c041-4077-a243-0ef059660483 (Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+lnexo) is in state [Failed,None], err=resources failed to join: Error during join for unit: 395A6DA3 [1]: unable to create instance: exit code 1, \n',
'geni_status': 'Failed',
'geni_urn': 'urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+fd120d4d-446c-4e07-a684-7ff7be1102c5#host-10',

3rd attempt, worked!!!

[lnevers@bbn.com]

Attempted to re-ran the 13 node linear scenario and it failed. First a capture of the createsliver:

$ omni.py createsliver -a eg-sm 13linear ./exorspec/exo-13vm-linear2.rspec 
INFO:omni:Loading config file /home/lnevers/.gcf/omni_config
INFO:omni:Using control framework pg
INFO:omni:Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+13linear expires on 2012-12-05 21:41:24 UTC
INFO:omni:Substituting AM nickname eg-sm with URL https://geni.renci.org:11443/orca/xmlrpc, URN unspecified_AM_URN
INFO:omni:Substituting AM nickname eg-sm with URL https://geni.renci.org:11443/orca/xmlrpc, URN unspecified_AM_URN
INFO:omni:Creating sliver(s) from rspec file ./exorspec/exo-13vm-linear2.rspec for slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+13linear
INFO:omni:Got return from CreateSliver for slice 13linear at https://geni.renci.org:11443/orca/xmlrpc:
INFO:omni:<?xml version="1.0" ?>
INFO:omni:  <!-- Reserved resources for:
	Slice: 13linear
	at AM:
	URN: unspecified_AM_URN
	URL: https://geni.renci.org:11443/orca/xmlrpc
 -->
INFO:omni:  <rspec type="manifest" xmlns="http://www.geni.net/resources/rspec/3" xmlns:ns2="http://hpn.east.isi.edu/rspec/ext/stitch/0.1/" xmlns:ns3="http://groups.geni.net/exogeni/attachment/wiki/RspecExtensions/slice-info/1" xmlns:ns4="http://groups.geni.net/exogeni/attachment/wiki/RspecExtensions/sliver-info/1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.geni.net/resources/rspec/3 http://www.geni.net/resources/rspec/3/manifest.xsd http://hpn.east.isi.edu/rspec/ext/stitch/0.1/ http://hpn.east.isi.edu/rspec/ext/stitch/0.1/stitch-schema.xsd http://groups.geni.net/exogeni/attachment/wiki/RspecExtensions/slice-info/1 http://groups.geni.net/exogeni/attachment/wiki/RspecExtensions/slice-info/1/slice_info.xsd?format=raw http://groups.geni.net/exogeni/attachment/wiki/RspecExtensions/sliver-info/1 http://groups.geni.net/exogeni/attachment/wiki/RspecExtensions/sliver-info/1/sliver_info.xsd?format=raw">  
      <node client_id="host-10" component_id="urn:publicid:IDN+exogeni.net:bbnvmsite+node+orca-vm-cloud" component_manager_id="urn:publicid:IDN+exogeni.net:bbnvmsite+authority+am" component_name="orca-vm-cloud" exclusive="true" sliver_id="urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#host-10">    
            <sliver_type name="m1.small">      
                  <disk_image name="http://geni-images.renci.org/images/standard/debian/deb6-neuca-v1.0.5.xml" version="104ea3d824906f0e13cebb89b14df232290553b1 "/>      
            </sliver_type>    
            <services/>    
            <interface client_id="host-10:if1">      
                  <ip address="172.16.10.1" netmask="255.255.255.0" type="ipv4"/>      
            </interface>    
            <interface client_id="host-10:if0">      
                  <ip address="172.16.9.2" netmask="255.255.255.0" type="ipv4"/>      
            </interface>    
            <ns4:geni_sliver_info creation_time="2012-12-04T15:43:55.044-05:00" creator_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers" expiration_time="2012-12-18T14:43:55.044-05:00" start_time="2012-12-04T15:43:55.044-05:00" state="Nascent"/>    
      </node>  
      <node client_id="host-4" component_id="urn:publicid:IDN+exogeni.net:bbnvmsite+node+orca-vm-cloud" component_manager_id="urn:publicid:IDN+exogeni.net:bbnvmsite+authority+am" component_name="orca-vm-cloud" exclusive="true" sliver_id="urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#host-4">    
            <sliver_type name="m1.small">      
                  <disk_image name="http://geni-images.renci.org/images/standard/debian/deb6-neuca-v1.0.5.xml" version="104ea3d824906f0e13cebb89b14df232290553b1 "/>      
            </sliver_type>    
            <services/>    
            <interface client_id="host-4:if1">      
                  <ip address="172.16.4.1" netmask="255.255.255.0" type="ipv4"/>      
            </interface>    
            <interface client_id="host-4:if0">      
                  <ip address="172.16.3.2" netmask="255.255.255.0" type="ipv4"/>      
            </interface>    
            <ns4:geni_sliver_info creation_time="2012-12-04T15:43:55.044-05:00" creator_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers" expiration_time="2012-12-18T14:43:55.044-05:00" start_time="2012-12-04T15:43:55.044-05:00" state="Nascent"/>    
      </node>  
      <node client_id="host-2" component_id="urn:publicid:IDN+exogeni.net:bbnvmsite+node+orca-vm-cloud" component_manager_id="urn:publicid:IDN+exogeni.net:bbnvmsite+authority+am" component_name="orca-vm-cloud" exclusive="true" sliver_id="urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#host-2">    
            <sliver_type name="m1.small">      
                  <disk_image name="http://geni-images.renci.org/images/standard/debian/deb6-neuca-v1.0.5.xml" version="104ea3d824906f0e13cebb89b14df232290553b1 "/>      
            </sliver_type>    
            <services/>    
            <interface client_id="host-2:if0">      
                  <ip address="172.16.1.2" netmask="255.255.255.0" type="ipv4"/>      
            </interface>    
            <interface client_id="host-2:if1">      
                  <ip address="172.16.2.1" netmask="255.255.255.0" type="ipv4"/>      
            </interface>    
            <ns4:geni_sliver_info creation_time="2012-12-04T15:43:55.044-05:00" creator_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers" expiration_time="2012-12-18T14:43:55.044-05:00" start_time="2012-12-04T15:43:55.044-05:00" state="Nascent"/>    
      </node>  
      <node client_id="host-9" component_id="urn:publicid:IDN+exogeni.net:bbnvmsite+node+orca-vm-cloud" component_manager_id="urn:publicid:IDN+exogeni.net:bbnvmsite+authority+am" component_name="orca-vm-cloud" exclusive="true" sliver_id="urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#host-9">    
            <sliver_type name="m1.small">      
                  <disk_image name="http://geni-images.renci.org/images/standard/debian/deb6-neuca-v1.0.5.xml" version="104ea3d824906f0e13cebb89b14df232290553b1 "/>      
            </sliver_type>    
            <services/>    
            <interface client_id="host-9:if1">      
                  <ip address="172.16.9.1" netmask="255.255.255.0" type="ipv4"/>      
            </interface>    
            <interface client_id="host-9:if0">      
                  <ip address="172.16.8.2" netmask="255.255.255.0" type="ipv4"/>      
            </interface>    
            <ns4:geni_sliver_info creation_time="2012-12-04T15:43:55.044-05:00" creator_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers" expiration_time="2012-12-18T14:43:55.044-05:00" start_time="2012-12-04T15:43:55.044-05:00" state="Nascent"/>    
      </node>  
      <node client_id="host-13" component_id="urn:publicid:IDN+exogeni.net:bbnvmsite+node+orca-vm-cloud" component_manager_id="urn:publicid:IDN+exogeni.net:bbnvmsite+authority+am" component_name="orca-vm-cloud" exclusive="true" sliver_id="urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#host-13">    
            <sliver_type name="m1.small">      
                  <disk_image name="http://geni-images.renci.org/images/standard/debian/deb6-neuca-v1.0.5.xml" version="104ea3d824906f0e13cebb89b14df232290553b1 "/>      
            </sliver_type>    
            <services/>    
            <interface client_id="host-13:if0">      
                  <ip address="172.16.12.2" netmask="255.255.255.0" type="ipv4"/>      
            </interface>    
            <ns4:geni_sliver_info creation_time="2012-12-04T15:43:55.044-05:00" creator_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers" expiration_time="2012-12-18T14:43:55.044-05:00" start_time="2012-12-04T15:43:55.044-05:00" state="Nascent"/>    
      </node>  
      <node client_id="host-7" component_id="urn:publicid:IDN+exogeni.net:bbnvmsite+node+orca-vm-cloud" component_manager_id="urn:publicid:IDN+exogeni.net:bbnvmsite+authority+am" component_name="orca-vm-cloud" exclusive="true" sliver_id="urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#host-7">    
            <sliver_type name="m1.small">      
                  <disk_image name="http://geni-images.renci.org/images/standard/debian/deb6-neuca-v1.0.5.xml" version="104ea3d824906f0e13cebb89b14df232290553b1 "/>      
            </sliver_type>    
            <services/>    
            <interface client_id="host-7:if0">      
                  <ip address="172.16.6.2" netmask="255.255.255.0" type="ipv4"/>      
            </interface>    
            <interface client_id="host-7:if1">      
                  <ip address="172.16.7.1" netmask="255.255.255.0" type="ipv4"/>      
            </interface>    
            <ns4:geni_sliver_info creation_time="2012-12-04T15:43:55.044-05:00" creator_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers" expiration_time="2012-12-18T14:43:55.044-05:00" start_time="2012-12-04T15:43:55.044-05:00" state="Nascent"/>    
      </node>  
      <node client_id="host-11" component_id="urn:publicid:IDN+exogeni.net:bbnvmsite+node+orca-vm-cloud" component_manager_id="urn:publicid:IDN+exogeni.net:bbnvmsite+authority+am" component_name="orca-vm-cloud" exclusive="true" sliver_id="urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#host-11">    
            <sliver_type name="m1.small">      
                  <disk_image name="http://geni-images.renci.org/images/standard/debian/deb6-neuca-v1.0.5.xml" version="104ea3d824906f0e13cebb89b14df232290553b1 "/>      
            </sliver_type>    
            <services/>    
            <interface client_id="host-11:if0">      
                  <ip address="172.16.10.2" netmask="255.255.255.0" type="ipv4"/>      
            </interface>    
            <interface client_id="host-11:if1">      
                  <ip address="172.16.11.1" netmask="255.255.255.0" type="ipv4"/>      
            </interface>    
            <ns4:geni_sliver_info creation_time="2012-12-04T15:43:55.044-05:00" creator_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers" expiration_time="2012-12-18T14:43:55.044-05:00" start_time="2012-12-04T15:43:55.044-05:00" state="Nascent"/>    
      </node>  
      <node client_id="host-12" component_id="urn:publicid:IDN+exogeni.net:bbnvmsite+node+orca-vm-cloud" component_manager_id="urn:publicid:IDN+exogeni.net:bbnvmsite+authority+am" component_name="orca-vm-cloud" exclusive="true" sliver_id="urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#host-12">    
            <sliver_type name="m1.small">      
                  <disk_image name="http://geni-images.renci.org/images/standard/debian/deb6-neuca-v1.0.5.xml" version="104ea3d824906f0e13cebb89b14df232290553b1 "/>      
            </sliver_type>    
            <services/>    
            <interface client_id="host-12:if0">      
                  <ip address="172.16.11.2" netmask="255.255.255.0" type="ipv4"/>      
            </interface>    
            <interface client_id="host-12:if1">      
                  <ip address="172.16.12.1" netmask="255.255.255.0" type="ipv4"/>      
            </interface>    
            <ns4:geni_sliver_info creation_time="2012-12-04T15:43:55.044-05:00" creator_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers" expiration_time="2012-12-18T14:43:55.044-05:00" start_time="2012-12-04T15:43:55.044-05:00" state="Nascent"/>    
      </node>  
      <node client_id="host-1" component_id="urn:publicid:IDN+exogeni.net:bbnvmsite+node+orca-vm-cloud" component_manager_id="urn:publicid:IDN+exogeni.net:bbnvmsite+authority+am" component_name="orca-vm-cloud" exclusive="true" sliver_id="urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#host-1">    
            <sliver_type name="m1.small">      
                  <disk_image name="http://geni-images.renci.org/images/standard/debian/deb6-neuca-v1.0.5.xml" version="104ea3d824906f0e13cebb89b14df232290553b1 "/>      
            </sliver_type>    
            <services/>    
            <interface client_id="host-1:if0">      
                  <ip address="172.16.1.1" netmask="255.255.255.0" type="ipv4"/>      
            </interface>    
            <ns4:geni_sliver_info creation_time="2012-12-04T15:43:55.044-05:00" creator_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers" expiration_time="2012-12-18T14:43:55.044-05:00" start_time="2012-12-04T15:43:55.044-05:00" state="Nascent"/>    
      </node>  
      <node client_id="host-8" component_id="urn:publicid:IDN+exogeni.net:bbnvmsite+node+orca-vm-cloud" component_manager_id="urn:publicid:IDN+exogeni.net:bbnvmsite+authority+am" component_name="orca-vm-cloud" exclusive="true" sliver_id="urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#host-8">    
            <sliver_type name="m1.small">      
                  <disk_image name="http://geni-images.renci.org/images/standard/debian/deb6-neuca-v1.0.5.xml" version="104ea3d824906f0e13cebb89b14df232290553b1 "/>      
            </sliver_type>    
            <services/>    
            <interface client_id="host-8:if0">      
                  <ip address="172.16.7.2" netmask="255.255.255.0" type="ipv4"/>      
            </interface>    
            <interface client_id="host-8:if1">      
                  <ip address="172.16.8.1" netmask="255.255.255.0" type="ipv4"/>      
            </interface>    
            <ns4:geni_sliver_info creation_time="2012-12-04T15:43:55.044-05:00" creator_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers" expiration_time="2012-12-18T14:43:55.044-05:00" start_time="2012-12-04T15:43:55.044-05:00" state="Nascent"/>    
      </node>  
      <node client_id="host-3" component_id="urn:publicid:IDN+exogeni.net:bbnvmsite+node+orca-vm-cloud" component_manager_id="urn:publicid:IDN+exogeni.net:bbnvmsite+authority+am" component_name="orca-vm-cloud" exclusive="true" sliver_id="urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#host-3">    
            <sliver_type name="m1.small">      
                  <disk_image name="http://geni-images.renci.org/images/standard/debian/deb6-neuca-v1.0.5.xml" version="104ea3d824906f0e13cebb89b14df232290553b1 "/>      
            </sliver_type>    
            <services/>    
            <interface client_id="host-3:if0">      
                  <ip address="172.16.2.2" netmask="255.255.255.0" type="ipv4"/>      
            </interface>    
            <interface client_id="host-3:if1">      
                  <ip address="172.16.3.1" netmask="255.255.255.0" type="ipv4"/>      
            </interface>    
            <ns4:geni_sliver_info creation_time="2012-12-04T15:43:55.044-05:00" creator_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers" expiration_time="2012-12-18T14:43:55.044-05:00" start_time="2012-12-04T15:43:55.044-05:00" state="Nascent"/>    
      </node>  
      <node client_id="host-6" component_id="urn:publicid:IDN+exogeni.net:bbnvmsite+node+orca-vm-cloud" component_manager_id="urn:publicid:IDN+exogeni.net:bbnvmsite+authority+am" component_name="orca-vm-cloud" exclusive="true" sliver_id="urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#host-6">    
            <sliver_type name="m1.small">      
                  <disk_image name="http://geni-images.renci.org/images/standard/debian/deb6-neuca-v1.0.5.xml" version="104ea3d824906f0e13cebb89b14df232290553b1 "/>      
            </sliver_type>    
            <services/>    
            <interface client_id="host-6:if0">      
                  <ip address="172.16.5.2" netmask="255.255.255.0" type="ipv4"/>      
            </interface>    
            <interface client_id="host-6:if1">      
                  <ip address="172.16.6.1" netmask="255.255.255.0" type="ipv4"/>      
            </interface>    
            <ns4:geni_sliver_info creation_time="2012-12-04T15:43:55.044-05:00" creator_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers" expiration_time="2012-12-18T14:43:55.044-05:00" start_time="2012-12-04T15:43:55.044-05:00" state="Nascent"/>    
      </node>  
      <node client_id="host-5" component_id="urn:publicid:IDN+exogeni.net:bbnvmsite+node+orca-vm-cloud" component_manager_id="urn:publicid:IDN+exogeni.net:bbnvmsite+authority+am" component_name="orca-vm-cloud" exclusive="true" sliver_id="urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#host-5">    
            <sliver_type name="m1.small">      
                  <disk_image name="http://geni-images.renci.org/images/standard/debian/deb6-neuca-v1.0.5.xml" version="104ea3d824906f0e13cebb89b14df232290553b1 "/>      
            </sliver_type>    
            <services/>    
            <interface client_id="host-5:if0">      
                  <ip address="172.16.4.2" netmask="255.255.255.0" type="ipv4"/>      
            </interface>    
            <interface client_id="host-5:if1">      
                  <ip address="172.16.5.1" netmask="255.255.255.0" type="ipv4"/>      
            </interface>    
            <ns4:geni_sliver_info creation_time="2012-12-04T15:43:55.044-05:00" creator_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers" expiration_time="2012-12-18T14:43:55.044-05:00" start_time="2012-12-04T15:43:55.044-05:00" state="Nascent"/>    
      </node>  
      <link client_id="lan25" sliver_id="urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#lan25" vlantag="unknown">    
            <interface_ref client_id="host-6:if1"/>    
            <interface_ref client_id="host-7:if0"/>    
            <ns4:geni_sliver_info creation_time="2012-12-04T15:43:55.044-05:00" creator_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers" expiration_time="2012-12-18T14:43:55.044-05:00" start_time="2012-12-04T15:43:55.044-05:00" state="Nascent"/>    
      </link>  
      <link client_id="lan210" sliver_id="urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#lan210" vlantag="unknown">    
            <interface_ref client_id="host-11:if1"/>    
            <interface_ref client_id="host-12:if0"/>    
            <ns4:geni_sliver_info creation_time="2012-12-04T15:43:55.044-05:00" creator_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers" expiration_time="2012-12-18T14:43:55.044-05:00" start_time="2012-12-04T15:43:55.044-05:00" state="Nascent"/>    
      </link>  
      <link client_id="lan28" sliver_id="urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#lan28" vlantag="unknown">    
            <interface_ref client_id="host-10:if0"/>    
            <interface_ref client_id="host-9:if1"/>    
            <ns4:geni_sliver_info creation_time="2012-12-04T15:43:55.044-05:00" creator_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers" expiration_time="2012-12-18T14:43:55.044-05:00" start_time="2012-12-04T15:43:55.044-05:00" state="Nascent"/>    
      </link>  
      <link client_id="lan22" sliver_id="urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#lan22" vlantag="unknown">    
            <interface_ref client_id="host-3:if1"/>    
            <interface_ref client_id="host-4:if0"/>    
            <ns4:geni_sliver_info creation_time="2012-12-04T15:43:55.044-05:00" creator_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers" expiration_time="2012-12-18T14:43:55.044-05:00" start_time="2012-12-04T15:43:55.044-05:00" state="Nascent"/>    
      </link>  
      <link client_id="lan26" sliver_id="urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#lan26" vlantag="unknown">    
            <interface_ref client_id="host-8:if0"/>    
            <interface_ref client_id="host-7:if1"/>    
            <ns4:geni_sliver_info creation_time="2012-12-04T15:43:55.044-05:00" creator_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers" expiration_time="2012-12-18T14:43:55.044-05:00" start_time="2012-12-04T15:43:55.044-05:00" state="Nascent"/>    
      </link>  
      <link client_id="lan211" sliver_id="urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#lan211" vlantag="unknown">    
            <interface_ref client_id="host-12:if1"/>    
            <interface_ref client_id="host-13:if0"/>    
            <ns4:geni_sliver_info creation_time="2012-12-04T15:43:55.044-05:00" creator_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers" expiration_time="2012-12-18T14:43:55.044-05:00" start_time="2012-12-04T15:43:55.044-05:00" state="Nascent"/>    
      </link>  
      <link client_id="lan23" sliver_id="urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#lan23" vlantag="unknown">    
            <interface_ref client_id="host-5:if0"/>    
            <interface_ref client_id="host-4:if1"/>    
            <ns4:geni_sliver_info creation_time="2012-12-04T15:43:55.044-05:00" creator_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers" expiration_time="2012-12-18T14:43:55.044-05:00" start_time="2012-12-04T15:43:55.044-05:00" state="Nascent"/>    
      </link>  
      <link client_id="lan27" sliver_id="urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#lan27" vlantag="unknown">    
            <interface_ref client_id="host-8:if1"/>    
            <interface_ref client_id="host-9:if0"/>    
            <ns4:geni_sliver_info creation_time="2012-12-04T15:43:55.044-05:00" creator_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers" expiration_time="2012-12-18T14:43:55.044-05:00" start_time="2012-12-04T15:43:55.044-05:00" state="Nascent"/>    
      </link>  
      <link client_id="lan29" sliver_id="urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#lan29" vlantag="unknown">    
            <interface_ref client_id="host-11:if0"/>    
            <interface_ref client_id="host-10:if1"/>    
            <ns4:geni_sliver_info creation_time="2012-12-04T15:43:55.044-05:00" creator_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers" expiration_time="2012-12-18T14:43:55.044-05:00" start_time="2012-12-04T15:43:55.044-05:00" state="Nascent"/>    
      </link>  
      <link client_id="lan24" sliver_id="urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#lan24" vlantag="unknown">    
            <interface_ref client_id="host-6:if0"/>    
            <interface_ref client_id="host-5:if1"/>    
            <ns4:geni_sliver_info creation_time="2012-12-04T15:43:55.044-05:00" creator_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers" expiration_time="2012-12-18T14:43:55.044-05:00" start_time="2012-12-04T15:43:55.044-05:00" state="Nascent"/>    
      </link>  
      <link client_id="lan20" sliver_id="urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#lan20" vlantag="unknown">    
            <interface_ref client_id="host-2:if0"/>    
            <interface_ref client_id="host-1:if0"/>    
            <ns4:geni_sliver_info creation_time="2012-12-04T15:43:55.044-05:00" creator_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers" expiration_time="2012-12-18T14:43:55.044-05:00" start_time="2012-12-04T15:43:55.044-05:00" state="Nascent"/>    
      </link>  
      <link client_id="lan21" sliver_id="urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#lan21" vlantag="unknown">    
            <interface_ref client_id="host-3:if0"/>    
            <interface_ref client_id="host-2:if1"/>    
            <ns4:geni_sliver_info creation_time="2012-12-04T15:43:55.044-05:00" creator_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers" expiration_time="2012-12-18T14:43:55.044-05:00" start_time="2012-12-04T15:43:55.044-05:00" state="Nascent"/>    
      </link>  
      <ns3:geni_slice_info state="unknown" urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+13linear" uuid="a3b379cc-4206-4ad8-869b-95b2b65c51ea"/>  
</rspec>
INFO:omni: ------------------------------------------------------------
INFO:omni: Completed createsliver:

  Options as run:
		aggregate: ['eg-sm']
		framework: pg

  Args: createsliver 13linear ./exorspec/exo-13vm-linear2.rspec

  Result Summary: Got Reserved resources RSpec from geni-renci-org-11443-orca 
INFO:omni: ============================================================

Here is the failure in the sliverstatus:

$ omni.py sliverstatus -a eg-sm 13linear                                   
INFO:omni:Loading config file /home/lnevers/.gcf/omni_config
INFO:omni:Using control framework pg
INFO:omni:Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+13linear expires on 2012-12-05 21:41:24 UTC
INFO:omni:Substituting AM nickname eg-sm with URL https://geni.renci.org:11443/orca/xmlrpc, URN unspecified_AM_URN
INFO:omni:Status of Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+13linear:
INFO:omni:Sliver status for Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+13linear at AM URL https://geni.renci.org:11443/orca/xmlrpc
INFO:omni:{
  "geni_status": "failed", 
  "geni_urn": "urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+13linear", 
  "geni_resources": [
    {
      "orca_expires": "Tue Dec 18 14:43:55 EST 2012", 
      "geni_urn": "urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#host-11", 
      "geni_error": "", 
      "geni_status": "Ticketed"
    }, 
    {
      "orca_expires": "Tue Dec 18 14:43:55 EST 2012", 
      "geni_urn": "urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#host-10", 
      "geni_error": "", 
      "geni_status": "Ticketed"
    }, 
    {
      "orca_expires": "Tue Dec 18 14:43:55 EST 2012", 
      "geni_urn": "urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#host-13", 
      "geni_error": "", 
      "geni_status": "Ticketed"
    }, 
    {
      "orca_expires": "Tue Dec 18 14:43:55 EST 2012", 
      "geni_urn": "urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#host-12", 
      "geni_error": "", 
      "geni_status": "Ticketed"
    }, 
    {
      "orca_expires": "Tue Dec 18 14:43:55 EST 2012", 
      "geni_urn": "urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#lan25", 
      "geni_error": "", 
      "geni_status": "Active"
    }, 
    {
      "orca_expires": "Tue Dec 18 14:43:55 EST 2012", 
      "geni_urn": "urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#lan26", 
      "geni_error": "", 
      "geni_status": "Active"
    }, 
    {
      "orca_expires": "Tue Dec 18 14:43:55 EST 2012", 
      "geni_urn": "urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#lan23", 
      "geni_error": "", 
      "geni_status": "Active"
    }, 
    {
      "orca_expires": "Tue Dec 18 14:43:55 EST 2012", 
      "geni_urn": "urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#lan24", 
      "geni_error": "", 
      "geni_status": "Active"
    }, 
    {
      "orca_expires": "Tue Dec 18 14:43:55 EST 2012", 
      "geni_urn": "urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#lan29", 
      "geni_error": "", 
      "geni_status": "Active"
    }, 
    {
      "orca_expires": "Tue Dec 18 14:43:55 EST 2012", 
      "geni_urn": "urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#lan27", 
      "geni_error": "", 
      "geni_status": "Active"
    }, 
    {
      "orca_expires": "Tue Dec 18 14:43:55 EST 2012", 
      "geni_urn": "urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#lan28", 
      "geni_error": "", 
      "geni_status": "Active"
    }, 
    {
      "orca_expires": "Tue Dec 18 14:43:55 EST 2012", 
      "geni_urn": "urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#lan21", 
      "geni_error": "", 
      "geni_status": "Active"
    }, 
    {
      "orca_expires": "Tue Dec 18 14:43:55 EST 2012", 
      "geni_urn": "urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#lan22", 
      "geni_error": "", 
      "geni_status": "Active"
    }, 
    {
      "orca_expires": "Tue Dec 18 14:43:55 EST 2012", 
      "geni_urn": "urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#lan20", 
      "geni_error": "", 
      "geni_status": "Active"
    }, 
    {
      "orca_expires": "Tue Dec 18 14:43:55 EST 2012", 
      "geni_urn": "urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#host-1", 
      "geni_error": "Reservation 0e294f19-58f4-4d1b-a92c-ea4a3368d905 (Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+13linear) is in state [Failed,None]\n\nLast ticket update: Insufficient resources for specified start time, Failing reservation:0e294f19-58f4-4d1b-a92c-ea4a3368d905\n\nTicket events\nInsufficient resources for specified start time, Failing reservation:0e294f19-58f4-4d1b-a92c-ea4a3368d905", 
      "geni_status": "Failed"
    }, 
    {
      "orca_expires": "Tue Dec 18 14:43:55 EST 2012", 
      "geni_urn": "urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#host-2", 
      "geni_error": "", 
      "geni_status": "Ticketed"
    }, 
    {
      "orca_expires": "Tue Dec 18 14:43:55 EST 2012", 
      "geni_urn": "urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#host-3", 
      "geni_error": "", 
      "geni_status": "Ticketed"
    }, 
    {
      "orca_expires": "Tue Dec 18 14:43:55 EST 2012", 
      "geni_urn": "urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#host-4", 
      "geni_error": "", 
      "geni_status": "Ticketed"
    }, 
    {
      "orca_expires": "Tue Dec 18 14:43:55 EST 2012", 
      "geni_urn": "urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#host-5", 
      "geni_error": "Reservation 6de29f56-0fb7-4304-bbf8-c2716ffe0ade (Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+13linear) is in state [Failed,None]\n\nLast ticket update: Insufficient resources for specified start time, Failing reservation:6de29f56-0fb7-4304-bbf8-c2716ffe0ade\n\nTicket events\nInsufficient resources for specified start time, Failing reservation:6de29f56-0fb7-4304-bbf8-c2716ffe0ade", 
      "geni_status": "Failed"
    }, 
    {
      "orca_expires": "Tue Dec 18 14:43:55 EST 2012", 
      "geni_urn": "urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#host-6", 
      "geni_error": "Reservation 7c8cce97-7702-4d19-aa10-8fcde66ef593 (Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+13linear) is in state [Failed,None]\n\nLast ticket update: Insufficient resources for specified start time, Failing reservation:7c8cce97-7702-4d19-aa10-8fcde66ef593\n\nTicket events\nInsufficient resources for specified start time, Failing reservation:7c8cce97-7702-4d19-aa10-8fcde66ef593", 
      "geni_status": "Failed"
    }, 
    {
      "orca_expires": "Tue Dec 18 14:43:55 EST 2012", 
      "geni_urn": "urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#host-7", 
      "geni_error": "Reservation df8862d2-4a81-4d97-9da8-5a6b852f351e (Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+13linear) is in state [Failed,None]\n\nLast ticket update: Insufficient resources for specified start time, Failing reservation:df8862d2-4a81-4d97-9da8-5a6b852f351e\n\nTicket events\nInsufficient resources for specified start time, Failing reservation:df8862d2-4a81-4d97-9da8-5a6b852f351e", 
      "geni_status": "Failed"
    }, 
    {
      "orca_expires": "Tue Dec 18 14:43:55 EST 2012", 
      "geni_urn": "urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#host-8", 
      "geni_error": "Reservation 74abe4b4-308c-4666-9c48-1a7dad2c0127 (Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+13linear) is in state [Failed,None]\n\nLast ticket update: Insufficient resources for specified start time, Failing reservation:74abe4b4-308c-4666-9c48-1a7dad2c0127\n\nTicket events\nInsufficient resources for specified start time, Failing reservation:74abe4b4-308c-4666-9c48-1a7dad2c0127", 
      "geni_status": "Failed"
    }, 
    {
      "orca_expires": "Tue Dec 18 14:43:55 EST 2012", 
      "geni_urn": "urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#host-9", 
      "geni_error": "Reservation 60f932f2-c93c-4603-b590-dc75a2a53b0c (Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+13linear) is in state [Failed,None]\n\nLast ticket update: Insufficient resources for specified start time, Failing reservation:60f932f2-c93c-4603-b590-dc75a2a53b0c\n\nTicket events\nInsufficient resources for specified start time, Failing reservation:60f932f2-c93c-4603-b590-dc75a2a53b0c", 
      "geni_status": "Failed"
    }, 
    {
      "orca_expires": "Tue Dec 18 14:43:55 EST 2012", 
      "geni_urn": "urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#lan210", 
      "geni_error": "", 
      "geni_status": "Active"
    }, 
    {
      "orca_expires": "Tue Dec 18 14:43:55 EST 2012", 
      "geni_urn": "urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+3d5a7123-744c-4ea1-9c39-980f6b5a6b9f#lan211", 
      "geni_error": "", 
      "geni_status": "Active"
    }
  ]
}
INFO:omni: ------------------------------------------------------------
INFO:omni: Completed sliverstatus:

  Options as run:
		aggregate: ['eg-sm']
		framework: pg

  Args: sliverstatus 13linear

  Result Summary: Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+13linear expires on 2012-12-05 21:41:24 UTC
Returned status of slivers on 1 of 1 possible aggregates. 
INFO:omni: ============================================================

[ibaldin@renci.org]

I don't think this is a bug. People are using racks and contention for resources is possible. As you can tell from the error message the problem was not enough resources at that time.

[ibaldin@renci.org]

I just created a linear 13-node slice successfully.

[lnevers@bbn.com]

ok, I will try again to setup a 13 node linear setup and will update the ticket when done.

[lnevers@bbn.com]

Successfully created a 13 VM Linear sliver, was able to login to each node and exchange traffic. Closing ticket.