Opened 9 years ago

Last modified 9 years ago

#190 new

ExoSM Aggregate does not catch ticket update failure for unavailable resources in manifest

Reported by: lnevers@bbn.com Owned by: somebody
Priority: major Milestone:
Component: AM Version: SPIRAL5
Keywords: Cc:
Dependencies:

Description

This problem was originally found as part of http://groups.geni.net/exogeni/ticket/188

In some cases the overall createsliver results from the ExoSM return a sliver manifest rather than an error result when a resource is not available. Below is a capture that shows that the content of the manifest includes a "Last ticket update: Insufficient resources" message, but this failure is not part of the overall failure indication from the Aggregate.

+ omni.py -a eg-sm createsliver EG-CT-2 EG-CT-2-uh.rspec
INFO:omni:Loading config file /home/lnevers/.gcf/omni_config
INFO:omni:Using control framework portal
INFO:omni:Substituting AM nickname eg-sm with URL https://geni.renci.org:11443/orca/xmlrpc, URN unspecified_AM_URN
INFO:omni:Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+EG-CT-2 expires within 1 day on 2013-07-03 19:50:44 UTC
INFO:omni:Substituting AM nickname eg-sm with URL https://geni.renci.org:11443/orca/xmlrpc, URN unspecified_AM_URN
INFO:omni:Substituting AM nickname eg-sm with URL https://geni.renci.org:11443/orca/xmlrpc, URN unspecified_AM_URN
INFO:omni:Creating sliver(s) from rspec file EG-CT-2-uh.rspec for slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+EG-CT-2
INFO:omni:Got return from CreateSliver for slice EG-CT-2 at https://geni.renci.org:11443/orca/xmlrpc:
INFO:omni:<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
INFO:omni:  <!-- Reserved resources for:
	Slice: EG-CT-2
	at AM:
	URN: unspecified_AM_URN
	URL: https://geni.renci.org:11443/orca/xmlrpc
 -->
INFO:omni:  
<rspec type="manifest" xsi:schemaLocation="http://www.geni.net/resources/rspec/3 http://www.geni.net/resources/rspec/3/manifest.xsd http://hpn.east.isi.edu/rspec/ext/stitch/0.1/ http://hpn.east.isi.edu/rspec/ext/stitch/0.1/stitch-schema.xsd http://groups.geni.net/exogeni/attachment/wiki/RspecExtensions/slice-info/1 http://groups.geni.net/exogeni/attachment/wiki/RspecExtensions/slice-info/1/slice_info.xsd?format=raw http://groups.geni.net/exogeni/attachment/wiki/RspecExtensions/sliver-info/1 http://groups.geni.net/exogeni/attachment/wiki/RspecExtensions/sliver-info/1/sliver_info.xsd?format=raw http://www.geni.net/resources/rspec/ext/postBootScript/1 http://www.geni.net/resources/rspec/ext/postBootScript/1/request.xsd" xmlns:ns2="http://hpn.east.isi.edu/rspec/ext/stitch/0.1/" xmlns="http://www.geni.net/resources/rspec/3" xmlns:ns4="http://groups.geni.net/exogeni/attachment/wiki/RspecExtensions/sliver-info/1" xmlns:ns3="http://groups.geni.net/exogeni/attachment/wiki/RspecExtensions/slice-info/1" xmlns:ns5="http://www.geni.net/resources/rspec/ext/postBootScript/1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <node sliver_id="urn:publicid:IDN+exogeni.net:uhvmsite+sliver+f2fd2254-f2b5-4279-9e24-e68190505a68:BM-1" exclusive="true" component_name="orca-xcat-cloud" component_manager_id="urn:publicid:IDN+exogeni.net:uhvmsite+authority+am" component_id="urn:publicid:IDN+exogeni.net:uhvmsite+node+orca-xcat-cloud" client_id="BM-1">
        <location latitude="29.72327" longitude="-95.34269" country="Unspecified"/>
        <sliver_type name="ExoGENI-M4">
            <disk_image version="d1044d9162bd7851e3fc2c57a8251ad6b3641c0c" name="http://geni-images.renci.org/images/standard/debian/deb6-neuca-v1.0.8.xml"/>
        </sliver_type>
        <services/>
        <interface mac_address="fe:16:3e:00:5a:a8" client_id="BM-1:if0">
            <ip type="ipv4" netmask="255.255.255.0" address="172.16.1.1"/>
        </interface>
        <ns4:geni_sliver_info error="Reservation 31b86490-0462-418b-9161-90e97c832d64 (Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+EG-CT-2) is in state [Failed,None]

Last ticket update: Insufficient resources for specified start time, Failing reservation:31b86490-0462-418b-9161-90e97c832d64

Ticket events
Insufficient resources for specified start time, Failing reservation:31b86490-0462-418b-9161-90e97c832d64" destroyed="true" state="failed" start_time="2013-07-03T14:06:54.000Z" expiration_time="2013-07-03T19:50:44.000Z" creation_time="2013-07-03T14:06:54.000Z" creator_urn="lnevers@bbn.com, urn:publicid:IDN+ch.geni.net+user+lnevers"/>
    </node>
    <node sliver_id="urn:publicid:IDN+exogeni.net:uhvmsite+sliver+f2fd2254-f2b5-4279-9e24-e68190505a68:VM-1" exclusive="false" component_name="orca-vm-cloud" component_manager_id="urn:publicid:IDN+exogeni.net:uhvmsite+authority+am" component_id="urn:publicid:IDN+exogeni.net:uhvmsite+node+orca-vm-cloud" client_id="VM-1">
        <location latitude="29.72327" longitude="-95.34269" country="Unspecified"/>
        <sliver_type name="m1.small">
            <disk_image version="64ad567ce3b1c0dbaa15bad673bbf556a9593e1c" name="http://geni-images.renci.org/images/standard/debian/deb6-neuca-v1.0.6.xml"/>
        </sliver_type>
        <services>
            <ns5:services_post_boot_script type="velocity">#!/bin/bash
# Automatically generated boot script
execString=&amp;quot;/bin/sh -c \&amp;quot;sudo yum install iperf -y\&amp;quot;&amp;quot;
eval $execString

</ns5:services_post_boot_script>
        </services>
        <interface mac_address="fe:16:3e:00:67:66" client_id="VM-1:if0">
            <ip type="ipv4" netmask="255.255.255.0" address="172.16.1.2"/>
        </interface>
        <ns4:geni_sliver_info state="configuring" start_time="2013-07-03T14:06:54.000Z" expiration_time="2013-07-03T19:50:44.000Z" creation_time="2013-07-03T14:06:54.000Z" creator_urn="lnevers@bbn.com, urn:publicid:IDN+ch.geni.net+user+lnevers"/>
    </node>
    <link vlantag="unknown" sliver_id="urn:publicid:IDN+exogeni.net:uhvmsite+sliver+f2fd2254-f2b5-4279-9e24-e68190505a68:lan0" client_id="lan0">
        <interface_ref client_id="BM-1:if0"/>
        <interface_ref client_id="VM-1:if0"/>
        <ns4:geni_sliver_info state="configuring" start_time="2013-07-03T14:06:54.000Z" expiration_time="2013-07-03T19:50:44.000Z" creation_time="2013-07-03T14:06:54.000Z" creator_urn="lnevers@bbn.com, urn:publicid:IDN+ch.geni.net+user+lnevers"/>
    </link>
    <ns3:geni_slice_info state="configuring" uuid="250428c3-7266-4111-a29a-f4682a90ec22" urn="urn:publicid:IDN+ch.geni.net:ln-prj+slice+EG-CT-2"/>
</rspec>

INFO:omni: ------------------------------------------------------------
INFO:omni: Completed createsliver:

  Options as run:
		aggregate: ['eg-sm']
		framework: portal
		project: ln-prj

  Args: createsliver EG-CT-2 EG-CT-2-uh.rspec

  Result Summary: Got Reserved resources RSpec from geni-renci-org-11443-orca 
INFO:omni: ============================================================
+ sleep 900
+ omni.py -a eg-sm sliverstatus EG-CT-2
INFO:omni:Loading config file /home/lnevers/.gcf/omni_config
INFO:omni:Using control framework portal
INFO:omni:Substituting AM nickname eg-sm with URL https://geni.renci.org:11443/orca/xmlrpc, URN unspecified_AM_URN
INFO:omni:Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+EG-CT-2 expires within 1 day on 2013-07-03 19:50:44 UTC
INFO:omni:Substituting AM nickname eg-sm with URL https://geni.renci.org:11443/orca/xmlrpc, URN unspecified_AM_URN
INFO:omni:Status of Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+EG-CT-2:
INFO:omni: ------------------------------------------------------------
INFO:omni: Completed sliverstatus:

  Options as run:
		aggregate: ['eg-sm']
		framework: portal
		project: ln-prj

  Args: sliverstatus EG-CT-2

  Result Summary: Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+EG-CT-2 expires within 1 day(s) on 2013-07-03 19:50:44 UTC

Failed to get SliverStatus on EG-CT-2 at AM https://geni.renci.org:11443/orca/xmlrpc: 
Error from Aggregate: code 2: ERROR: There are no reservations in the slice with 
sliceId = urn:publicid:IDN+ch.geni.net:ln-prj+slice+EG-CT-2.
Returned status of slivers on 0 of 1 possible aggregates. 

Change History (1)

comment:1 Changed 9 years ago by lnevers@bbn.com

Another occurrence of this problem. I ran a created a sliver with a bare metal node which returned a manifest with no errors or warnings, but from talking to Victor on IRC, I found out that both bare metal nodes were in use by Victor at the time of the request. Here is the createsliver that requested the "unavailable" bare metal:

$ omni.py createsliver EG-CT-2b -a sm-eg EG-CT-2-uh.rspec                                      
INFO:omni:Loading config file /home/lnevers/.gcf/omni_config
INFO:omni:Using control framework portal
INFO:omni:Substituting AM nickname sm-eg with URL https://geni.renci.org:11443/orca/xmlrpc, URN unspecified_AM_URN
INFO:omni:Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+EG-CT-2b expires on 2013-07-10 16:52:42 UTC
INFO:omni:Substituting AM nickname sm-eg with URL https://geni.renci.org:11443/orca/xmlrpc, URN unspecified_AM_URN
INFO:omni:Substituting AM nickname sm-eg with URL https://geni.renci.org:11443/orca/xmlrpc, URN unspecified_AM_URN
INFO:omni:Creating sliver(s) from rspec file EG-CT-2-uh.rspec for slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+EG-CT-2b
INFO:omni:Got return from CreateSliver for slice EG-CT-2b at https://geni.renci.org:11443/orca/xmlrpc:
INFO:omni:<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
INFO:omni:  <!-- Reserved resources for:
	Slice: EG-CT-2b
	at AM:
	URN: unspecified_AM_URN
	URL: https://geni.renci.org:11443/orca/xmlrpc
 -->
INFO:omni:  
<rspec type="manifest" xsi:schemaLocation="http://www.geni.net/resources/rspec/3 http://www.geni.net/resources/rspec/3/manifest.xsd http://hpn.east.isi.edu/rspec/ext/stitch/0.1/ http://hpn.east.isi.edu/rspec/ext/stitch/0.1/stitch-schema.xsd http://groups.geni.net/exogeni/attachment/wiki/RspecExtensions/slice-info/1 http://groups.geni.net/exogeni/attachment/wiki/RspecExtensions/slice-info/1/slice_info.xsd?format=raw http://groups.geni.net/exogeni/attachment/wiki/RspecExtensions/sliver-info/1 http://groups.geni.net/exogeni/attachment/wiki/RspecExtensions/sliver-info/1/sliver_info.xsd?format=raw http://www.geni.net/resources/rspec/ext/postBootScript/1 http://www.geni.net/resources/rspec/ext/postBootScript/1/request.xsd" xmlns:ns2="http://hpn.east.isi.edu/rspec/ext/stitch/0.1/" xmlns="http://www.geni.net/resources/rspec/3" xmlns:ns4="http://groups.geni.net/exogeni/attachment/wiki/RspecExtensions/sliver-info/1" xmlns:ns3="http://groups.geni.net/exogeni/attachment/wiki/RspecExtensions/slice-info/1" xmlns:ns5="http://www.geni.net/resources/rspec/ext/postBootScript/1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <node sliver_id="urn:publicid:IDN+exogeni.net:uhvmsite+sliver+b4041a4d-7514-4c5f-81e7-a4b993a9224b:BM-1" exclusive="true" component_name="orca-xcat-cloud" component_manager_id="urn:publicid:IDN+exogeni.net:uhvmsite+authority+am" component_id="urn:publicid:IDN+exogeni.net:uhvmsite+node+orca-xcat-cloud" client_id="BM-1">
        <location latitude="29.72327" longitude="-95.34269" country="Unspecified"/>
        <sliver_type name="ExoGENI-M4">
            <disk_image version="d1044d9162bd7851e3fc2c57a8251ad6b3641c0c" name="http://geni-images.renci.org/images/standard/debian/deb6-neuca-v1.0.8.xml"/>
        </sliver_type>
        <services/>
        <interface mac_address="fe:16:3e:00:c4:7d" client_id="BM-1:if0">
            <ip type="ipv4" netmask="255.255.255.0" address="172.16.1.1"/>
        </interface>
        <ns4:geni_sliver_info state="configuring" start_time="2013-07-03T14:26:27.000-04:00" expiration_time="2013-07-10T12:52:42.000-04:00" creation_time="2013-07-03T14:26:27.000-04:00" creator_urn="lnevers@bbn.com, urn:publicid:IDN+ch.geni.net+user+lnevers"/>
    </node>
    <node sliver_id="urn:publicid:IDN+exogeni.net:uhvmsite+sliver+b4041a4d-7514-4c5f-81e7-a4b993a9224b:VM-1" exclusive="false" component_name="orca-vm-cloud" component_manager_id="urn:publicid:IDN+exogeni.net:uhvmsite+authority+am" component_id="urn:publicid:IDN+exogeni.net:uhvmsite+node+orca-vm-cloud" client_id="VM-1">
        <location latitude="29.72327" longitude="-95.34269" country="Unspecified"/>
        <sliver_type name="m1.small">
            <disk_image version="64ad567ce3b1c0dbaa15bad673bbf556a9593e1c" name="http://geni-images.renci.org/images/standard/debian/deb6-neuca-v1.0.6.xml"/>
        </sliver_type>
        <services>
            <ns5:services_post_boot_script type="velocity">#!/bin/bash
# Automatically generated boot script
execString=&amp;quot;/bin/sh -c \&amp;quot;sudo yum install iperf -y\&amp;quot;&amp;quot;
eval $execString

</ns5:services_post_boot_script>
        </services>
        <interface mac_address="fe:16:3e:00:32:cf" client_id="VM-1:if0">
            <ip type="ipv4" netmask="255.255.255.0" address="172.16.1.2"/>
        </interface>
        <ns4:geni_sliver_info state="configuring" start_time="2013-07-03T14:26:27.000-04:00" expiration_time="2013-07-10T12:52:42.000-04:00" creation_time="2013-07-03T14:26:27.000-04:00" creator_urn="lnevers@bbn.com, urn:publicid:IDN+ch.geni.net+user+lnevers"/>
    </node>
    <link vlantag="unknown" sliver_id="urn:publicid:IDN+exogeni.net:uhvmsite+sliver+b4041a4d-7514-4c5f-81e7-a4b993a9224b:lan0" client_id="lan0">
        <interface_ref client_id="BM-1:if0"/>
        <interface_ref client_id="VM-1:if0"/>
        <ns4:geni_sliver_info state="configuring" start_time="2013-07-03T14:26:27.000-04:00" expiration_time="2013-07-10T12:52:42.000-04:00" creation_time="2013-07-03T14:26:27.000-04:00" creator_urn="lnevers@bbn.com, urn:publicid:IDN+ch.geni.net+user+lnevers"/>
    </link>
    <ns3:geni_slice_info state="configuring" uuid="ad9ad4ef-74e7-49d2-954e-13469af2f5d5" urn="urn:publicid:IDN+ch.geni.net:ln-prj+slice+EG-CT-2b"/>
</rspec>

INFO:omni: ------------------------------------------------------------
INFO:omni: Completed createsliver:

  Options as run:
		aggregate: ['sm-eg']
		framework: portal
		project: ln-prj

  Args: createsliver EG-CT-2b EG-CT-2-uh.rspec

  Result Summary: Got Reserved resources RSpec from geni-renci-org-11443-orca 
INFO:omni: ============================================================
Note: See TracTickets for help on using tickets.