wiki:GENIExperimenter/ExperimentExample-stitching

Version 49 (modified by lnevers@bbn.com, 5 years ago) (diff)

--

GENI Network Stitching Example

This page uses GCF 2.6 to show an example experiment that creates a network stitching path between the GPO InstaGENI and the Utah InstaGENI racks.

The experiment uses GENI Clearinghouse credentials with the new Omni tool named stitcher. GENI network stitching does not require any special configuration. If you haven't already configured Omni, please follow these configuration instructions. For a list of other GENI sites that support stitching see the GENI Network Stitching Sites page. If you need further assistance, please contact help@geni.net.

1. Design the Stitching Experiment

Design/Setup

GENI network stitching is available for the aggregates listed in the GENI Network Stitching Sites page. This example shows a simple stitching scenario with one server VM in the GPO InstaGENI rack and one client VM in the Utah InstaGENI rack. The server-gpo host is configured as a Web, Iperf and PingPlus server; it also has a routable IP address that is used to access the web server and display results for all traffic types.

We define an RSpec with 2 nodes and one link between them. The link must specify the 2 aggregates that are the endpoints of the link. The tools will do the rest of the work required to stitch the topology. This is the request RSpec used for this experiment:

<?xml version="1.0" encoding="UTF-8"?>
<rspec type="request" 
        xmlns="http://www.geni.net/resources/rspec/3" 
        xmlns:emulab="http://www.protogeni.net/resources/rspec/ext/emulab/1"
        xmlns:planetlab="http://www.planet-lab.org/resources/sfa/ext/planetlab/1" 
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
        xsi:schemaLocation="http://www.geni.net/resources/rspec/3 
        http://www.geni.net/resources/rspec/3/request.xsd">  

  <node client_id="server-gpo" component_manager_id="urn:publicid:IDN+instageni.gpolab.bbn.com+authority+cm" exclusive="false">
    <sliver_type name="emulab-openvz"/>
    <services>
      <execute command="sudo /local/install-script.sh" shell="sh"/>
      <install install_path="/local" url="http://www.gpolab.bbn.com/~lnevers/StitchDemo.tar.gz"/>
   </services>
    <emulab:routable_control_ip xmlns:emulab="http://www.protogeni.net/resources/rspec/ext/emulab/1"/>
    <interface client_id="server-gpo:if0">
      <ip address="192.168.4.1" netmask="255.255.255.0" type="ipv4"/>
    </interface>
  </node>
  <node client_id="client-utah" component_manager_id="urn:publicid:IDN+utah.geniracks.net+authority+cm" exclusive="false">
    <sliver_type name="emulab-openvz"/>
   <services>
      <execute command="sudo /local/install-script.sh" shell="sh"/>
      <install install_path="/local" url="http://www.gpolab.bbn.com/~lnevers/StitchDemo.tar.gz"/>
    </services>
    <interface client_id="client-utah:if0">
      <ip address="192.168.4.2" netmask="255.255.255.0" type="ipv4"/>
    </interface>
  </node>
  <link client_id="link">
    <component_manager name="urn:publicid:IDN+instageni.gpolab.bbn.com+authority+cm"/>
    <component_manager name="urn:publicid:IDN+utah.geniracks.net+authority+cm"/>
    <interface_ref client_id="server-gpo:if0"/>
    <interface_ref client_id="client-utah:if0"/>
    <property source_id="server-gpo:if0" dest_id="client-utah:if0" capacity="20000"/>
    <property source_id="client-utah:if0" dest_id="server-gpo:if0" capacity="20000"/>
  </link>
</rspec>

Notes:

  • In the above RSpec, stitcher uses the link information in the component_manager name to contact the GENI Stitching Computation Service (SCS) and determine a path between the end-point hosts.
  • The default bandwidth allocation is 100 MB, but for the example we set the link to 20 MB, which is sufficient to show the experiment working.

2. Execute the Stitching experiment

Execute

We now use stitcher to set up a stitched topology between the two nodes.

  • Stitcher simplifies experiment set-up because it creates slivers at all aggregates, including the network stitching path.
  • Stitcher determines all your experiment compute resource aggregates from your RSpec. (You no longer need to enter “-a aggregateName”.)
  • Stitcher uses the MAX Stitching Computation Service (SCS) to determine VLAN paths needed for your experiment.
  • VLANs are a scarce commodity, and failures can happen. Stitcher will retry if it fails to create a sliver at an aggregate.

We first create a slice for this experiment:

$ omni.py createslice gpo2utah
14:44:56 INFO    : Loading agg_nick_cache file '/home/lnevers/.gcf/agg_nick_cache'
14:44:56 INFO    : Loading config file /home/lnevers/.gcf/omni_config
14:44:56 INFO    : Using control framework portal
14:44:56 INFO    : Member Authority is https://ch.geni.net/MA (from config)
14:44:56 INFO    : Slice Authority is https://ch.geni.net/SA (from config)
14:45:00 INFO    : Created slice with Name gpo2utah, URN urn:publicid:IDN+ch.geni.net:Luisa_Nevers-Test-Proj_01+slice+gpo2utah, Expiration 2014-06-13 18:45:00+00:00
14:45:00 INFO    :  ------------------------------------------------------
14:45:00 INFO    :  Completed createslice:
  Options as run:
		framework: portal
		project: Luisa_Nevers-Test-Proj_01

  Args: createslice gpo2utah
  Result Summary: Created slice with Name gpo2utah, URN urn:publicid:IDN+ch.geni.net:Luisa_Nevers-Test-Proj_01+slice+gpo2utah, Expiration 2014-06-13 18:45:00+00:00
14:45:00 INFO    :  ======================================================

Next, we create all necessary slivers using stitcher which will use the RSpec to determine which aggregates are used, so there is no need to specify the -a aggregate option. Also the stitcher records the aggregates associated with the slice in a file in the ~/.gcf directory, for later operations with the slice. To create slivers at all aggregates, we type this command:

$ stitcher.py createsliver gpo2utah stitch-demo.rspec -o
14:45:06 INFO    : Configured logging from file /home/lnevers/gcf-2.6-rc6/src/gcf/stitcher_logging.conf
14:45:07 INFO    : Reading slice gpo2utah credential...   <=(1)
14:45:07 INFO    : Slice urn:publicid:IDN+ch.geni.net:Luisa_Nevers-Test-Proj_01+slice+gpo2utah expires on 2014-06-13 18:45:00 UTC
14:45:07 INFO    : Calling SCS...                         <=(2)
14:45:08 INFO    : Multi-AM reservation will include resources from these aggregates:
14:45:08 INFO    : 	<Aggregate utah-ig>               <=(2)
14:45:08 INFO    : 	<Aggregate gpo-ig>
14:45:08 INFO    : 	<Aggregate utah-pg>
14:45:08 INFO    : 	<Aggregate ion>
14:45:08 INFO    : Stitcher doing createsliver at <Aggregate gpo-ig>...   <=(3)
14:45:25 INFO    : ... Allocation at <Aggregate gpo-ig> complete.
14:45:25 INFO    : Stitcher doing createsliver at <Aggregate utah-pg>...
14:46:13 INFO    : ... Allocation at <Aggregate utah-pg> complete.
14:46:13 INFO    : Stitcher doing createsliver at <Aggregate utah-ig>...
14:46:35 INFO    : ... Allocation at <Aggregate utah-ig> complete.
14:46:35 INFO    : Stitcher doing createsliver at <Aggregate ion>...
14:47:19 INFO    : DCN AM <Aggregate ion>: must wait for status ready....
14:47:19 INFO    : Pausing 30 seconds to let circuit become ready...
14:48:04 INFO    : Getting credential from file /tmp/slice-ch-geni-net-Luisa_Nevers-Test-Proj_01-gpo2utah-for-lnevers-cred.xml
14:48:04 INFO    : ... Allocation at <Aggregate ion> complete.
14:48:04 INFO    : All aggregates are complete.
14:48:04 INFO    : Your resources expire at 3 different times at different AMs. The first expiration is 2014-06-06 23:46:15 UTC at <Aggregate utah-ig> and <Aggregate utah-pg>. Second expiration is 2014-06-07T18:46:39 UTC. 
14:48:04 INFO    : Saved combined reservation RSpec at 4 AMs to file 
                 '/home/lnevers/gcf-test/stitch-test/gpo2utah-manifest-rspec-stitching-combined.xml' <=(4)
Success: Reserved resources in slice gpo2utah at 4 Aggregates (including 2 intermediate aggregate(s) not in the original request), creating 1 stitched link(s).

(1) Slice credentials are checked and rspeclint is run on the stitch-demo-utah.rspec request RSpec, if rspeclint is found in your search path.
(2) Stitcher contacts the Stitching Computation Service (SCS) which returns the list of AMs needed to establish the stitching path, including the Internet2 ION aggregate.
(3) Stitcher generates a request RSpec for each aggregates and creates a sliver at each aggregate(path & end-points).
(4) A combined manifest RSpec is assembled that shows all aggregates and VLANs used in the slice.

The above stitched scenario sliver creation accomplishes several steps that previously required manual set-up:

  • Switch configuration at rack endpoints
  • Static circuit configuration to reach ION in the aggregate
  • ION/OSCARS creation of a dynamic circuit with VLAN translation
  • A static connection over UEN
  • A routed connection across PG Utah
  • A dynamic circuit and then a fixed circuit to Utah InstaGENI.

The overall information about allocated resources that can be found in the file are shown here:

Below is part of the gpo2utah-manifest-rspec-stitching-combined.xml file showing the stitching manifest that includes all hops along the stitched VLAN path:

<stitching lastUpdateTime="20140606:11:45:07" xmlns="http://hpn.east.isi.edu/rspec/ext/stitch/0.1/">
<path id="link">
<hop id="1">
<link id="urn:publicid:IDN+instageni.gpolab.bbn.com+interface+procurve2:5.24.ion.newy.ae0">
<trafficEngineeringMetric> 10  </trafficEngineeringMetric>
<capacity> 20000  </capacity>
<switchingCapabilityDescriptor>
<switchingcapType> l2sc</switchingcapType>
<encodingType> ethernet</encodingType>
<switchingCapabilitySpecificInfo>
<switchingCapabilitySpecificInfo_L2sc>
<interfaceMTU> 1500</interfaceMTU>
<vlanRangeAvailability> 3727</vlanRangeAvailability>
<suggestedVLANRange> 3727</suggestedVLANRange>
<vlanTranslation> false</vlanTranslation>
</switchingCapabilitySpecificInfo_L2sc>
</switchingCapabilitySpecificInfo>
</switchingCapabilityDescriptor>
</link>
<nextHop> 2  </nextHop>
</hop>
<hop id="2">
<link id="urn:publicid:IDN+ion.internet2.edu+interface+rtr.newy:ae0:bbn">
<trafficEngineeringMetric> 10 </trafficEngineeringMetric>
<capacity> 20000.0 </capacity>
<switchingCapabilityDescriptor>
<switchingcapType> l2sc </switchingcapType>
<encodingType> ethernet </encodingType>
<switchingCapabilitySpecificInfo>
<switchingCapabilitySpecificInfo_L2sc>
<interfaceMTU> 9000 </interfaceMTU>
<vlanRangeAvailability> 3727 </vlanRangeAvailability>
<suggestedVLANRange> 3727 </suggestedVLANRange>
<vlanTranslation> true </vlanTranslation>
</switchingCapabilitySpecificInfo_L2sc>
</switchingCapabilitySpecificInfo>
</switchingCapabilityDescriptor>
</link>
<nextHop> 3 </nextHop>
</hop>
<hop id="3">
<link id="urn:publicid:IDN+ion.internet2.edu+interface+rtr.atla:ge-10/3/2:protogeni">
<trafficEngineeringMetric> 10 </trafficEngineeringMetric>
<capacity> 20000.0 </capacity>
<switchingCapabilityDescriptor>
<switchingcapType> l2sc </switchingcapType>
<encodingType> ethernet </encodingType>
<switchingCapabilitySpecificInfo>
<switchingCapabilitySpecificInfo_L2sc>
<interfaceMTU> 9000 </interfaceMTU>
<vlanRangeAvailability> 805 </vlanRangeAvailability>
<suggestedVLANRange> 805 </suggestedVLANRange>
<vlanTranslation> true </vlanTranslation>
</switchingCapabilitySpecificInfo_L2sc>
</switchingCapabilitySpecificInfo>
</switchingCapabilityDescriptor>
</link>
<nextHop> 4 </nextHop>
</hop>
<hop id="4">
<link id="urn:publicid:IDN+emulab.net+interface+procurve-pgeni-atla:3.21">
<trafficEngineeringMetric> 10  </trafficEngineeringMetric>
<capacity> 20000  </capacity>
<switchingCapabilityDescriptor>
<switchingcapType> l2sc</switchingcapType>
<encodingType> ethernet</encodingType>
<switchingCapabilitySpecificInfo>
<switchingCapabilitySpecificInfo_L2sc>
<interfaceMTU> 1500</interfaceMTU>
<vlanRangeAvailability> 805</vlanRangeAvailability>
<suggestedVLANRange> 805</suggestedVLANRange>
<vlanTranslation> false</vlanTranslation>
</switchingCapabilitySpecificInfo_L2sc>
</switchingCapabilitySpecificInfo>
</switchingCapabilityDescriptor>
</link>
<nextHop> 5</nextHop>
</hop>
<hop id="5">
<link id="urn:publicid:IDN+emulab.net+interface+procurveA:3.19">
<trafficEngineeringMetric> 10  </trafficEngineeringMetric>
<capacity> 20000  </capacity>
<switchingCapabilityDescriptor>
<switchingcapType> l2sc</switchingcapType>
<encodingType> ethernet</encodingType>
<switchingCapabilitySpecificInfo>
<switchingCapabilitySpecificInfo_L2sc>
<interfaceMTU> 1500</interfaceMTU>
<vlanRangeAvailability> 805</vlanRangeAvailability>
<suggestedVLANRange> 805</suggestedVLANRange>
<vlanTranslation> false</vlanTranslation>
</switchingCapabilitySpecificInfo_L2sc>
</switchingCapabilitySpecificInfo>
</switchingCapabilityDescriptor>
</link>
<nextHop> 6</nextHop>
</hop>
<hop id="6">
<link id="urn:publicid:IDN+utah.geniracks.net+interface+procurve2:1.19">
<trafficEngineeringMetric> 10  </trafficEngineeringMetric>
<capacity> 20000  </capacity>
<switchingCapabilityDescriptor>
<switchingcapType> l2sc</switchingcapType>
<encodingType> ethernet</encodingType>
<switchingCapabilitySpecificInfo>
<switchingCapabilitySpecificInfo_L2sc>
<interfaceMTU> 1500</interfaceMTU>
<vlanRangeAvailability> 805</vlanRangeAvailability>
<suggestedVLANRange> 805</suggestedVLANRange>
<vlanTranslation> false</vlanTranslation>
</switchingCapabilitySpecificInfo_L2sc>
</switchingCapabilitySpecificInfo>
</switchingCapabilityDescriptor>
</link>
<nextHop> null</nextHop>
</hop>
</path>
</stitching>

In the stitching definition:

  • <hop id="1"> is for the GPO InstaGENI switch (procurve2:5.24.ion.newy.ae0) and uses VLAN 3727.
  • <hop id="2"> is for the Internet2 router in New York (rtr.newy:ae0:bbn) and uses VLAN 3727.
  • <hop id="3"> is for the Internet2 router in Atlanta (rtr.atla:ge-10/3/2:protogeni) and uses VLAN 805
  • <hop id="4"> is for PG Utah switch that connects PG Utah to ION (procurve-pgeni-atla:3.21) and uses VLAN 805
  • <hop id="5"> is for the Utah PG switch that connects to Utah IG (procurveA:3.19) and uses VLAN 805
  • <hop id="6"> is for the Utah InstaGENI switch (procurve2:1.19) and uses VLAN 805

Also notice that each of the hops show the bandwidth that was originally requested:

   <capacity> 20000 </capacity>

At this point the experiment has been set up, and you may determine the login information at each compute aggregate:

$ readyToLogin.py gpo2utah -a ig-gpo
...
================================================================================
LOGIN INFO for AM: https://boss.instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am/2.0
================================================================================

For more login info, see the section entitled:
	 'Providing a private key to ssh' in 'readyToLogin.py -h'

server-gpo's geni_status is: ready (am_status:ready) 
User lnevers logs in to server-gpo using:
	ssh -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@pcvm1-26.instageni.gpolab.bbn.com



$ readyToLogin.py gpo2utah -a ig-utah
...

================================================================================
LOGIN INFO for AM: https://boss.utah.geniracks.net:12369/protogeni/xmlrpc/am/2.0
================================================================================

For more login info, see the section entitled:
	 'Providing a private key to ssh' in 'readyToLogin.py -h'

client-utah's geni_status is: ready (am_status:ready)
User lnevers logs in to client-utah using:
	ssh -p 30522  -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@pc1.utah.geniracks.net

We will use the server-gpo login to determine the routable IP address that was requested for the Web Server:

$ ssh -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@pcvm1-26.instageni.gpolab.bbn.com
[lnevers@server-gpo ~]$  /sbin/ifconfig
eth999    Link encap:Ethernet  HWaddr 02:84:C8:6D:99:DA  
          inet addr:192.1.242.172  Bcast:192.1.242.255  Mask:255.255.255.128  <=(a)
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:24503 errors:0 dropped:0 overruns:0 frame:0
          TX packets:10400 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:25757338 (24.5 MiB)  TX bytes:557996 (544.9 KiB)

eth3727   Link encap:Ethernet  HWaddr 02:09:15:E4:32:ED  
          inet addr:192.168.4.1  Bcast:192.168.4.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:19288 errors:0 dropped:0 overruns:0 frame:0
          TX packets:13587 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:27216849 (25.9 MiB)  TX bytes:1598817 (1.5 MiB)

(a) publically routable IP address is 192.1.242.161

We can now connect to the web server to verify that iperf tests were run and that the client completed wget operations:


Select the Web Server Statistics link, and verify that wget was sucessfull for images:


Next, select the Iperf Server Logs link to verify that the iperf test completed:

Before starting the PingPlus portion of the experiment, verify that no information existing for PingPlus by selecting the link Layer 2 PingPlus Statistics .

Now we start the Layer 2 PingPlus experiment, by logging into client-utah, determining its dataplane MAC address and then shutting down IP on that interface:

$ ssh -p 32826  -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@pc1.utah.geniracks.net
Last login: Fri Jan 31 09:05:19 2014 from sendaria.gpolab.bbn.com
[lnevers@client-utah ~]$  /sbin/ifconfig
eth978    Link encap:Ethernet  HWaddr 02:42:BF:A4:A5:45  
          inet addr:192.168.4.2  Bcast:192.168.4.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:11704 errors:0 dropped:0 overruns:0 frame:0
          TX packets:19519 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:626218 (611.5 KiB)  TX bytes:28038324 (26.7 MiB)

eth999    Link encap:Ethernet  HWaddr 02:5C:66:E8:6B:AE  
          inet addr:172.17.1.7  Bcast:172.31.255.255  Mask:255.240.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:18192 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9638 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:23002760 (21.9 MiB)  TX bytes:582428 (568.7 KiB)


[lnevers@client-utah ~]$ sudo /sbin/ifconfig eth973 0
[lnevers@client-utah ~]$ 

Now on the server, shutdown IP on the dataplane interface and run Layer 2 !PingPLus:

[lnevers@server-gpo ~]$ sudo /sbin/ifconfig eth3711 0
[lnevers@server-gpo ~]$ sudo /local/scripts/runPingPlus.sh 02:42:BF:A4:A5:45 eth3711
[lnevers@server-gpo ~]$

Now, we go back to the web server and select the Layer 2 PingPlus Statistics link. You will find new statistics:

We have now completed the experiment.

3. Finish the Stitching Experiment

Finish

Now that the experiment is completed, we release all resources:

$ stitcher deletesliver gpo2utah
11:28:11 INFO     stitcher: Loading agg_nick_cache file '/home/lnevers/.gcf/agg_nick_cache'
11:28:11 INFO     stitcher: Loading config file /home/lnevers/.gcf/omni_config
11:28:11 INFO     stitcher: Using control framework portal
11:28:11 INFO     stitcher: Reading stitching slice gpo2utah aggregates from file ch\.geni\.net.ln-prj.gpo2utah-amlist.txt
11:28:11 INFO     stitcher: Adding aggregate option https://boss.utah.geniracks.net:12369/protogeni/xmlrpc/am (urn:publicid:IDN+utah.geniracks.net+authority+cm)
11:28:11 INFO     stitcher: Adding aggregate option https://boss.instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am (urn:publicid:IDN+instageni.gpolab.bbn.com+authority+cm)
11:28:11 INFO     stitcher: Adding aggregate option https://www.emulab.net:12369/protogeni/xmlrpc/am (urn:publicid:IDN+emulab.net+authority+cm)
11:28:11 INFO     stitcher: Adding aggregate option http://geni-am.net.internet2.edu:12346 (urn:publicid:IDN+ion.internet2.edu+authority+am)
11:28:11 INFO     omni: Loading agg_nick_cache file '/home/lnevers/.gcf/agg_nick_cache'
11:28:11 INFO     omni: Loading config file /home/lnevers/.gcf/omni_config
11:28:11 INFO     omni: Using control framework portal
11:28:12 INFO     omni: Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+gpo2utah expires on 2014-02-07 15:15:42 UTC
11:29:13 INFO     omni:  (PG log url - look here for details on any failures: https://boss.utah.geniracks.net/spewlogfile.php3?logfile=5522324d8d14f2c9aa950314d5d86881)
11:29:13 INFO     omni: Deleted sliver urn:publicid:IDN+ch.geni.net:ln-prj+slice+gpo2utah on unspecified_AM_URN at https://boss.utah.geniracks.net:12369/protogeni/xmlrpc/am
11:30:04 INFO     omni:  (PG log url - look here for details on any failures: https://www.emulab.net/spewlogfile.php3?logfile=12a477823c87d0e4f6417a15c780788f)
11:30:04 INFO     omni: Deleted sliver urn:publicid:IDN+ch.geni.net:ln-prj+slice+gpo2utah on unspecified_AM_URN++ at https://www.emulab.net:12369/protogeni/xmlrpc/am
11:30:08 INFO     omni: Deleted sliver urn:publicid:IDN+ch.geni.net:ln-prj+slice+gpo2utah on unspecified_AM_URN+++ at http://geni-am.net.internet2.edu:12346
11:31:01 INFO     omni:  (PG log url - look here for details on any failures: https://boss.instageni.gpolab.bbn.com/spewlogfile.php3?logfile=ba4ddf3cc03e020a56f6e325ed0cc9ae)
11:31:01 INFO     omni: Deleted sliver urn:publicid:IDN+ch.geni.net:ln-prj+slice+gpo2utah on unspecified_AM_URN+ at https://boss.instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am
Deleted slivers on 4 out of a possible 4 aggregates

Stitcher uses the files ch\.geni\.net.ln-prj.gpo2utah-amlist.txt which has the AM list associated with the slice. Note, that is this file does not exist in the current working directory, stitcher will try to delete the sliver at all know aggregates, but you will still have to delete the sliver manually at the Internet2/ION aggregate (http://geni-am.net.internet2.edu:12346). Now the sliver has been deleted and compute and network resources are released at all aggregates.

Additional Stitching Scenarios

Additional stitching scenarios have been tested and some example request RSpecs can be used as a reference at this location.

If additional help is needed please contact help@geni.net.

Attachments (6)