BBN head not is not responding to AM API requests

[lnevers@bbn.com]

Attempts to access the BBN Head node are failing:

$ omni.py listresources -a exobbn
INFO:omni:Loading config file omni_config
INFO:omni:Using control framework pgeni
INFO:omni:Substituting AM nickname exobbn with URL https://bbn-hn.exogeni.net:11443/orca/xmlrpc, URN unspecified_AM_URN
ERROR:omni.protogeni:Call for List Resources at https://bbn-hn.exogeni.net:11443/orca/xmlrpc failed.: Unknown socket error: [Errno 111] Connection refused
ERROR:omni.protogeni:    ..... Run with --debug for more information
INFO:omni:Listed resources on 0 out of 1 possible aggregates.
INFO:omni:Got no resources. No resources from AM https://bbn-hn.exogeni.net:11443/orca/xmlrpc: [Errno 111] Connection refused
INFO:omni: ------------------------------------------------------------
INFO:omni: Completed listresources:

  Options as run:
		aggregate: exobbn
		framework: pgeni
		native: True

  Args: listresources

  Result Summary: Got no resources. No resources from AM https://bbn-hn.exogeni.net:11443/orca/xmlrpc: [Errno 111] Connection refused 
INFO:omni: ============================================================

[jbs@bbn.com]

It looks like FOAM is up, but FlowVisor? is down:

[10:09:52] jbs@jericho:/home/jbs
+$ omni -a https://bbn-hn.exogeni.gpolab.bbn.com:3626/foam/gapi/1 listresources
INFO:omni:Loading config file /home/jbs/.gcf/omni_config
INFO:omni:Using control framework gpolab
ERROR:omni.protogeni:Call for List Resources at https://bbn-hn.exogeni.gpolab.bbn.com:3626/foam/gapi/1 failed. Server says: <Fault 1: "<type 'exceptions.Exception'>:[Errno 111] Connection refused">
INFO:omni:Listed resources on 0 out of 1 possible aggregates.
INFO:omni:Got no resources. No resources from AM https://bbn-hn.exogeni.gpolab.bbn.com:3626/foam/gapi/1: <Fault 1: "<type 'exceptions.Exception'>:[Errno 111] Connection refused">
INFO:omni: ------------------------------------------------------------
INFO:omni: Completed listresources:

  Options as run:
                aggregate: https://bbn-hn.exogeni.gpolab.bbn.com:3626/foam/gapi/1
                framework: gpolab
                native: True

  Args: listresources

  Result Summary: Got no resources. No resources from AM https://bbn-hn.exogeni.gpolab.bbn.com:3626/foam/gapi/1: <Fault 1: "<type 'exceptions.Exception'>:[Errno 111] Connection refused"> 
INFO:omni: ============================================================

and

[10:11:13] jbs@bbn-hn:/home/jbs
+$ ps -ef | grep flowv
jbs      14018 13915  0 10:11 pts/4    00:00:00 grep flowv

I haven't investigated beyond that.

[lnevers@bbn.com]

Head node behaviour is changing, requesting listresources from ​https://bbn-hn.exogeni.net:11443/orca/xmlrpc now results in the Manifest header being reported and no resources, as shown her:

$ omni.py listresources -a exobbn   
INFO:omni:Loading config file omni_config
INFO:omni:Using control framework pgeni
INFO:omni:Substituting AM nickname exobbn with URL https://bbn-hn.exogeni.net:11443/orca/xmlrpc, URN unspecified_AM_URN
INFO:omni:Listed resources on 1 out of 1 possible aggregates.
INFO:omni:<?xml version="1.0" ?>
INFO:omni:<!-- Resources at AM:
	URN: unspecified_AM_URN
	URL: https://bbn-hn.exogeni.net:11443/orca/xmlrpc
 -->
INFO:omni:<rspec type="advertisement" xmlns="http://www.geni.net/resources/rspec/3" xmlns:ns2="http://hpn.east.isi.edu/rspec/ext/stitch/0.1/" xmlns:ns3="http://www.protogeni.net/resources/rspec/ext/emulab/1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.geni.net/resources/rspec/3 http://www.geni.net/resources/rspec/3/ad.xsd http://hpn.east.isi.edu/rspec/ext/stitch/0.1/ http://hpn.east.isi.edu/rspec/ext/stitch/0.1/stitch-schema.xsd http://www.protogeni.net/resources/rspec/ext/emulab/1 http://www.protogeni.net/resources/rspec/ext/emulab/1/ptop_extension.xsd"/>
INFO:omni: ------------------------------------------------------------
INFO:omni: Completed listresources:

  Options as run:
		aggregate: exobbn
		framework: pgeni
		native: True

  Args: listresources

  Result Summary: Retrieved resources from 1 aggregates.
Wrote rspecs from 1 aggregates. 
INFO:omni: ============================================================

[ibaldin@renci.org]

Something is torqued. We should probably close slices open on this sm and restart it.

[lnevers@bbn.com]

Ilia, do you want me to delete my slices? or are you offering to clear the existing slices/state?

[jbs@bbn.com]

The exogeni-ops folks were able to get things running again.