Custom Query (98 matches)

While executing IG-EXP-2 an experiment that has two concurrent slivers by two different users, lost ssh access to the nodes for one of the experiments.

Sequence of events that lead to the problem:

As user ​lnevers@bbn.com executed the following:

1. Create slice named "2vm"
2. Created sliver with 2 OpenVZ nodes using default image.
3. Logged into each of the assigned hosts, both on pc5, port 30011 and port 30010.
4. Ran some commands and left a ping running while executing 5,6, and 7 below.

As user ​lnever1@bbn.com executed the following:

5. Created a slice "lnubuntu12b"
6. Created a sliver with 2 ubuntu 12 custom images
7. Logged into each of the assigned hosts (pc1 and pc4) and ran some command

As user ​lnevers@bbn.com executed the following:

8. Exited the two ssh sessions from 3 above, to log back in and verify that key were still in place.
9. Tried to ssh back in, but connection is refused by the assigned hosts pc5, port 30011 and port 30010.

   lnevers@arendia:~$ ssh pc5.utah.geniracks.net -p 30010
   ssh: connect to host pc5.utah.geniracks.net port 30010: Connection refused
   lnevers@arendia:~$ ssh pc5.utah.geniracks.net -p 30011
   ssh: connect to host pc5.utah.geniracks.net port 30011: Connection refused
   

The user ​lnevers@bbn.com can no longer login into the active sliver nodes. Have the ​lnevers@bbn.com keys been overwritten?

Both experiment are left running.

How would a site admin figure out what experiments are running on a given shared node? It's definitely possible to login to the node and view logs, and that's not too bad. However, for physical nodes, i often make use of the web UI, in particular ​https://boss.utah.geniracks.net/nodecontrol_list.php3?showtype=dl360, which (in red dot mode) maps physical nodes to experiments.

Maybe ​https://boss.utah.geniracks.net/showpool.php should have an optional "shownode" argument, which would cause it to show a table of experiments on that node?

I think this would have some value, so the value-to-effort proposition would be good, especially if it's easy to do. But let me know if there's an alternative easy way to get this information via the web UI "given a physical host which is acting as a shared node, what experiments are using resources on that host?"

The full information about my experiment (from ​https://boss.utah.geniracks.net/showexp.php3?experiment=952#details) shows:

Virtual Node Info:
ID              Type         OS              Qualified Name
--------------- ------------ --------------- --------------------
phys1 (pc2)     pc           FEDORA15-STD    phys1.ecgtest.pgeni-gpolab-bbn-com.utah.geniracks.net
virt1 (pc3)     pcvm         OPENVZ-STD      virt1.ecgtest.pgeni-gpolab-bbn-com.utah.geniracks.net

Of those, the phys1 address is in DNS, and is a valid way to get into the node:

capybara,[~],08:19(0)$ host phys1.ecgtest.pgeni-gpolab-bbn-com.utah.geniracks.net
phys1.ecgtest.pgeni-gpolab-bbn-com.utah.geniracks.net is an alias for pc2.utah.geniracks.net.
pc2.utah.geniracks.net has address 155.98.34.12

However, the virt1 address is not in DNS:

capybara,[~],08:19(0)$ host virt1.ecgtest.pgeni-gpolab-bbn-com.utah.geniracks.net
Host virt1.ecgtest.pgeni-gpolab-bbn-com.utah.geniracks.net not found: 3(NXDOMAIN)

Should it be?