Opened 12 years ago

Closed 11 years ago

#31 closed (fixed)

how do i view control IP addresses for terminated VMs or control MAC addresses for any VMs?

Reported by: chaos@bbn.com Owned by: somebody
Priority: major Milestone: IG-MON-3
Component: Monitoring Version: SPIRAL4
Keywords: Cc:
Dependencies:

Description

In testing IG-MON-3 step 5, i found that the manifest cached for each ProtoGENI experiment reports dataplane IP addresses and MAC addresses (or at least it should do so fully once [instaticket:26] has been resolved). However, the only way i was easily able to find control IPs assigned to VMs was the interfaces table: 

mysql> select * from interfaces where IP="172.17.3.1";
+---------+------+------+--------------+------------+-----------+-------------+----------------+-------+------+---------------+--------+--------+----------+------+-------+--------------------------------------+---------+
| node_id | card | port | mac          | IP         | IPaliases | mask        | interface_type | iface | role | current_speed | duplex | rtabid | vnode_id | whol | trunk | uuid                                 | logical |
+---------+------+------+--------------+------------+-----------+-------------+----------------+-------+------+---------------+--------+--------+----------+------+-------+--------------------------------------+---------+
| pcvm3-1 |    0 |    1 | 000000000000 | 172.17.3.1 | NULL      | 255.240.0.0 | generic        | eth0  | ctrl | 0             | full   |      0 | NULL     |    0 |     0 | e5b9ad51-a379-11e1-af1c-00009b6224df |       1 | 
+---------+------+------+--------------+------------+-----------+-------------+----------------+-------+------+---------------+--------+--------+----------+------+-------+--------------------------------------+---------+
1 row in set (0.00 sec)

This table appears to contain IPs only for running experiments, and does not appear to contain real MAC addresses.

  1. How can i find what control IPs were assigned to what VMs belonging to now-terminated experiments?
  2. How can i find out MAC address assignments for these VMs?

Change History (11)

comment:1 Changed 12 years ago by chaos@bbn.com

From Leigh: 

> 1. How can i find what control IPs were assigned to what VMs belonging to     
> now-terminated experiments?

The historical info is in the node_history table. The issue is how              
to get you the info you want in a reasonably easy manner. I have been           
working in it, but there is still work to do. See:

https://boss.utah.geniracks.net/shownodehistory.php3?count=200&reverse=1        

Still needs work.

> 2. How can i find out MAC address assignments for these VMs?                  

We do not store any historical info for mac addresses.

I've realised i need to modify Step 5 somewhat, since as written it is looking up IP addresses associated with slivers, but the relevant requirement (D.7) / criterion (VII.18) is to start from an IP or VLAN and lookup sliver/experimenter information. So i will do that, and then revisit after the shownodehistory.php3 UI has been ironed out a bit more.

comment:2 Changed 12 years ago by chaos@bbn.com

I modified IG-MON-3 step 5 to more closely match an actual requirement, changing:

  • Determine any IP addresses assigned by InstaGENI to each VM in each terminated experiment.

to:

  • Given a control IP address which InstaGENI had assigned to a now-terminated VM, determine which experiment was given that control IP.
  • Given a data plane IP address which an experimenter had requested for a now-terminated VM, determine which experiment was given that IP.

Leigh also reported: 

Okay, I have worked on this again, and I think I have it working reasonable     
well. See:

        https://boss.utah.geniracks.net/shownodehistory.php3

Ignore the stuff before the search boxes ... The first box allows you to        
look at a specific point in time. If you are not looking at a specific          
node, then you will get a listing of what every node in the testbed was      
doing at that moment (except free nodes, no point in that). If you are          
looking at, say, pc5 and you type in a date, you will get a listing of what     
the pc5 did from that point on.

The next two search boxes are obvious; search for a specific node or an
IP.

There are some oddities that are difficult to explain. Suffice to say that      
the interface is somewhat determined by the fact that Utah's Emulab has 5.5     
million records, and we seem to be adding about a half million a year. We     
had to come up with a way to present the data in such a way that was            
responsive and not overwhelming. There are certainly much better ways to do     
this, but UI fanciness tends to fall low on the priority list.

I'll try this out sometime this weekend.

comment:3 Changed 12 years ago by ricci@cs.utah.edu

We've done this, including IP addresses, please check if it's to your satisfaction.

comment:4 Changed 12 years ago by chaos@bbn.com

Okay, here goes: i created a sliver containing some VMs, logged into one of the VMs, and ran ifconfig, getting: 

eth999    Link encap:Ethernet  HWaddr 02:31:56:AF:9B:B5
          inet addr:172.17.3.5  Bcast:172.31.255.255  Mask:255.240.0.0
          inet6 addr: fe80::31:56ff:feaf:9bb5/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:136 errors:0 dropped:0 overruns:0 frame:0
          TX packets:119 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:16157 (15.7 KiB)  TX bytes:10787 (10.5 KiB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:3 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:294 (294.0 b)  TX bytes:294 (294.0 b)

mv5.19    Link encap:Ethernet  HWaddr 02:2C:D2:18:8C:85
          inet addr:10.10.1.2  Bcast:10.10.1.255  Mask:255.255.255.0
          inet6 addr: fe80::2c:d2ff:fe18:8c85/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:5 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:188 (188.0 b)  TX bytes:0 (0.0 b)

Then i browsed to ​https://boss.utah.geniracks.net/shownodehistory.php3, and tried:

Node Pid Eid Slice Allocated By Allocation Date Duration
pcvm3-5 pgeni-gpolab-bbn-com ecgtest geniuser 2012-09-14 12:17:14 15m38s

I also tried a new sliver, containing one VM on which i asked for a public control IP, and got: 

$ host virt1.ecgtest.pgeni-gpolab-bbn-com.utah.geniracks.net
virt1.ecgtest.pgeni-gpolab-bbn-com.utah.geniracks.net is an alias for pcvm3-4.utah.geniracks.net.
pcvm3-4.utah.geniracks.net has address 155.98.34.130

On that node, i see: 

eth999    Link encap:Ethernet  HWaddr 02:69:21:AD:9B:76  
          inet addr:155.98.34.130  Bcast:155.98.34.255  Mask:255.255.255.0
          inet6 addr: fe80::69:21ff:fead:9b76/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:517 errors:0 dropped:0 overruns:0 frame:0
          TX packets:109 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:35250 (34.4 KiB)  TX bytes:9582 (9.3 KiB)

Summary to follow.

comment:5 Changed 12 years ago by chaos@bbn.com

Outstanding questions based on current state:

  • Searching for the control MAC addr seems to work, but the caveat is that the resulting table only gives a sliver name if the experiment is terminated --- it links to a page which exists only for a running experiment and not for a terminated one. Would it be possible to link to information about terminated experiments too, such as the URN of the creator or the rspec used or anything like that?
  • Searching for any of private control IP addr, public control IP addr, dataplane MAC address, or dataplane IP addr, seem to return no results. Which of those do you expect should return results?

comment:6 Changed 12 years ago by chaos@bbn.com

Updates based on Leigh's responses:

  • When there's a table of results, click on the green dot in the slice column to get information about terminated slivers, including the desired creator URN and rspec information.
  • Dataplane MAC and IP lookups are not supported.
  • Control plane public and private IP lookups should be supported:
    • Leigh and i both verified that we could successfully look up an IP, 172.17.3.4, which belonged to an active sliver at the time of lookup.
    • However, i believe that when an IP doesn't belong to any active sliver, it can't be looked up.
    • Leigh saw a bug with public IP lookups --- i'm not sure if it's the same bug or a different one, but i should verify both once that's fixed.

comment:7 Changed 12 years ago by chaos@bbn.com

Leigh, the lookups of private IPs seem to be working okay now.

Are the lookups of public IPs doing the right thing? If i browse to ​https://boss.utah.geniracks.net/shownodehistory.php3?IP=155.98.34.130&showall=&when=epoch&search3=Search, i don't get any address in the "search for IP" box, but instead it puts "pcvm5-4" in the "search for Node" box. My assumption is that it's doing that because, right now, the active DNS lookup is: 

$ host 155.98.34.130
130.34.98.155.in-addr.arpa is an alias for 130.6/9.34.98.155.in-addr.arpa.
130.6/9.34.98.155.in-addr.arpa domain name pointer pcvm5-4.utah.geniracks.net.

Is the IP address 155.98.34.130 always associated with the hostname pcvm5-4 (which is VM 4 on pc5)? If so, does that have implications we care about for use of public IPs? If not, this seems like a bug in this lookup, because i believe we actually want the history of the use of the IP 155.98.34.130, not the history of the hostname with which that IP happens to be temporarily associated right now.

Also, a very very minor thing: i think it would be more user-friendly to sort these entries in reverse chronological order (most recent first), rather than chronological order (oldest first). What do y'all think?

comment:8 Changed 12 years ago by chaos@bbn.com

Leigh fixed the public IP lookup bug, so that looks better now (as far as i can tell). However, the sort order is now very strange: when i browse to ​https://boss.utah.geniracks.net/shownodehistory.php3?IP=155.98.34.130&showall=&when=epoch&search3=Search, i get a bunch of records in an order i can't figure out at all --- maybe it's by node and then chronological or something?

So: can we sort this list, and what way should it be sorted? I still suggest reverse chronological (by allocation time, say), but am open to other things.

comment:9 Changed 11 years ago by chaos@bbn.com

My last question on this was whether the table given in shownodehistory.php3 could be sorted by some useful field, and i suggested reverse chronological order. The example i gave was ​https://boss.utah.geniracks.net/shownodehistory.php3?IP=155.98.34.130&showall=&when=epoch&search3=Search.

This is a minor request --- the functionality this ticket requested is all here --- but on the other hand, as these lists get long, it will become pretty difficult to find recent entries. So if this is easy, it'd be nice to have.

comment:10 in reply to:  9 Changed 11 years ago by chaos@bbn.com

Ping. I last asked:

My last question on this was whether the table given in shownodehistory.php3 could be sorted by some useful field, and i suggested reverse chronological order. The example i gave was ​https://boss.utah.geniracks.net/shownodehistory.php3?IP=155.98.34.130&showall=&when=epoch&search3=Search.

This is a minor request --- the functionality this ticket requested is all here --- but on the other hand, as these lists get long, it will become pretty difficult to find recent entries. So if this is easy, it'd be nice to have.

comment:11 Changed 11 years ago by chaos@bbn.com

Resolution: → fixed
Status: new → closed

Leigh's status update: 

Hi. The format of this table makes this more difficult then I had hoped. My     
quick try did not work well when I applied it to our 10+ years of history       
records, so I need to spend some real time looking at.

So, its on the list waiting till I have time to work on it.

No worries --- i was hoping it was easy, but, if it's not, then it'll happen when there's time.

Given that the major functionality this ticket requested is done, i'm going to go ahead and close it.

Note: See TracTickets for help on using tickets.