Changes between Version 39 and Version 40 of GENIRacksHome/InstageniRacks/AcceptanceTestStatus/IG-MON-3


Ignore:
Timestamp:
03/06/13 13:00:30 (7 years ago)
Author:
lnevers@bbn.com
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • GENIRacksHome/InstageniRacks/AcceptanceTestStatus/IG-MON-3

    v39 v40  
    754754
    755755
    756 }}}
    757 
    758 As administrator, log into the OpenVZ host pc2.instageni.gpolab.bbn.com
    759 
    760 
    761 
    762 
    763 
    764 
    765  * Now, use the OpenVZ host itself to view activity:
    766    * As an admin, login to pc5.utah.geniracks.net
    767    * Poking around, i was led to a couple of prospective data sources:
    768      * Logs in `/var/emulab`
    769      * The `vzctl` RPM, containing a number of OpenVZ control commands
    770    * The latter seems to give a list of running VMs easily:
    771 {{{
    772 vhost1,[/var/emulab],05:00(1)$ sudo vzlist -a
     756As administrator, log into the OpenVZ host pc2 and determine list of VMs running:
     757
     758{{{
     759[lnevers@boss ~]$ ssh pc2
     760Last login: Wed Feb  6 15:10:36 2013 from boss.instageni.gpolab.bbn.com
     761[lnevers@vhost2 ~]$
     762[lnevers@vhost2 ~]$ sudo vzlist -a
    773763      CTID      NPROC STATUS    IP_ADDR         HOSTNAME
    774          1         15 running   -               virt1.ecgtest.pgeni-gpolab-bbn-com.utah.geniracks.net
    775 }}}
    776    * I also see a command to figure out which container is running a given PID.  Suppose i run top and am concerned about an sshd process chewing up all system CPU:
    777 {{{
    778     PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND         
    779   51817 20001     20   0  116m 3780  872 R 94.4  0.0   0:05.74 sshd             
    780 }}}
    781    * Since the user is numeric, i can assume this process is probably running in a container, so find out which one:
    782 {{{
    783 vhost1,[/var/emulab],05:05(0)$ sudo vzpid 51766
    784 Pid     CTID    Name
    785 51766   1       sshd
    786 chaos      51804   51163  0 05:04 pts/0    00:00:00 grep --color=auto ssh
    787 }}}
    788    * and then look up the container info as above.
    789    * The files in `/var/emulab` give details about how each experiment was created.  In particular:
    790 {{{
    791 Information about experiment startup attributes:
    792   /var/emulab/boot/tmcc.pcvm5-1/
    793   /var/emulab/boot/tmcc.pcvm5-2/
    794 
    795 Logs of experiment progress:
    796   /var/emulab/logs/tbvnode-pcvm5-1.log
    797   /var/emulab/logs/tbvnode-pcvm5-2.log
    798   /var/emulab/logs/tmccproxy.pcvm5-1.log
    799   /var/emulab/logs/tmccproxy.pcvm5-2.log
    800 }}}
    801    * These may be useful for running and terminated experiments ''if'' the context IDs are unique.
    802 
    803 === Results of testing step 4: 2012-05-21 ===
    804 
    805  * Per-host view of current state:
    806    * From [https://boss.utah.geniracks.net/nodecontrol_list.php3?showtype=dl360] in red dot mode, i can once again see that pc4 is allocated as phys1 to `pgeni-gpolab-bbn-com/ecgtest`.
    807    * I can see that pc1 and pc3 are configured as OpenVZ shared hosts, but i can't see what experiments they are running.
    808  * Per-experiment view of current state:
    809    * Browse to [https://boss.utah.geniracks.net/genislices.php] and find one slice running on the Component Manager:
    810 {{{
    811 ID   HRN                         Created             Expires
    812 535  bbn-pgeni.ecgtest (ecgtest) 2012-05-21 13:19:28 2012-05-22 10:02:36
    813 }}}
    814    * Click `(ecgtest)` to view the details of that experiment at [https://boss.utah.geniracks.net/showexp.php3?experiment=536#details].
    815    * This shows what nodes it's using, including that its VM has been put on pc3:
    816 {{{
    817 Physical Node Mapping:
    818 ID              Type         OS              Physical
    819 --------------- ------------ --------------- ------------
    820 phys1           dl360        FEDORA15-STD    pc4
    821 virt1           pcvm         OPENVZ-STD      pcvm3-1 (pc3)
    822 }}}
    823    * Here are some other interesting things, all of which are similar to Friday's test:
    824 {{{
    825 IP Port allocation:
    826 Low             High
    827 --------------- ------------
    828 30000           30255
    829 
    830 SSHD Port allocation ('ssh -p portnum'):
    831 ID              Port       SSH command
    832 --------------- ---------- ----------------------
    833 
    834 Physical Lan/Link Mapping:
    835 ID              Member          IP              MAC                  NodeID
    836 --------------- --------------- --------------- -------------------- ---------
    837 phys1-virt1-0   phys1:0         10.10.1.1       e8:39:35:b1:ec:9e    pc4
    838                                                 1/1 <-> 1/37         procurve2
    839 phys1-virt1-0   virt1:0         10.10.1.2                            pcvm3-1
    840 }}}
    841  * Now, use the OpenVZ host itself to view activity:
    842    * As an admin, login to pc3.utah.geniracks.net
    843    * Everything seems similar to when i looked Friday:
    844 {{{
    845 vhost2,[~],13:57(0)$ sudo vzlist -a
    846       CTID      NPROC STATUS    IP_ADDR         HOSTNAME
    847          1         19 running   -               virt1.ecgtest.pgeni-gpolab-bbn-com.utah.geniracks.net
    848 }}}
    849 
    850 === Results of testing step 4: 2012-05-26 ===
    851 
    852  * Per-host view of current state:
    853    * From [https://boss.utah.geniracks.net/nodecontrol_list.php3?showtype=dl360] in red dot mode, i can once again see that pc2 is allocated as phys1 to `pgeni-gpolab-bbn-com/ecgtest`.
    854    * I can see that pc3 and pc5 are configured as OpenVZ shared hosts, but i can't see what experiments they are running.
    855    * Using [https://boss.utah.geniracks.net/showpool.php], i can see that pc3 is running one VM and pc5 is running zero, but not what experiments each is running.  I opened [instaticket:35] to ask whether a node-to-experiment mapping would be an easy modification to `showpool.php`.
    856  * Per-experiment view of current state:
    857    * Browse to [https://boss.utah.geniracks.net/genislices.php] and find two slices running on the Component Manager:
    858 {{{
    859 ID   HRN                                 Created             Expires
    860 949  bbn-pgeni.lnubuntu12b (lnubuntu12b) 2012-05-25 09:54:08 2012-05-29 18:00:00
    861 951  bbn-pgeni.ecgtest (ecgtest)         2012-05-26 05:30:20 2012-06-29 18:00:00
    862 }}}
    863    * Click `(ecgtest)` to view the details of that experiment at [https://boss.utah.geniracks.net/showexp.php3?experiment=952#details].
    864    * This shows what nodes it's using, including that its VM has been put on pc3:
    865 {{{
    866 Physical Node Mapping:
    867 ID              Type         OS              Physical
    868 --------------- ------------ --------------- ------------
    869 phys1           dl360        FEDORA15-STD    pc2
    870 virt1           pcvm         OPENVZ-STD      pcvm3-1 (pc3)
    871 }}}
    872    * Here are some other interesting things:
    873 {{{
    874 IP Port allocation:
    875 Low             High
    876 --------------- ------------
    877 30000           30255
    878 
    879 SSHD Port allocation ('ssh -p portnum'):
    880 ID              Port       SSH command
    881 --------------- ---------- ----------------------
    882 
    883 Physical Lan/Link Mapping:
    884 ID              Member          IP              MAC                  NodeID
    885 --------------- --------------- --------------- -------------------- ---------
    886 phys1-virt1-0   phys1:0         10.10.1.1       e8:39:35:b1:0c:7e    pc2
    887                                                 1/1 <-> 1/28         procurve2
    888 phys1-virt1-0   virt1:0         10.10.1.2       00:00:0a:0a:01:02    pcvm3-1
    889 }}}
    890  * So, indeed, a MAC address is reported for the virtual node.  However, virt1 itself still says:
    891 {{{
    892 mv1.1     Link encap:Ethernet  HWaddr 82:01:0A:0A:01:02 
    893           inet addr:10.10.1.2  Bcast:10.10.1.255  Mask:255.255.255.0
    894 }}}
    895  * Now, use the OpenVZ host itself to view activity:
    896    * As an admin, login to pc3.utah.geniracks.net
    897    * Everything seems similar to when i looked Friday:
    898 {{{
    899 vhost2,[~],06:00(0)$ sudo vzlist -a
    900       CTID      NPROC STATUS    IP_ADDR         HOSTNAME
    901          1         19 running   -               virt1.ecgtest.pgeni-gpolab-bbn-com.utah.geniracks.net
    902 }}}
    903 
    904 Earlier, i said:
    905  * Using [https://boss.utah.geniracks.net/showpool.php], i can see that pc3 is running one VM and pc5 is running zero, but not what experiments each is running.  I opened [instaticket:35] to ask whether a node-to-experiment mapping would be an easy modification to `showpool.php`.
    906  * Leigh responded that this is already at the bottom of the node UI, see e.g. [https://boss.utah.geniracks.net/shownode.php3?node_id=pc3] (which you can get to by clicking through from showpool.php.  So this is good for our purposes.
     764         1         16 running   -               VM-1.EG-MON-3.pgeni-gpolab-bbn-com.instageni.gpolab.bbn.com
     765         2         16 running   -               VM-1.ln-pgeni-cred.pgeni-gpolab-bbn-com.instageni.gpolab.bbn.com
     766         3         37 running   -               gpo-ig.IG-CT-4.pgeni-gpolab-bbn-com.instageni.gpolab.bbn.com
     767         4         25 running   -               gpo-ig.SHORT-IG-CT-4.pgeni-gpolab-bbn-com.instageni.gpolab.bbn.com
     768         5         16 running   -               gpo-ig2.SHORT-IG-CT-4.pgeni-gpolab-bbn-com.instageni.gpolab.bbn.com
     769         6         22 running   -               VM-2.EG-MON-3.pgeni-gpolab-bbn-com.instageni.gpolab.bbn.com
     770         7         22 running   -               gpo-ig2.IG-CT-4.pgeni-gpolab-bbn-com.instageni.gpolab.bbn.com
     771         8         23 running   -               server.portal01.panther.instageni.gpolab.bbn.com
     772         9         16 running   -               gpo-ig.IG-MON-3-OF.pgeni-gpolab-bbn-com.instageni.gpolab.bbn.com
     773        10         22 running   -               gpo-ig2.IG-MON-3-OF.pgeni-gpolab-bbn-com.instageni.gpolab.bbn.com
     774[lnevers@vhost2 ~]$
     775}}}
     776
     777While logged in to the OpenVZ host pc2, determine which container is running a given PID:
     778{{{
     779   UID          PID    PPID  C STIME TTY          TIME CMD
     780   20001      15017   13395  0 10:07 ?        00:00:07 /usr/bin/iperf -s
     781
     782  [lnevers@vhost2 ~]$ sudo vzpid 15017
     783  Pid   CTID    Name
     784  15017 6       iperf
     785}}}
     786
     787From the container list we can see that the CID 6 is hostname VM-2.EG-MON-3.pgeni-gpolab-bbn-com.instageni.gpolab.bbn.com.
     788
     789Look up container experiment information, for example pc-vm2-6 (CID 6), see start-up attributes in ''/var/emulab/boot/tmcc.pcvm2-5''.
     790{{{
     791[lnevers@vhost2 ~]$ ls /var/emulab/boot/tmcc.pcvm2-5
     792accounts  creator    disks         ifconfig    linkdelay  mounts          nodeuuid    programs  startupcmd  tiptunnels  traceinfo  userenv
     793blobs     delay      eventkey      jailconfig  manifest   nodeattributes  nseconfigs  routing   syncserver  tpmblob     trafgens   watchdoginfo
     794bridges   dhcpdconf  firewallinfo  keyhash     motelog    nodeid          plabconfig  rpms      tarballs    tpmpubkey   tunnels
     795[lnevers@vhost2 ~]$ more /var/emulab/boot/tmcc.pcvm2-5/ifconfig
     796INTERFACE IFACETYPE=vlan INET=10.42.13.42 MASK=255.255.255.0 ID=11 VMAC=02907d4e0e71 PMAC=10604B9B8216 RTABID=200 ENCAPSULATE=0 LAN=openflow-mesoscale-1 VT
     797AG=1750
     798}}}
     799
     800Also review container for experiment logs:
     801{{{
     802[lnevers@vhost2 ~]$ ls /var/emulab/logs
     803bootsetup.debug          tbvnode-pcvm2-10.all  tbvnode-pcvm2-14.log  tbvnode-pcvm2-4.log  tbvnode-pcvm2-9.all     tmccproxy.pcvm2-2.log
     804dhclient-enter.log       tbvnode-pcvm2-10.log  tbvnode-pcvm2-15.log  tbvnode-pcvm2-5.all  tbvnode-pcvm2-9.log     tmccproxy.pcvm2-3.log
     805dhclient-exit.log        tbvnode-pcvm2-11.all  tbvnode-pcvm2-1.all   tbvnode-pcvm2-5.log  tmccproxy.pcvm2-10.log  tmccproxy.pcvm2-4.log
     806dhclient.log             tbvnode-pcvm2-11.log  tbvnode-pcvm2-1.log   tbvnode-pcvm2-6.all  tmccproxy.pcvm2-11.log  tmccproxy.pcvm2-5.log
     807emulab-systemd-swap.log  tbvnode-pcvm2-12.all  tbvnode-pcvm2-2.all   tbvnode-pcvm2-6.log  tmccproxy.pcvm2-12.log  tmccproxy.pcvm2-6.log
     808emulab-watchdog.log      tbvnode-pcvm2-12.log  tbvnode-pcvm2-2.log   tbvnode-pcvm2-7.all  tmccproxy.pcvm2-13.log  tmccproxy.pcvm2-7.log
     809progagent.debug          tbvnode-pcvm2-13.all  tbvnode-pcvm2-3.all   tbvnode-pcvm2-7.log  tmccproxy.pcvm2-14.log  tmccproxy.pcvm2-8.log
     810progwrap.debug           tbvnode-pcvm2-13.log  tbvnode-pcvm2-3.log   tbvnode-pcvm2-8.all  tmccproxy.pcvm2-15.log  tmccproxy.pcvm2-9.log
     811prunelvms                tbvnode-pcvm2-14.all  tbvnode-pcvm2-4.all   tbvnode-pcvm2-8.log  tmccproxy.pcvm2-1.log
     812
     813[lnevers@vhost2 ~]$ ls -l /var/emulab/logs/tbvnode-pcvm2-6*
     814-rw-r--r-- 1 root root 433483 Mar  6 10:02 /var/emulab/logs/tbvnode-pcvm2-6.all
     815-rw-r--r-- 1 root root  13756 Mar  6 10:02 /var/emulab/logs/tbvnode-pcvm2-6.log
     816[lnevers@vhost2 ~]$
     817
     818[lnevers@vhost2 ~]$ more /var/emulab/logs/tbvnode-pcvm2-6.log
     819TIMESTAMP: 10:02:23:478087 vnodesetup starting the real work
     820TIMESTAMP: 10:02:23:487690 vnodesetup calling into libsetup
     821Checking Testbed generic VM configuration ...
     822TIMESTAMP: 10:02:23:542273 vnodesetup back from libsetup
     823TIMESTAMP: 10:02:23:580189 vnodesetup (12570) waiting on vnode (12595)
     824TIMESTAMP: 10:02:23:580331 vnodesetup (12595) execing mkvnode.pl
     825libvnode: debug=1
     826TIMESTAMP: 03/06/2013 10:02:23:764243 starting openvz rootPreConfig()
     827TIMESTAMP: 03/06/2013 10:02:23:764494 finished openvz rootPreConfig()
     828GENVNODETYPE openvz
     829libops:
     830$VAR1 = 'openvz';
     831$VAR2 = {
     832          'vnodeUnmount' => sub { "DUMMY" },
     833          'vnodeDestroy' => sub { "DUMMY" },
     834          'init' => sub { "DUMMY" },
     835          'rootPostConfig' => sub { "DUMMY" },
     836          'vnodeCreate' => sub { "DUMMY" },
     837          'vnodeConfigResources' => sub { "DUMMY" },
     838          'setDebug' => sub { "DUMMY" },
     839          'vnodePreConfig' => sub { "DUMMY" },
     840          'vnodeConfigDevices' => sub { "DUMMY" },
     841          'vnodeReboot' => sub { "DUMMY" },
     842          'vnodeState' => sub { "DUMMY" },
     843          'vnodeExec' => sub { "DUMMY" },
     844          'rootPreConfig' => sub { "DUMMY" },
     845          'rootPreConfigNetwork' => sub { "DUMMY" },
     846          'vnodePreConfigExpNetwork' => sub { "DUMMY" },
     847           ....
     848
     849}}}
     850
    907851
    908852== Step 5: get information about terminated experiments ==