774 | | 1 15 running - virt1.ecgtest.pgeni-gpolab-bbn-com.utah.geniracks.net |
775 | | }}} |
776 | | * I also see a command to figure out which container is running a given PID. Suppose i run top and am concerned about an sshd process chewing up all system CPU: |
777 | | {{{ |
778 | | PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND |
779 | | 51817 20001 20 0 116m 3780 872 R 94.4 0.0 0:05.74 sshd |
780 | | }}} |
781 | | * Since the user is numeric, i can assume this process is probably running in a container, so find out which one: |
782 | | {{{ |
783 | | vhost1,[/var/emulab],05:05(0)$ sudo vzpid 51766 |
784 | | Pid CTID Name |
785 | | 51766 1 sshd |
786 | | chaos 51804 51163 0 05:04 pts/0 00:00:00 grep --color=auto ssh |
787 | | }}} |
788 | | * and then look up the container info as above. |
789 | | * The files in `/var/emulab` give details about how each experiment was created. In particular: |
790 | | {{{ |
791 | | Information about experiment startup attributes: |
792 | | /var/emulab/boot/tmcc.pcvm5-1/ |
793 | | /var/emulab/boot/tmcc.pcvm5-2/ |
794 | | |
795 | | Logs of experiment progress: |
796 | | /var/emulab/logs/tbvnode-pcvm5-1.log |
797 | | /var/emulab/logs/tbvnode-pcvm5-2.log |
798 | | /var/emulab/logs/tmccproxy.pcvm5-1.log |
799 | | /var/emulab/logs/tmccproxy.pcvm5-2.log |
800 | | }}} |
801 | | * These may be useful for running and terminated experiments ''if'' the context IDs are unique. |
802 | | |
803 | | === Results of testing step 4: 2012-05-21 === |
804 | | |
805 | | * Per-host view of current state: |
806 | | * From [https://boss.utah.geniracks.net/nodecontrol_list.php3?showtype=dl360] in red dot mode, i can once again see that pc4 is allocated as phys1 to `pgeni-gpolab-bbn-com/ecgtest`. |
807 | | * I can see that pc1 and pc3 are configured as OpenVZ shared hosts, but i can't see what experiments they are running. |
808 | | * Per-experiment view of current state: |
809 | | * Browse to [https://boss.utah.geniracks.net/genislices.php] and find one slice running on the Component Manager: |
810 | | {{{ |
811 | | ID HRN Created Expires |
812 | | 535 bbn-pgeni.ecgtest (ecgtest) 2012-05-21 13:19:28 2012-05-22 10:02:36 |
813 | | }}} |
814 | | * Click `(ecgtest)` to view the details of that experiment at [https://boss.utah.geniracks.net/showexp.php3?experiment=536#details]. |
815 | | * This shows what nodes it's using, including that its VM has been put on pc3: |
816 | | {{{ |
817 | | Physical Node Mapping: |
818 | | ID Type OS Physical |
819 | | --------------- ------------ --------------- ------------ |
820 | | phys1 dl360 FEDORA15-STD pc4 |
821 | | virt1 pcvm OPENVZ-STD pcvm3-1 (pc3) |
822 | | }}} |
823 | | * Here are some other interesting things, all of which are similar to Friday's test: |
824 | | {{{ |
825 | | IP Port allocation: |
826 | | Low High |
827 | | --------------- ------------ |
828 | | 30000 30255 |
829 | | |
830 | | SSHD Port allocation ('ssh -p portnum'): |
831 | | ID Port SSH command |
832 | | --------------- ---------- ---------------------- |
833 | | |
834 | | Physical Lan/Link Mapping: |
835 | | ID Member IP MAC NodeID |
836 | | --------------- --------------- --------------- -------------------- --------- |
837 | | phys1-virt1-0 phys1:0 10.10.1.1 e8:39:35:b1:ec:9e pc4 |
838 | | 1/1 <-> 1/37 procurve2 |
839 | | phys1-virt1-0 virt1:0 10.10.1.2 pcvm3-1 |
840 | | }}} |
841 | | * Now, use the OpenVZ host itself to view activity: |
842 | | * As an admin, login to pc3.utah.geniracks.net |
843 | | * Everything seems similar to when i looked Friday: |
844 | | {{{ |
845 | | vhost2,[~],13:57(0)$ sudo vzlist -a |
846 | | CTID NPROC STATUS IP_ADDR HOSTNAME |
847 | | 1 19 running - virt1.ecgtest.pgeni-gpolab-bbn-com.utah.geniracks.net |
848 | | }}} |
849 | | |
850 | | === Results of testing step 4: 2012-05-26 === |
851 | | |
852 | | * Per-host view of current state: |
853 | | * From [https://boss.utah.geniracks.net/nodecontrol_list.php3?showtype=dl360] in red dot mode, i can once again see that pc2 is allocated as phys1 to `pgeni-gpolab-bbn-com/ecgtest`. |
854 | | * I can see that pc3 and pc5 are configured as OpenVZ shared hosts, but i can't see what experiments they are running. |
855 | | * Using [https://boss.utah.geniracks.net/showpool.php], i can see that pc3 is running one VM and pc5 is running zero, but not what experiments each is running. I opened [instaticket:35] to ask whether a node-to-experiment mapping would be an easy modification to `showpool.php`. |
856 | | * Per-experiment view of current state: |
857 | | * Browse to [https://boss.utah.geniracks.net/genislices.php] and find two slices running on the Component Manager: |
858 | | {{{ |
859 | | ID HRN Created Expires |
860 | | 949 bbn-pgeni.lnubuntu12b (lnubuntu12b) 2012-05-25 09:54:08 2012-05-29 18:00:00 |
861 | | 951 bbn-pgeni.ecgtest (ecgtest) 2012-05-26 05:30:20 2012-06-29 18:00:00 |
862 | | }}} |
863 | | * Click `(ecgtest)` to view the details of that experiment at [https://boss.utah.geniracks.net/showexp.php3?experiment=952#details]. |
864 | | * This shows what nodes it's using, including that its VM has been put on pc3: |
865 | | {{{ |
866 | | Physical Node Mapping: |
867 | | ID Type OS Physical |
868 | | --------------- ------------ --------------- ------------ |
869 | | phys1 dl360 FEDORA15-STD pc2 |
870 | | virt1 pcvm OPENVZ-STD pcvm3-1 (pc3) |
871 | | }}} |
872 | | * Here are some other interesting things: |
873 | | {{{ |
874 | | IP Port allocation: |
875 | | Low High |
876 | | --------------- ------------ |
877 | | 30000 30255 |
878 | | |
879 | | SSHD Port allocation ('ssh -p portnum'): |
880 | | ID Port SSH command |
881 | | --------------- ---------- ---------------------- |
882 | | |
883 | | Physical Lan/Link Mapping: |
884 | | ID Member IP MAC NodeID |
885 | | --------------- --------------- --------------- -------------------- --------- |
886 | | phys1-virt1-0 phys1:0 10.10.1.1 e8:39:35:b1:0c:7e pc2 |
887 | | 1/1 <-> 1/28 procurve2 |
888 | | phys1-virt1-0 virt1:0 10.10.1.2 00:00:0a:0a:01:02 pcvm3-1 |
889 | | }}} |
890 | | * So, indeed, a MAC address is reported for the virtual node. However, virt1 itself still says: |
891 | | {{{ |
892 | | mv1.1 Link encap:Ethernet HWaddr 82:01:0A:0A:01:02 |
893 | | inet addr:10.10.1.2 Bcast:10.10.1.255 Mask:255.255.255.0 |
894 | | }}} |
895 | | * Now, use the OpenVZ host itself to view activity: |
896 | | * As an admin, login to pc3.utah.geniracks.net |
897 | | * Everything seems similar to when i looked Friday: |
898 | | {{{ |
899 | | vhost2,[~],06:00(0)$ sudo vzlist -a |
900 | | CTID NPROC STATUS IP_ADDR HOSTNAME |
901 | | 1 19 running - virt1.ecgtest.pgeni-gpolab-bbn-com.utah.geniracks.net |
902 | | }}} |
903 | | |
904 | | Earlier, i said: |
905 | | * Using [https://boss.utah.geniracks.net/showpool.php], i can see that pc3 is running one VM and pc5 is running zero, but not what experiments each is running. I opened [instaticket:35] to ask whether a node-to-experiment mapping would be an easy modification to `showpool.php`. |
906 | | * Leigh responded that this is already at the bottom of the node UI, see e.g. [https://boss.utah.geniracks.net/shownode.php3?node_id=pc3] (which you can get to by clicking through from showpool.php. So this is good for our purposes. |
| 764 | 1 16 running - VM-1.EG-MON-3.pgeni-gpolab-bbn-com.instageni.gpolab.bbn.com |
| 765 | 2 16 running - VM-1.ln-pgeni-cred.pgeni-gpolab-bbn-com.instageni.gpolab.bbn.com |
| 766 | 3 37 running - gpo-ig.IG-CT-4.pgeni-gpolab-bbn-com.instageni.gpolab.bbn.com |
| 767 | 4 25 running - gpo-ig.SHORT-IG-CT-4.pgeni-gpolab-bbn-com.instageni.gpolab.bbn.com |
| 768 | 5 16 running - gpo-ig2.SHORT-IG-CT-4.pgeni-gpolab-bbn-com.instageni.gpolab.bbn.com |
| 769 | 6 22 running - VM-2.EG-MON-3.pgeni-gpolab-bbn-com.instageni.gpolab.bbn.com |
| 770 | 7 22 running - gpo-ig2.IG-CT-4.pgeni-gpolab-bbn-com.instageni.gpolab.bbn.com |
| 771 | 8 23 running - server.portal01.panther.instageni.gpolab.bbn.com |
| 772 | 9 16 running - gpo-ig.IG-MON-3-OF.pgeni-gpolab-bbn-com.instageni.gpolab.bbn.com |
| 773 | 10 22 running - gpo-ig2.IG-MON-3-OF.pgeni-gpolab-bbn-com.instageni.gpolab.bbn.com |
| 774 | [lnevers@vhost2 ~]$ |
| 775 | }}} |
| 776 | |
| 777 | While logged in to the OpenVZ host pc2, determine which container is running a given PID: |
| 778 | {{{ |
| 779 | UID PID PPID C STIME TTY TIME CMD |
| 780 | 20001 15017 13395 0 10:07 ? 00:00:07 /usr/bin/iperf -s |
| 781 | |
| 782 | [lnevers@vhost2 ~]$ sudo vzpid 15017 |
| 783 | Pid CTID Name |
| 784 | 15017 6 iperf |
| 785 | }}} |
| 786 | |
| 787 | From the container list we can see that the CID 6 is hostname VM-2.EG-MON-3.pgeni-gpolab-bbn-com.instageni.gpolab.bbn.com. |
| 788 | |
| 789 | Look up container experiment information, for example pc-vm2-6 (CID 6), see start-up attributes in ''/var/emulab/boot/tmcc.pcvm2-5''. |
| 790 | {{{ |
| 791 | [lnevers@vhost2 ~]$ ls /var/emulab/boot/tmcc.pcvm2-5 |
| 792 | accounts creator disks ifconfig linkdelay mounts nodeuuid programs startupcmd tiptunnels traceinfo userenv |
| 793 | blobs delay eventkey jailconfig manifest nodeattributes nseconfigs routing syncserver tpmblob trafgens watchdoginfo |
| 794 | bridges dhcpdconf firewallinfo keyhash motelog nodeid plabconfig rpms tarballs tpmpubkey tunnels |
| 795 | [lnevers@vhost2 ~]$ more /var/emulab/boot/tmcc.pcvm2-5/ifconfig |
| 796 | INTERFACE IFACETYPE=vlan INET=10.42.13.42 MASK=255.255.255.0 ID=11 VMAC=02907d4e0e71 PMAC=10604B9B8216 RTABID=200 ENCAPSULATE=0 LAN=openflow-mesoscale-1 VT |
| 797 | AG=1750 |
| 798 | }}} |
| 799 | |
| 800 | Also review container for experiment logs: |
| 801 | {{{ |
| 802 | [lnevers@vhost2 ~]$ ls /var/emulab/logs |
| 803 | bootsetup.debug tbvnode-pcvm2-10.all tbvnode-pcvm2-14.log tbvnode-pcvm2-4.log tbvnode-pcvm2-9.all tmccproxy.pcvm2-2.log |
| 804 | dhclient-enter.log tbvnode-pcvm2-10.log tbvnode-pcvm2-15.log tbvnode-pcvm2-5.all tbvnode-pcvm2-9.log tmccproxy.pcvm2-3.log |
| 805 | dhclient-exit.log tbvnode-pcvm2-11.all tbvnode-pcvm2-1.all tbvnode-pcvm2-5.log tmccproxy.pcvm2-10.log tmccproxy.pcvm2-4.log |
| 806 | dhclient.log tbvnode-pcvm2-11.log tbvnode-pcvm2-1.log tbvnode-pcvm2-6.all tmccproxy.pcvm2-11.log tmccproxy.pcvm2-5.log |
| 807 | emulab-systemd-swap.log tbvnode-pcvm2-12.all tbvnode-pcvm2-2.all tbvnode-pcvm2-6.log tmccproxy.pcvm2-12.log tmccproxy.pcvm2-6.log |
| 808 | emulab-watchdog.log tbvnode-pcvm2-12.log tbvnode-pcvm2-2.log tbvnode-pcvm2-7.all tmccproxy.pcvm2-13.log tmccproxy.pcvm2-7.log |
| 809 | progagent.debug tbvnode-pcvm2-13.all tbvnode-pcvm2-3.all tbvnode-pcvm2-7.log tmccproxy.pcvm2-14.log tmccproxy.pcvm2-8.log |
| 810 | progwrap.debug tbvnode-pcvm2-13.log tbvnode-pcvm2-3.log tbvnode-pcvm2-8.all tmccproxy.pcvm2-15.log tmccproxy.pcvm2-9.log |
| 811 | prunelvms tbvnode-pcvm2-14.all tbvnode-pcvm2-4.all tbvnode-pcvm2-8.log tmccproxy.pcvm2-1.log |
| 812 | |
| 813 | [lnevers@vhost2 ~]$ ls -l /var/emulab/logs/tbvnode-pcvm2-6* |
| 814 | -rw-r--r-- 1 root root 433483 Mar 6 10:02 /var/emulab/logs/tbvnode-pcvm2-6.all |
| 815 | -rw-r--r-- 1 root root 13756 Mar 6 10:02 /var/emulab/logs/tbvnode-pcvm2-6.log |
| 816 | [lnevers@vhost2 ~]$ |
| 817 | |
| 818 | [lnevers@vhost2 ~]$ more /var/emulab/logs/tbvnode-pcvm2-6.log |
| 819 | TIMESTAMP: 10:02:23:478087 vnodesetup starting the real work |
| 820 | TIMESTAMP: 10:02:23:487690 vnodesetup calling into libsetup |
| 821 | Checking Testbed generic VM configuration ... |
| 822 | TIMESTAMP: 10:02:23:542273 vnodesetup back from libsetup |
| 823 | TIMESTAMP: 10:02:23:580189 vnodesetup (12570) waiting on vnode (12595) |
| 824 | TIMESTAMP: 10:02:23:580331 vnodesetup (12595) execing mkvnode.pl |
| 825 | libvnode: debug=1 |
| 826 | TIMESTAMP: 03/06/2013 10:02:23:764243 starting openvz rootPreConfig() |
| 827 | TIMESTAMP: 03/06/2013 10:02:23:764494 finished openvz rootPreConfig() |
| 828 | GENVNODETYPE openvz |
| 829 | libops: |
| 830 | $VAR1 = 'openvz'; |
| 831 | $VAR2 = { |
| 832 | 'vnodeUnmount' => sub { "DUMMY" }, |
| 833 | 'vnodeDestroy' => sub { "DUMMY" }, |
| 834 | 'init' => sub { "DUMMY" }, |
| 835 | 'rootPostConfig' => sub { "DUMMY" }, |
| 836 | 'vnodeCreate' => sub { "DUMMY" }, |
| 837 | 'vnodeConfigResources' => sub { "DUMMY" }, |
| 838 | 'setDebug' => sub { "DUMMY" }, |
| 839 | 'vnodePreConfig' => sub { "DUMMY" }, |
| 840 | 'vnodeConfigDevices' => sub { "DUMMY" }, |
| 841 | 'vnodeReboot' => sub { "DUMMY" }, |
| 842 | 'vnodeState' => sub { "DUMMY" }, |
| 843 | 'vnodeExec' => sub { "DUMMY" }, |
| 844 | 'rootPreConfig' => sub { "DUMMY" }, |
| 845 | 'rootPreConfigNetwork' => sub { "DUMMY" }, |
| 846 | 'vnodePreConfigExpNetwork' => sub { "DUMMY" }, |
| 847 | .... |
| 848 | |
| 849 | }}} |
| 850 | |