Changes between Version 14 and Version 15 of GENIRacksAdministration


Ignore:
Timestamp:
11/26/12 13:22:32 (11 years ago)
Author:
lnevers@bbn.com
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • GENIRacksAdministration

    v14 v15  
    33= GENI Racks Administration =
    44
    5 This page describes GENI racks administrative tasks and duties associated with each task.  For each rack type, a site contact coordinates delivery, installation, configuration, and maintenance of the rack.  In this very important role, you can rely on GPO support.   Please contact us at [mailto:help@geni.net] for any questions.  The GPO also provides a real-time public IRC chat room where engineers are often available, `chat.freenode.net` channel `#geni`, for debugging any issues you may encounter.  See [wiki:HowTo/ConnectToGENIChatRoom] for details.
     5This page describes GENI racks administrative tasks and duties associated with each GENI rack.  For each rack type, a site contact coordinates delivery, installation, configuration, and maintenance of the rack.  In this very important role, you can rely on GPO support.   Please contact us at [mailto:help@geni.net] for any questions.  The GPO also provides a real-time public IRC chat room where engineers are often available, `chat.freenode.net` channel `#geni`, for debugging any issues you may encounter.  See [wiki:HowTo/ConnectToGENIChatRoom] for details.
    66
    77== Site Requirements and Rack Installation ==
    8 The site contact works with the organization deploying the rack (GPO, RENCI or HP) to get get the rack (Starter, InstaGENI or ExoGENI, respectively) and site requirements for their specific site networks defined.  The site requirements include:
     8
     9The site contact works with the organization deploying the rack (GPO, RENCI or HP) to get get the rack (ExoGENI, InstaGENI, or Starter respectively) and site requirements for their specific site networks defined.  The site requirements include:
    910 * Network Setup - Define how the rack will connect to the Internet and to the GENI backbones. Ex Regional connections, connection speed, VLANs to be used, etc.
    1011 * Site Security Requirements- Determine engineering and procedures needed for rack connectivity, such as FlowVisor rules, IP filters, etc.
    … …  
    2021 * accessing compute and network resource consoles in the rack to support/manage experimenter resources or debug problems.
    2122 * ensure that security and usage procedures are followed.
     23
     24= ExoGENI Administration =
     25
     26ExoGENI rack administration tasks are to be defined and will be captured here when available.
     27
     28== Get ExoGENI rack Accounts ==
     29== Access Devices Consoles ==
     30== Monitoring ExoGENI rack Health ==
     31== Perform an experiment in your ExoGENI rack ==
     32== Install a VM image on your ExoGENI rack ==
     33== ExoGENI Racks Software/Firmware upgrades ==
     34----
    2235 
    23 = Starter Racks Administration =
    24 
    25 This section provides a few example of the administrative task on a Starter Rack.  Example administrative tasks for ExoGENI and InstaGENI racks are different, but will accomplish similar functions.
    26 
    27 == Get Starter rack Accounts ==
    28 
    29 '''Requesting an account'''
    30 
    31 Site operators should contact [mailto:gpo-infra@geni.net] to request sudo-capable login accounts on the Starter rack hosts by providing:
    32  * Preferred username
    33  * Preferred fullname
    34  * SSH public key for remote login
    35  * Hashed password for sudo obtained by running:
    36 {{{
    37 openssl passwd -1
    38 }}}
    39  and typing a password twice.  The resulting string should be of the form: `$1$xxxxxxxx$xxxxxxxxxxxxxxxxxxxxxx`
    40 
    41 ''' Policies for Unix account use '''
    42  * Remote account access is via public-key SSH only (no password-based login).
    43  * Do not run interactive sessions as root (don't use `sudo bash`, but instead run individual commands under sudo for logging).
    44  * Do not share account credentials.  We are happy to create individual accounts, or to give staffers who don't have logins access to our emergency account for outage debugging.
    45  * GPO staffers actively manage these systems using the puppet configuration management utility.  If you need to modify a system, please e-mail us at [mailto:gpo-infra@geni.net] to ensure that the desired change takes effect.
    46 
    47 '''Accounts on non-Unix rack devices'''
    48 
    49 Please contact [mailto:gpo-infra@geni.net] if you need login access to:
    50  * Control router or dataplane switch
    51  * IP KVM for remote console access
    52  * PDU for remote power control
    53 
    54 == Access Devices Consoles ==
    55 ''' Compute Resource consoles'''
    56  * The fold-out console in the rack can be used to view the consoles of any of the hosts in the rack.
    57  * The KVM hotkey for changing which device is displayed is `Ctrl Ctrl`.
    58 
    59 '''Network Devices Consoles'''
    60 The `monitor1` node in each rack can be used as a serial console for network devices located in that rack.
    61  * Login to `monitor1` using the console
    62  * Use screen to access the desired serial device, e.g.:
    63 {{{
    64 screen /dev/ttyS0
    65 }}}
    66  * When done using screen, kill the session by pressing: `Ctrl-a K`
    67 
    68 == Monitoring Starter rack Health ==
    69 
    70 '''Service Health'''
    71 
    72 GPO uses Nagios as a front-end for alerting about rack problems. The following services are monitored in the Starter Racks:
    73  * Resource problems with CPU, swap, or disk space on each host.
    74  * IP connectivity failures from the rack server to commodity internet (Google) and to the GPO lab.
    75  * Excessive CPU usage and excessive uplink broadcast traffic on the experimental switch.
    76  * Problems with standard experimental use of the Eucalyptus aggregate.
    77 
    78 The current state of monitored hosts and services at a given city can be viewed at:
    79  * [http://monitor.gpolab.bbn.com/nagios/cgi-bin/status.cgi?hostgroup=cities-cha&style=detail Chattanooga Status]
    80  * [http://monitor.gpolab.bbn.com/nagios/cgi-bin/status.cgi?hostgroup=cities-cle&style=detail Cleveland Status]
    81 
    82 If you would like to be added to any of these notifications, please contact us at [mailto:gpo-infra@geni.net].
    83 
    84 '''Compute Resources Health'''
    85 Unix hosts report system health information via ganglia to the [http://monitor.gpolab.bbn.com/ganglia/ GPO Monitoring Server]:
    86  * [http://monitor.gpolab.bbn.com/ganglia/?c=Chattanooga Chattanooga hosts]
    87  * [http://monitor.gpolab.bbn.com/ganglia/?c=Cleveland Cleveland hosts]
    88 
    89 '''Network Devices Health'''
    90 Network devices are polled for system health via SNMP, and that data is also available at the [http://monitor.gpolab.bbn.com/ganglia/ GPO Monitoring Server]:
    91  * [http://monitor.gpolab.bbn.com/ganglia/?c=Chattanooga Chattanooga devices]
    92  * [http://monitor.gpolab.bbn.com/ganglia/?c=Cleveland Cleveland devices]
    93 
    94 If you need read-only SNMP access to the network devices in a Starter rack, please contact [mailto:gpo-infra@geni.net]
    95 
    96 == Perform an experiment in your Starter rack ==
    97 
    98 '''1.''' In this example, we specify 2 VM instances using the same image, it is also possible to specify 2 separate instances using different images:
    99 {{{
    100 $ euca-run-instances -k mykey -n 2 emi-05AC15E0
    101 RESERVATION     r-47F80755      agosain agosain-default
    102 INSTANCE        i-45E007BF      emi-05AC15E0    0.0.0.0 0.0.0.0 pending mykey   0               m1.small        2011-10-21T02:06:22.451Z   cha-euca        eki-8F5A137E    eri-CB4F1461
    103 INSTANCE        i-335C067F      emi-05AC15E0    0.0.0.0 0.0.0.0 pending mykey   1               m1.small        2011-10-21T02:06:22.453Z   cha-euca        eki-8F5A137E    eri-CB4F1461
    104 }}}
    105 
    106 '''2.''' Login to the VMs. When connecting to your image you must use the private key from the Eucalyptus keypair you created above. The {{{-i}}} flag lets you specify the private key. Each image also has a specified username that you will use on instances. In the case of the Ubuntu 10.04 (Lucid) image, the username is "ubuntu". So the complete ssh command for this image is:
    107 {{{
    108 $ ssh -i mykey.priv ubuntu@192.1.243.56
    109 $ ssh -i mykey.priv ubuntu@192.1.243.53
    110 }}}
    111 
    112 '''3.''' Now that the VMs are running you can use an iperf client and server setup to exchange traffic between the two VMs. First, install the Iperf application on both VMs:
    113 {{{
    114 apt-get install iperf
    115 }}}
    116 Them, start the iperf server:
    117 {{{
    118 ubuntu@ip-10-153-0-67:~$ iperf -s
    119 ------------------------------------------------------------
    120 Server listening on TCP port 5001
    121 TCP window size: 85.3 KByte (default)
    122 ------------------------------------------------------------
    123 [  4] local 10.153.0.67 port 5001 connected with 10.153.0.66 port 52930
    124 [ ID] Interval       Transfer     Bandwidth
    125 [  4]  0.0-30.0 sec  1.92 GBytes    549 Mbits/sec
    126 }}}
    127 
    128 
    129 '''4.''' Then, connect to the private IP address of other VM and start the iperf client:
    130 {{{
    131 ubuntu@ip-10-153-0-66:~$ iperf -c 10.153.0.67 -t 30
    132 ------------------------------------------------------------
    133 Client connecting to 10.153.0.67, TCP port 5001
    134 TCP window size: 16.0 KByte (default)
    135 ------------------------------------------------------------
    136 [  3] local 10.153.0.66 port 52930 connected with 10.153.0.67 port 5001
    137 [ ID] Interval       Transfer     Bandwidth
    138 [  3]  0.0-30.0 sec  1.92 GBytes    549 Mbits/sec
    139 }}}
    140 '''5.''' Terminate your VM instances after you have completed your tests:
    141 {{{
    142 euca-terminate-instances i-38E807A1
    143 }}}
    144 
    145 == Install a VM image on your Starter rack ==
    146 
    147 The following procedure outlines an experimenter view into using the Starter racks Eucalyptus VMs as a resource for an experiment.
    148 
    149 To request an account for a GENI Starter Rack send an email request to [mailto:gpo-infra@geni.net] including the following details:
    150  * Preferred username and full name.
    151  * Public SSH public key for remote login into rack resources.
    152  * Provide an MD5 hash of the password for sudo use. Generated by {{{openssl passwd -1}}}
    153 
    154 1. Install Euca2ools (where???), which are command-line tools for interacting with the Eucalyptus open-source cloud-computing infrastructure.
    155 {{{
    156   $ sudo apt-get install euca2ools
    157 }}}
    158 
    159 2. Install Euca credentials. These credentials can be downloaded as a package from your Eucalyptus web site. If you do not have an account you can request one at ????  Once the account is verified and approved, go to the "Credentials" tab. In the "Credentials ZIP-file" section, click on the "Download Credentials" button. Locate the downloaded zip file (the location depends on your OS and web browser) and move it to a working directory.
    160 
    161 3. Unpack the credential and source the environment:
    162 {{{
    163   $ mkdir ~/euca
    164   $ mv ~/Downloads/euca2-myaccount-x509.zip ~/euca
    165   $ cd ~/euca
    166   $ unzip euca2-myaccount-x509.zip
    167   $ . eucarc
    168 }}}
    169 
    170 4. Add firewall rules to your euca instance, below ssh and ping are allowed in the example:
    171 {{{
    172   $ euca-authorize -P tcp -p 22 -s 0.0.0.0/0 default
    173   $ euca-authorize -P icmp -t -1:-1 -s 0.0.0.0/0 default
    174 }}}
    175 
    176 5. Generate key pair to connect to eauca instance:
    177 {{{
    178   $ euca-add-keypair mykey > mykey.priv
    179   $ chmod 600 mykey.priv
    180 }}}
    181 
    182 6. Show available images, start a euca instance with your newly generated keys:
    183 {{{
    184   $ euca-describe-images   # show list of available images
    185   IMAGE emi-48AA122D  ubuntu-9.04/ubuntu.9-04.x86-64.img.manifest.xml   chaos   available  public  x86_64       machine
    186   IMAGE emi-62E51726  ubuntu-10.04/lucid-server-cloudimg-amd64.img.manifest.xml tmitchel  available  public  x86_64 machine             
    187   $ euca-run-instances -k mykey emi-62E51726
    188 }}}
    189 
    190 7. Set public address for euca VM created above, by requesting for an address to be allocated and then assigning it to the specific euca instance:
    191 {{{
    192   $ euca-allocate-address    # will show address that is allocated to you
    193   ADDRESS       192.1.243.55
    194   $ euca-associate-address -i i-38E807A1 192.1.243.55 
    195 }}}
    196 
    197 8. You may now connect into the Euca VM:
    198 {{{
    199   $ ssh -i mykey.priv ubuntu@192.1.243.55
    200 }}}
    201 
    202 Your Euca instance may now be used to run an experiment.
    203 
    20436----
    20537= InstaGENI Administration =
    … …  
    21446== InstaGENI Racks Software/Firmware upgrades ==
    21547
     48
     49= Starter Racks Administration =
     50
     51This section provides a few example of the administrative task on a Starter Rack.  Example administrative tasks for ExoGENI and InstaGENI racks are different, but will accomplish similar functions.
     52
     53== Get Starter rack Accounts ==
     54
     55'''Requesting an account'''
     56
     57Site operators should contact [mailto:gpo-infra@geni.net] to request sudo-capable login accounts on the Starter rack hosts by providing:
     58 * Preferred username
     59 * Preferred fullname
     60 * SSH public key for remote login
     61 * Hashed password for sudo obtained by running:
     62{{{
     63openssl passwd -1
     64}}}
     65 and typing a password twice.  The resulting string should be of the form: `$1$xxxxxxxx$xxxxxxxxxxxxxxxxxxxxxx`
     66
     67''' Policies for Unix account use '''
     68 * Remote account access is via public-key SSH only (no password-based login).
     69 * Do not run interactive sessions as root (don't use `sudo bash`, but instead run individual commands under sudo for logging).
     70 * Do not share account credentials.  We are happy to create individual accounts, or to give staffers who don't have logins access to our emergency account for outage debugging.
     71 * GPO staffers actively manage these systems using the puppet configuration management utility.  If you need to modify a system, please e-mail us at [mailto:gpo-infra@geni.net] to ensure that the desired change takes effect.
     72
     73'''Accounts on non-Unix rack devices'''
     74
     75Please contact [mailto:gpo-infra@geni.net] if you need login access to:
     76 * Control router or dataplane switch
     77 * IP KVM for remote console access
     78 * PDU for remote power control
     79
     80== Access Devices Consoles ==
     81''' Compute Resource consoles'''
     82 * The fold-out console in the rack can be used to view the consoles of any of the hosts in the rack.
     83 * The KVM hotkey for changing which device is displayed is `Ctrl Ctrl`.
     84
     85'''Network Devices Consoles'''
     86The `monitor1` node in each rack can be used as a serial console for network devices located in that rack.
     87 * Login to `monitor1` using the console
     88 * Use screen to access the desired serial device, e.g.:
     89{{{
     90screen /dev/ttyS0
     91}}}
     92 * When done using screen, kill the session by pressing: `Ctrl-a K`
     93
     94== Monitoring Starter rack Health ==
     95
     96'''Service Health'''
     97
     98GPO uses Nagios as a front-end for alerting about rack problems. The following services are monitored in the Starter Racks:
     99 * Resource problems with CPU, swap, or disk space on each host.
     100 * IP connectivity failures from the rack server to commodity internet (Google) and to the GPO lab.
     101 * Excessive CPU usage and excessive uplink broadcast traffic on the experimental switch.
     102 * Problems with standard experimental use of the Eucalyptus aggregate.
     103
     104The current state of monitored hosts and services at a given city can be viewed at:
     105 * [http://monitor.gpolab.bbn.com/nagios/cgi-bin/status.cgi?hostgroup=cities-cha&style=detail Chattanooga Status]
     106 * [http://monitor.gpolab.bbn.com/nagios/cgi-bin/status.cgi?hostgroup=cities-cle&style=detail Cleveland Status]
     107
     108If you would like to be added to any of these notifications, please contact us at [mailto:gpo-infra@geni.net].
     109
     110'''Compute Resources Health'''
     111Unix hosts report system health information via ganglia to the [http://monitor.gpolab.bbn.com/ganglia/ GPO Monitoring Server]:
     112 * [http://monitor.gpolab.bbn.com/ganglia/?c=Chattanooga Chattanooga hosts]
     113 * [http://monitor.gpolab.bbn.com/ganglia/?c=Cleveland Cleveland hosts]
     114
     115'''Network Devices Health'''
     116Network devices are polled for system health via SNMP, and that data is also available at the [http://monitor.gpolab.bbn.com/ganglia/ GPO Monitoring Server]:
     117 * [http://monitor.gpolab.bbn.com/ganglia/?c=Chattanooga Chattanooga devices]
     118 * [http://monitor.gpolab.bbn.com/ganglia/?c=Cleveland Cleveland devices]
     119
     120If you need read-only SNMP access to the network devices in a Starter rack, please contact [mailto:gpo-infra@geni.net]
     121
     122== Perform an experiment in your Starter rack ==
     123
     124'''1.''' In this example, we specify 2 VM instances using the same image, it is also possible to specify 2 separate instances using different images:
     125{{{
     126$ euca-run-instances -k mykey -n 2 emi-05AC15E0
     127RESERVATION     r-47F80755      agosain agosain-default
     128INSTANCE        i-45E007BF      emi-05AC15E0    0.0.0.0 0.0.0.0 pending mykey   0               m1.small        2011-10-21T02:06:22.451Z   cha-euca        eki-8F5A137E    eri-CB4F1461
     129INSTANCE        i-335C067F      emi-05AC15E0    0.0.0.0 0.0.0.0 pending mykey   1               m1.small        2011-10-21T02:06:22.453Z   cha-euca        eki-8F5A137E    eri-CB4F1461
     130}}}
     131
     132'''2.''' Login to the VMs. When connecting to your image you must use the private key from the Eucalyptus keypair you created above. The {{{-i}}} flag lets you specify the private key. Each image also has a specified username that you will use on instances. In the case of the Ubuntu 10.04 (Lucid) image, the username is "ubuntu". So the complete ssh command for this image is:
     133{{{
     134$ ssh -i mykey.priv ubuntu@192.1.243.56
     135$ ssh -i mykey.priv ubuntu@192.1.243.53
     136}}}
     137
     138'''3.''' Now that the VMs are running you can use an iperf client and server setup to exchange traffic between the two VMs. First, install the Iperf application on both VMs:
     139{{{
     140apt-get install iperf
     141}}}
     142Them, start the iperf server:
     143{{{
     144ubuntu@ip-10-153-0-67:~$ iperf -s
     145------------------------------------------------------------
     146Server listening on TCP port 5001
     147TCP window size: 85.3 KByte (default)
     148------------------------------------------------------------
     149[  4] local 10.153.0.67 port 5001 connected with 10.153.0.66 port 52930
     150[ ID] Interval       Transfer     Bandwidth
     151[  4]  0.0-30.0 sec  1.92 GBytes    549 Mbits/sec
     152}}}
     153
     154
     155'''4.''' Then, connect to the private IP address of other VM and start the iperf client:
     156{{{
     157ubuntu@ip-10-153-0-66:~$ iperf -c 10.153.0.67 -t 30
     158------------------------------------------------------------
     159Client connecting to 10.153.0.67, TCP port 5001
     160TCP window size: 16.0 KByte (default)
     161------------------------------------------------------------
     162[  3] local 10.153.0.66 port 52930 connected with 10.153.0.67 port 5001
     163[ ID] Interval       Transfer     Bandwidth
     164[  3]  0.0-30.0 sec  1.92 GBytes    549 Mbits/sec
     165}}}
     166'''5.''' Terminate your VM instances after you have completed your tests:
     167{{{
     168euca-terminate-instances i-38E807A1
     169}}}
     170
     171== Install a VM image on your Starter rack ==
     172
     173The following procedure outlines an experimenter view into using the Starter racks Eucalyptus VMs as a resource for an experiment.
     174
     175To request an account for a GENI Starter Rack send an email request to [mailto:gpo-infra@geni.net] including the following details:
     176 * Preferred username and full name.
     177 * Public SSH public key for remote login into rack resources.
     178 * Provide an MD5 hash of the password for sudo use. Generated by {{{openssl passwd -1}}}
     179
     1801. Install Euca2ools (where???), which are command-line tools for interacting with the Eucalyptus open-source cloud-computing infrastructure.
     181{{{
     182  $ sudo apt-get install euca2ools
     183}}}
     184
     1852. Install Euca credentials. These credentials can be downloaded as a package from your Eucalyptus web site. If you do not have an account you can request one at ????  Once the account is verified and approved, go to the "Credentials" tab. In the "Credentials ZIP-file" section, click on the "Download Credentials" button. Locate the downloaded zip file (the location depends on your OS and web browser) and move it to a working directory.
     186
     1873. Unpack the credential and source the environment:
     188{{{
     189  $ mkdir ~/euca
     190  $ mv ~/Downloads/euca2-myaccount-x509.zip ~/euca
     191  $ cd ~/euca
     192  $ unzip euca2-myaccount-x509.zip
     193  $ . eucarc
     194}}}
     195
     1964. Add firewall rules to your euca instance, below ssh and ping are allowed in the example:
     197{{{
     198  $ euca-authorize -P tcp -p 22 -s 0.0.0.0/0 default
     199  $ euca-authorize -P icmp -t -1:-1 -s 0.0.0.0/0 default
     200}}}
     201
     2025. Generate key pair to connect to eauca instance:
     203{{{
     204  $ euca-add-keypair mykey > mykey.priv
     205  $ chmod 600 mykey.priv
     206}}}
     207
     2086. Show available images, start a euca instance with your newly generated keys:
     209{{{
     210  $ euca-describe-images   # show list of available images
     211  IMAGE emi-48AA122D  ubuntu-9.04/ubuntu.9-04.x86-64.img.manifest.xml   chaos   available  public  x86_64       machine
     212  IMAGE emi-62E51726  ubuntu-10.04/lucid-server-cloudimg-amd64.img.manifest.xml tmitchel  available  public  x86_64 machine             
     213  $ euca-run-instances -k mykey emi-62E51726
     214}}}
     215
     2167. Set public address for euca VM created above, by requesting for an address to be allocated and then assigning it to the specific euca instance:
     217{{{
     218  $ euca-allocate-address    # will show address that is allocated to you
     219  ADDRESS       192.1.243.55
     220  $ euca-associate-address -i i-38E807A1 192.1.243.55 
     221}}}
     222
     2238. You may now connect into the Euca VM:
     224{{{
     225  $ ssh -i mykey.priv ubuntu@192.1.243.55
     226}}}
     227
     228Your Euca instance may now be used to run an experiment.
     229
     230
     231
    216232----
    217 = ExoGENI Administration =
    218 
    219 ExoGENI rack administration tasks are to be defined and will be captured here when available.
    220 
    221 == Get ExoGENI rack Accounts ==
    222 == Access Devices Consoles ==
    223 == Monitoring ExoGENI rack Health ==
    224 == Perform an experiment in your ExoGENI rack ==
    225 == Install a VM image on your ExoGENI rack ==
    226 == ExoGENI Racks Software/Firmware upgrades ==
    227 
    228 
    229 
    230 ----
     233
    231234{{{
    232235#!html