Changes between Initial Version and Version 1 of GENIRacksHome/OpenGENIRacks/InstallationGuide


Ignore:
Timestamp:
05/23/14 11:36:40 (10 years ago)
Author:
Jeanne Ohren
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • GENIRacksHome/OpenGENIRacks/InstallationGuide

    v1 v1  
     1= GRAM Installation Guide =
     2
     3[[PageOutline(2-100,Table of Contents,inline,unnumbered)]]
     4
     5== Introduction ==
     6
     7== Architecture ==
     8
     9[[Image(OpenstackTwoNodeArchitecture.jpg)]]
     10
     11[[Image(OpenstackThreeNodeArchitectur.jpg)]]
     12
     13== Requirements ==
     14
     15=== Hardware Requirements ===
     16
     17=== Software Requirements ===
     18
     19==== Packages ====
     20The following Debian packages are required on the controller node
     21   * git
     22   * cloud-utils
     23
     24==== Ports ====
     25The following ports will be used by GRAM components.  Verify that these ports are not already in use.  If so, change the configuration of the gram component below to use a different port.
     26
     27   * Controller node
     28      * 8000: GRAM Clearinghouse (Unless you are using a different clearinghouse).  See [wiki:GENIRacksHome/GRAMRacks/InstallationGuide#ConfigureandStartClearinghouseontheControlNode this section] to change this port.
     29      * 8001: GRAM Aggregate Manager.  See [wiki:GENIRacksHome/GRAMRacks/InstallationGuide#ConfigureandStartAggregateManager this section] to change this  port.
     30      * 9000: VMOC Default Controller
     31      * 7001: VMOC Management.  See [wiki:GENIRacksHome/GRAMRacks/InstallationGuide#Setup this section] to change this port.
     32      * 6633: VMOC
     33
     34=== Openstack Requirements ===
     35
     36==== Folsom release with Quantum using !OpenvSwitch plugin ====
     37   * currently developing with Folsom version 2012.2.1 installed on Ubuntu 12.04
     38
     39   * Verify environment variables
     40      * The following variables should be set in the environment of any user that will be using the command line tools
     41      * Suggest adding to a "novarc" file and sourcing that file
     42{{{
     43export OS_TENANT_NAME=admin
     44export OS_USERNAME=admin
     45export OS_PASSWORD=<password set in keystone database for admin user>
     46export OS_AUTH_URL="http://localhost:5000/v2.0/"
     47export OS_REGION_NAME=RegionOne
     48export SERVICE_ENDPOINT="http://localhost:35357/v2.0"
     49export SERVICE_TOKEN=<token set in keystone.conf>
     50export OS_NO_CACHE=1
     51}}}
     52
     53   * Verify installed version:
     54{{{
     55    sudo dpkg -l | grep nova
     56    sudo dpkg -l | grep quantum
     57}}}
     58   * Verify running services on Controller and Network nodes:
     59{{{
     60    sudo service nova-api status
     61    sudo service nova-scheduler status
     62    sudo service keystone status
     63    sudo service quantum-server status
     64    sudo service quantum-l3-agent status
     65    sudo service quantum-dhcp-agent status
     66    sudo service quantum-openvswitch-plugin-agent status
     67    sudo service rabbitmq-server status
     68    sudo service mysql status
     69    sudo service openvswitch-switch status
     70}}}
     71
     72   * Verify OVS configuration on the Controller node:
     73      * There should be a qg port on br-ex for each external network
     74
     75{{{
     76$ sudo ovs-vsctl show
     77107352c3-a0bb-4598-a3a3-776c5da0b62b
     78    Bridge "br-eth1"
     79        Port "phy-br-eth1"
     80            Interface "phy-br-eth1"
     81        Port "eth1"
     82            Interface "eth1"
     83        Port "br-eth1"
     84            Interface "br-eth1"
     85                type: internal
     86    Bridge br-ex
     87        Port br-ex
     88            Interface br-ex
     89                type: internal
     90        Port "eth2"
     91            Interface "eth2"
     92        Port "qg-9816149f-9c"
     93            Interface "qg-9816149f-9c"
     94                type: internal
     95    Bridge br-int
     96        Port "int-br-eth1"
     97            Interface "int-br-eth1"
     98        Port br-int
     99            Interface br-int
     100                type: internal
     101    ovs_version: "1.4.0+build0"
     102}}}
     103
     104   * Verify running services on Compute nodes:
     105{{{
     106     sudo service nova-compute status
     107     sudo service quantum-plugin-openvswitch-agent status
     108     sudo service openvswitch-switch status
     109
     110}}}
     111
     112   * Verify OVS configuration on the Comput nodes:
     113      * Assuming no VMs on the compute node
     114{{{
     115$ sudo ovs-vsctl show
     1164ec3588c-5c8f-4d7f-8626-49909e0e4e02
     117    Bridge br-int
     118        Port br-int
     119            Interface br-int
     120                type: internal
     121        Port "int-br-eth1"
     122            Interface "int-br-eth1"
     123    Bridge "br-eth1"
     124        Port "phy-br-eth1"
     125            Interface "phy-br-eth1"
     126        Port "br-eth1"
     127            Interface "br-eth1"
     128                type: internal
     129        Port "eth1"
     130            Interface "eth1"
     131    ovs_version: "1.4.0+build0"
     132}}}
     133
     134   * Verify all expected services registered with Nova
     135      * Expect to see nova-cert, nova-consoleauth, and nova-scheduler on the controller node and nova-compute on each compute node.  All should have State = :-) (not XXX)
     136{{{
     137   $ sudo nova-manage service list
     138   Binary           Host                                 Zone             Status     State Updated_At
     139   nova-cert        pridevcontrol                        nova             enabled    :-)   2013-02-07 20:47:38
     140   nova-consoleauth pridevcontrol                        nova             enabled    :-)   2013-02-07 20:47:37
     141   nova-scheduler   pridevcontrol                        nova             enabled    :-)   2013-02-07 20:47:38
     142   nova-compute     pridevcompute1                       nova             enabled    :-)   2013-02-07 20:47:33
     143   nova-compute     pridevcompute2                       nova             enabled    :-)   2013-02-07 20:47:35
     144}}}
     145 
     146==== KVM virtualization ====
     147   * Verify KVM is installed and able to use hardware virtualization:
     148       * NOTE: kvm-ok is part of the cpu-checker package
     149{{{
     150   $ kvm -version
     151   QEMU emulator version 1.0 (qemu-kvm-1.0), Copyright (c) 2003-2008 Fabrice Bellard
     152   $ kvm-ok
     153   INFO: /dev/kvm exists
     154   KVM acceleration can be used
     155   $ sudo service libvirt-bin status
     156   libvirt-bin start/running, process 2537
     157}}}
     158
     159==== Metadata service requirements ====
     160   * Nova should have set up a NAT rule for metadata services
     161{{{
     162$ sudo iptables -t nat -L
     163...
     164Chain quantum-l3-agent-PREROUTING (1 references)
     165target     prot opt source               destination         
     166DNAT       tcp  --  anywhere             169.254.169.254      tcp dpt:http to:10.10.8.71:8775
     167...
     168
     169}}}
     170==== Image requirements ====
     171   * Currently, nova images must meet the following requirements for GRAM:
     172      1.  Must be based on Ubuntu 12.04/12.10
     173      1.  Must have 1 interface configured in /etc/network/interfaces.  Remaining interfaces will configured based on the rspec configuration.
     174      1.  Must have the following packages installed:
     175          * cloud-utils
     176          * openssh-server
     177          * bash
     178          * apt
     179
     180== Set up Environment ==
     181Make sure that "source /etc/novarc" or the equivalent is run for all users (omniuser and gram specifically).  The environment variables set by this script are required to run Openstack commands.
     182 
     183== Install Tarballs ==
     184
     185=== Controller node ===
     186
     187For this installation you will need to download or otherwise obtain the following software and credentials.  Place these tarballs in /tmp on the controller node.
     188
     189   * gcf-<version>.tgz
     190      * download from http://www.gpolab.bbn.com/local-sw or provided by GRAM team
     191   * gram-<version>.tgz
     192      * provided by GRAM team
     193   * gramcerts: certificates
     194      * provided by GRAM team
     195
     196
     197== Install Instance Images on Controller Node ==
     198
     199  * Make sure you have images installed for the VMs.  See [wiki:GENIRacksHome/GRAMRacks/InstallationGuide#Imagerequirements image requirements] above.  You can add images using the following command:
     200{{{
     201glance image-create --name=<image-name> --is-public=true --container-format=bare \
     202      --disk-format=qcow2 < <path to image>
     203}}}
     204
     205  See [http://docs.openstack.org/trunk/openstack-compute/admin/content/starting-images.html] for information on downloading images.
     206
     207== GCF Installation and Configuration on Controller Node==
     208
     209   * Install gcf tarball
     210{{{
     211$ cd /opt
     212$ sudo tar -zxvf /tmp/gcf-2.2.tar.gz
     213$ sudo ln -s gcf-2.2 gcf
     214}}}
     215
     216== GRAM Installation and Configuration on Controller Node==
     217
     218=== Setup ===
     219   * Create an entry for the controller node in its /etc/hosts file.  For example, if the name of your controller is controller.example.com and its IP address is 198.51.100.10, create an /etc/hosts entry that looks like:
     220{{{
     221198.51.100.10  controller.example.com controller
     222}}}
     223
     224   * Create gram user on controller node (use your favorite password and take note of this)
     225{{{
     226$ sudo useradd -m -s /bin/bash gram
     227$ sudo adduser gram sudo
     228$ sudo passwd gram
     229}}}
     230
     231   * Add GCF paths to /home/gram/.bashrc.  You must use sudo when editing this file.
     232{{{
     233export PATH=/home/gram/gram/src:/opt/gcf/src:$PATH
     234export PYTHONPATH=/opt/gcf/src:/home/gram/gram/src
     235}}}
     236
     237   * Install Clearinghouse and Aggregate Manager certificates
     238{{{
     239$ sudo mkdir -m 777 /etc/gram
     240$ cd /etc/gram
     241$ sudo tar -zxvf /tmp/gramcerts.tgz
     242}}}
     243
     244   * Install gram tarball
     245{{{
     246$ su - gram
     247$ tar -zxvf /tmp/gram-0.1.tgz
     248$ ln -s gram-0.1 gram
     249$ mkdir .gcf
     250$ cp gram/gcf_config .gcf
     251$ exit
     252}}}
     253
     254     * Change the default OS image name in {{{/home/gram/gram/src/gram/am/gram/config.py}}} to match what is installed on this rack.
     255
     256     * If you need to change the VMOC Management Port, change the vmoc_interface_port variable in {{{/home/gram/gram/src/gram/am/gram/config.py}}}.
     257{{{
     258# PORT on which to communicate to VMOC interface manager
     259vmoc_interface_port = 7001
     260}}}
     261
     262
     263=== Install and configure VMOC and default controller ===
     264
     265   * Install POX controller software
     266{{{
     267$ cd /opt
     268$ sudo git clone http://github.com/noxrepo/pox
     269$ sudo git checkout -b betta remotes/origin/betta
     270}}}
     271
     272   * Install and start GRAM VMOC services
     273{{{
     274$ sudo cp /home/gram/gram/src/services/gram-vmoc.conf /etc/init
     275$ sudo cp /home/gram/gram/src/services/gram-ctrl.conf /etc/init
     276$ cd /etc/init.d
     277$ sudo ln -s /lib/init/upstart-job gram-vmoc
     278$ sudo ln -s /lib/init/upstart-job gram-ctrl
     279$ sudo service gram-vmoc start
     280$ sudo service gram-ctrl start
     281}}}
     282
     283   * Verify the VMOC services are running
     284      * If services are not running you can check the logs /var/log/upstart/gram-vmoc.log and /var/log/upstart/gram-ctrl.log
     285
     286{{{
     287$ sudo service gram-vmoc status
     288gram-vmoc start/running, process 15200
     289$ sudo service gram-ctrl status
     290gram-ctrl start/running, process 15220
     291
     292$ netstat -an | grep ":9000"
     293tcp        0      0 0.0.0.0:9000            0.0.0.0:*               LISTEN     
     294$ netstat -an | grep ":7001"
     295tcp        0      0 0.0.0.0:7001            0.0.0.0:*               LISTEN     
     296$ netstat -an | grep ":6633"
     297tcp        0      0 0.0.0.0:6633            0.0.0.0:*               LISTEN 
     298}}}
     299
     300=== Configure and Start Clearinghouse on the Control Node ===
     301
     302    * Edit /home/gram/.gcf/gcf_config and set the Clearinghouse host to the hostname of the controller (e.g. mycontroller)
     303{{{
     304[clearinghouse]
     305# Address at which the Clearinghouse should listen. 0.0.0.0 means all interfaces.
     306host=mycontroller
     307port=8000
     308}}}
     309
     310   * Install and start Clearinghouse service
     311      * Clearinghouse will run on port 8000
     312
     313{{{
     314$ sudo apt-get xmlsec1
     315$ sudo cp /home/gram/gram/src/services/gram-ch.conf /etc/init
     316$ cd /etc/init.d
     317$ sudo ln -s /lib/init/upstart-job gram-ch
     318$ sudo service gram-ch start
     319}}}
     320
     321   * Verify the Clearinghouse service is running on port 8000
     322      * If service is not running you can check the logs /var/log/upstart/gram-ch.log
     323{{{
     324$ sudo service gram-ch status
     325gram-ch start/running, process 15253
     326$ netstat -an | grep 8000
     327tcp        0      0 128.33.251.71:8000      0.0.0.0:*               LISTEN 
     328}}}
     329
     330=== Configure and Start Aggregate Manager ===
     331
     332   * Edit /home/gram/.gcf/gcf_config and set the host to the hostname of the controller (e.g. mycontroller)
     333{{{
     334[aggregate_manager]
     335name=gram
     336
     337# Address that the AM listens on
     338host=mycontroller
     339port=8001
     340
     341}}}
     342
     343   * Install and start Aggregate Manager service
     344      * AM will run on port 8001
     345      * AM V2 will run on port 8002
     346
     347{{{
     348$ sudo cp /home/gram/gram/src/services/gram-am.conf /etc/init
     349$ sudo cp /home/gram/gram/src/services/gram-amv2.conf /etc/init
     350$ cd /etc/init.d
     351$ sudo ln -s /lib/init/upstart-job gram-am
     352$ sudo ln -s /lib/init/upstart-job gram-amv2
     353$ sudo service gram-am start
     354$ sudo service gram-amv2 start
     355}}}
     356
     357   * Verify the AM service is running on port 8001
     358      * If service is not running you can check the logs /var/log/upstart/gram-am.log
     359{{{
     360$ sudo service gram-am status
     361gram-am start/running, process 15253
     362$ netstat -an | grep 8001
     363tcp        0      0 128.33.251.71:8001      0.0.0.0:*               LISTEN 
     364}}}
     365
     366   * Verify the AM V2 service is running on port 8002
     367      * If service is not running you can check the logs /var/log/upstart/gram-amv2.log
     368{{{
     369$ sudo service gram-amv2 status
     370gram-amv2 start/running, process 15253
     371$ netstat -an | grep 8002
     372tcp        0      0 128.33.251.71:8002      0.0.0.0:*               LISTEN 
     373}}}
     374
     375=== Configure GRAM SSH Proxy Tool on Controller Node ===
     376
     377   * Install gram-ssh-proxy in /usr/local/bin
     378{{{
     379$ cd /home/gram/gram/src/gram/am/gram
     380$ sudo make
     381}}}
     382
     383   * Verify the tool has been installed
     384{{{
     385$ ls -l /usr/local/bin
     386total 20
     387-rwsr-xr-x 1 root root 18058 Feb  9 15:56 gram_ssh_proxy
     388}}}
     389
     390=== Configure test OMNI environment ===
     391
     392   * Create test omni user on controller node
     393{{{
     394$ sudo useradd -m -s /bin/bash omniuser
     395$ sudo adduser omniuser sudo
     396$ sudo passwd omniuser
     397$ sudo mkdir /home/omniuser/.ssh
     398$ sudo chown omniuser.omniuser /home/omniuser/.ssh
     399}}}
     400     ''Remember the password picked for omniuser.''
     401
     402   * Add GCF paths to /home/omniuser/.bashrc.  You must use sudo when editing this file.
     403{{{
     404export PATH=/opt/gcf/src:$PATH
     405export PYTHONPATH=/opt/gcf/src
     406}}}
     407
     408   * Create an SSH keypair to be installed on the VMs
     409{{{
     410$ su - omniuser
     411$ ssh-keygen
     412Generating public/private rsa key pair.
     413Enter file in which to save the key (/home/omniuser/.ssh/id_rsa): /home/omniuser/.ssh/vmkey
     414Enter passphrase (empty for no passphrase):
     415Enter same passphrase again:
     416Your identification has been saved in /home/omniuser/.ssh/vmkey.
     417Your public key has been saved in /home/omniuser/.ssh/vmkey.pub.
     418The key fingerprint is:
     4192f:f2:62:de:3a:3d:c6:42:0d:ad:48:a8:6b:fe:81:8a omniuser@mycontroller
     420The key's randomart image is:
     421+--[ RSA 2048]----+
     422|                 |
     423|                 |
     424|   .   .         |
     425|  . . . .        |
     426| . . . +S        |
     427|. . . o ..       |
     428| o . ..o. .      |
     429|+.  . =+=.       |
     430|E... oo*o.       |
     431+-----------------+
     432}}}
     433
     434   * Configure OMNI
     435      * Start with default omni_config file
     436{{{
     437   $ mkdir /home/omniuser/.gcf
     438   $ cp /home/gram/gram/omni_config /home/omniuser/.gcf
     439}}}
     440
     441      * Edit /home/omniuser/.gcf/omni_config
     442         * Set the host for the AM and CH to the hostname of the controller (e.g. mycontroller)
     443{{{
     444[aggregate_nicknames]
     445gram=,https://mycontroller:8001
     446
     447# Where the gcf-ch server is listening
     448ch=https://mycontroller:8000
     449}}}
     450
     451         * Add an entry for omniuser to {{{/home/omniuser/.gcf/omni_config}}}.   Look for the line {{{users =}}} and edit it to read:
     452{{{
     453users = omniuser
     454}}}
     455
     456         * Add information about this user to the {{{Define users here}}} section of the {{{omni_config}}} file.
     457{{{
     458[omniuser]
     459urn=urn:publicid:IDN+geni.net:gpo:gcf+user+omniuser
     460keys=~/.ssh/vmkey.pub
     461}}}
     462
     463         * Install clearinghouse and aggregate manager certs in omniuser's ~/.gcf directory
     464{{{
     465cd /home/omniuser/.gcf
     466cp /etc/gram/am-cert.pem .
     467cp /etc/gram/am-key.pem .
     468cp /etc/gram/ch-cert.pem .
     469cp /etc/gram/ch-key.pem .
     470}}}
     471
     472=== Verify that OMNI can communicate with the AM ===
     473
     474{{{
     475$ omni.py -V3 -a gram getversion
     476INFO:omni:Loading config file /home/omniuser/.gcf/omni_config
     477INFO:omni:Using control framework my_gcf
     478INFO:omni:Substituting AM nickname gram with URL https://gramcontrol:8001, URN unspecified_AM_URN
     479WARNING:omni:You asked to use AM API 2, but the AM(s) you are contacting do not all speak that version.
     480WARNING:omni:At the URLs you are contacting, all your AMs speak AM API v3.
     481WARNING:omni:Switching to AM API v3. Next time call Omni with '-V3'.
     482INFO:omni:Substituting AM nickname gram with URL https://gramcontrol:8001, URN unspecified_AM_URN
     483INFO:omni:AM URN: unspecified_AM_URN (url: https://gramcontrol:8001) has version:
     484INFO:omni:{   'geni_ad_rspec_versions': [   {   'extensions': [],
     485                                      'namespace': 'http://www.geni.net/resources/rspec/3',
     486                                      'schema': 'http://www.geni.net/resources/rspec/3/ad.xsd',
     487                                      'type': 'GENI',
     488                                      'version': '3'}],
     489    'geni_api': 3,
     490    'geni_api_versions': {   '3': 'https://gramcontrol:8001/'},
     491    'geni_credential_types': [{   'geni_type': 'geni_sfa',
     492                                     'geni_version': '3'}],
     493    'geni_request_rspec_versions': [   {   'extensions': [],
     494                                           'namespace': 'http://www.geni.net/resources/rspec/3',
     495                                           'schema': 'http://www.geni.net/resources/rspec/3/request.xsd',
     496                                           'type': 'GENI',
     497                                           'version': '3'}]}
     498INFO:omni: ------------------------------------------------------------
     499INFO:omni: Completed getversion:
     500
     501  Options as run:
     502                aggregate: ['gram']
     503                api_version: 3
     504                framework: my_gcf
     505                noGetVersionCache: True
     506
     507  Args: getversion
     508
     509  Result Summary: Your AMs do not all speak requested API v2. At the URLs you are contacting, all your AMs speak AM API v3. Switching to AM API v3. Next time call Omni with '-V3'.
     510
     511Got version for https://gramcontrol:8001
     512 
     513INFO:omni: ============================================================
     514}}}
     515
     516== Test Basic AM Functionality ==
     517
     518TODO:  Basic OMNI commands that should be run and how to verify they were successful
     519
     520TODO: The canned rspecs (the ones with open-flow controllers listed) that have pridev addresses hard-coded. We should include instructions to edit these.  These instructions should go in this section along with the commands to test basic OF functionality