[[PageOutline]] = GIMI Documentation = == Services == For each service we need documentation for installation and administration. === Labwiki === [Jack] === Job Service === [Thierry] === OML server === * Install software sources (http://oml.mytestbed.net/projects/oml/wiki/Installation#Debian-FedoraCentOS) {{{ echo 'deb http://download.opensuse.org/repositories/home:/cdwertmann:/oml/xUbuntu_14.04/ /' | sudo tee /etc/apt/sources.list.d/oml2.list sudo apt-get update }}} * Install software packages for the OML server and desired database (http://oml.mytestbed.net/projects/oml/wiki/Installation#Collection-Point-Server-and-Database-Tools) {{{ sudo apt-get install oml2-server }}} * If using PostgreSQL, a user must be created, and the oml2-server command line changed (http://oml.mytestbed.net/projects/oml/wiki/Installation#Using-the-PostgreSQL-Backend) {{{ createuser -U DBADMIN --pwprompt --no-superuser --createdb --no-createrole --no-replication oml # Remove the --no-replication if there is an error (<9.2) sudo sed -i "s/^#OPTS/OPTS/;s/PASS/OMLPASSWORD/" /etc/default/oml2-server # Replace OMLPASSWORD with the password entered at the previous step }}} * Test that everything work (http://oml.mytestbed.net/projects/oml/wiki/Installation#Testing) {{{ oml2-generator --amplitude 1 --frequency 1000 --samples 10 --sample-interval .1 --oml-id localservertest --oml-domain installtest --oml-collect localhost }}} * This will require the OML clients tools to be installed too (http://oml.mytestbed.net/projects/oml/wiki/Installation#Client-library) {{{ sudo apt-get install liboml2 oml2-example }}} * Open the firewall ports (3003 by default; http://oml.mytestbed.net/projects/oml/wiki/Installation#Troubleshooting) {{{ sudo iptables -I INPUT -p tcp --dport 3003 -j ACCEPT sudo ip6tables -I INPUT -p tcp --dport 3003 -j ACCEPT # optional steps to save firewall configuration across reboots: sudo apt-get install iptables-persistent sudo service iptables-persistent save }}} * Test from a remote machine (http://oml.mytestbed.net/projects/oml/wiki/Installation#Full-Reporting-Chain) {{{ oml2-generator --amplitude 1 --frequency 1000 --samples 10 --sample-interval .1 --oml-id remoteservertest --oml-domain installtest --oml-collect SERVERNAME }}} === AMQP === [Thierry] === GIMI Experiment Service === [Jack or Thierry?] === iRODS === iRODS, which stands for integrated Rule-Oriented Digital System, is an open-source data management software in use at research organizations and government agencies worldwide. iRODS acts as middleware sitting on top of the distributed storage systems and underneath the user applications that require archiving, processing and disseminating data in a transparent and automated fashion. For more detailed and comprehensive descriptions, please refer to [http://irods.org/]. In GIMI, we use iRODS as the back-end system to provide archive and automated processing functions to experimenters. We configured two iRODS instances, one as the production data system, the other as the environment for development purposes. The production system has the iCAT running on geni-gimi.renci.org, and two data servers running on geni-gimi.renci.org and emmy9.casa.umass.edu. geni-gimi.renci.org is currently allocated with ~2TB disk space, which can be further increased depending on its usage. To interact with iRODS, various user interfaces are provided. For example, iCommand ([https://wiki.irods.org/index.php/icommands]) provides a UNIX-like command line interface to users who are already familiar with UNIX commands (such as ls, cp, rm, etc). iRODS also provides a rich set of APIs. For example, Jargon, a JAVA client API for iRODS, provides the REST functions so that GENI experimenters can easily create their iRODS accounts via the GENI portal. iRODS is equipped with a powerful rule language and rule engine. We developed iRODS policies and rules that recognize metadata files created based on specific schemas and automate data ingestion to help experimenters more efficiently organize and share their data. Currently our iRODS instances are running release 3.3. == Images == a) '''InstaGENI Ubuntu 12.04 Image''' {{{ Image URN: urn:publicid:IDN+instageni.illinois.edu+image+ch-geni-net:gimidev2 Image URL: https://www.instageni.illinois.edu/image_metadata.php?uuid=df9ce62b-ec1a-11e3-bc1a-000000000000 }}} b) '''InstaGENI Ubuntu 12.04 with OVS and Trema Image''' (**) {{{ For the same manager, you can use id=urn:publicid:IDN+instageni.nysernet.org+image+ch-geni-net:gimiovstrema2 For other managers, you can use url=https://www.instageni.nysernet.org/image_metadata.php?uuid=551210c7-ecca-11e3-9973-000000000000 }}} c) '''ExoGENI Ubuntu 12.04 Image''' {{{ Image URL: http://emmy9.casa.umass.edu/Disk_Images/ExoGENI/Ubuntu1204/ubuntu1204gimidev.xml version=2a3c19e07232a5d3bb6a300eca0d0db7ce7aca79 }}} === Installation Procedure for Ubuntu 12.04 === The detailed procedure for installing OMF6 RC maybe found [http://mytestbed.net/doc/omf/file.INSTALLATION.html#Ubuntu at this link] and for installing OML [http://mytestbed.net/projects/oml/wiki/Installation#Debian-FedoraCentOS at this link] In brief: 1. Boot IG VM with Default Ubuntu 12.04 image [[BR]] 2. Login to VM and continue as root [[BR]] 3. Install OMF RC [[BR]] {{{ #apt-get update #apt-get install build-essential libxml2-dev libxslt-dev libssl-dev #apt-get install ruby1.9.1-dev ruby1.9.1 #apt-get install ruby1.9.3 #gem install omf_rc --pre #gem install file-tail (Optional*) #install_omf_rc -i -c }}} ** For Creating Image b), "gem install trema" is executed in the above step following gem install omf_rc. For more information on Trema, [http://trema.github.io/trema/ click here] 4. Test if OMF RC is installed {{{ #echo "--- :uid: samplehostname2 :uri: amqp://gimi3.casa.umass.edu :environment: production :debug: false" > /etc/omf_rc/config.yml #start omf_rc tail /var/log/omf_rc.log }}} After running the above you should see {{{ [2014-06-04 22:29:50 +0000] INFO OmfRc::Runner: Starting OMF Resource Controller version '6.1.1' [2014-06-04 22:29:50 +0000] INFO OmfRc::Runner: Connected using {:proto=>:amqp, :user=>"guest", :domain=>"127.0.0.1"} }}} 5. Install OML Client {{{ $sudo sh -c "echo 'deb http://download.opensuse.org/repositories/home:/cdwertmann:/oml/xUbuntu_12.04/ /' >> /etc/apt/sources.list.d/oml2.list" $wget http://download.opensuse.org/repositories/home:cdwertmann:oml/xUbuntu_12.04/Release.key $sudo apt-key add - < Release.key $sudo apt-get update $sudo apt-get install oml2 $sudo apt-get install oml2-apps }}} 6. Check if OML Client is installed correctly {{{ root@Node1:~# dpkg -l | grep "oml" ii gpslogger-oml2 2.10.0-mytestbed1 Record and store GPS measurements using OML ii iperf-oml2 2.10.0-mytestbed1 Internet Protocol bandwidth measuring tool, with OML support ii liboml2 2.11.0-mytestbed1 OML: The O? Measurement Library (metapackage) ii liboml2-9 2.11.0-mytestbed1 OML: The O? Measurement Library ii liboml2-dev 2.11.0-mytestbed1 OML measurement library headers ii nmetrics-oml2 2.10.0-mytestbed1 Measure and record system information from libsigar using OML ii oml2 2.11.0-mytestbed1 OML: The O? Measurement Library Suite (Metapackage) ii oml2-apps 2.10.0-mytestbed1 A meta-package that installs the complete standalone OML2 applications collection ii oml2-proxy-server 2.11.0-mytestbed1 OML proxy server ii oml2-proxycon 2.11.0-mytestbed1 OML proxy server control script ii oml2-server 2.11.0-mytestbed1 OML measurement server ii otg2-oml2 2.10.0-mytestbed1 Orbit Traffic Generator ii ripwavemon-oml2 2.10.0-mytestbed1 Report statistics from a Navini RipWave modem ii trace-oml2 2.10.0-mytestbed1 Measure and record libtrace data using OML }}} *Only required if you want to parse data from a continuously growing file as shown [ here] == Postboot scripts == === InstaGENI === {{{ #!/bin/bash read -r slice1 /etc/hostname /bin/hostname -F /etc/hostname echo 'amqp://emmy9.casa.umass.edu' > /var/omf/communication_url echo $host-$slice > /var/omf/node_uri restart omf_rc }}} === ExoGENI === {{{ #!/bin/bash host=$1 slice=`ruby -e "print '$2'[/[^+]*$/]"` echo $host > /etc/hostname /bin/hostname -F /etc/hostname apt-get install psmisc echo 'amqp://emmy9.casa.umass.edu' > /var/omf/communication_url echo $host-$slice > /var/omf/node_uri if grep 'Ubuntu' /etc/issue; then restart omf_rc else /etc/init.d/omf_rc restart fi }}} [Divya] What is done in the postboot script? What may need to change over time? == Tool Usage == [Divya/Jeanne] === Tutorials === || '''Instructions''' || '''Last Updated''' || '''Last Tested''' || || [http://groups.geni.net/geni/wiki/GEC17Agenda/AdvancedOpenFlow/Procedure OpenFlow Load Balancer Tutorial] || TBD || TBD || || [http://groups.geni.net/geni/wiki/GEC20Agenda/LabWiki Experimentation and Instrumentation with Labwiki] || TBD || TBD || || [http://groups.geni.net/geni/wiki/GEC18Agenda/LabWikiAndOEDL The GENI Experiment Lifecycle using LabWiki and OEDL] || TBD || TBD || === OEDL Scripts === * [http://mytestbed.net/projects/omf6/wiki/OEDLOMF6 OEDL Documentation for OMF6] === OML Applications === * http://oml.mytestbed.net/projects/oml/repository/revisions/master/entry/README.md#L169 * http://oml.mytestbed.net/projects/oml/repository/revisions/master/entry/example/liboml2/README === Troubleshooting === * [http://groups.geni.net/geni/wiki/GIMIv3/TroubleshootingLabwiki Troubleshooting Experiments in Labwiki] === How to Write a !LabWiki application for CSV files === http://groups.geni.net/geni/wiki/HowTo/LabWikiGraphUsingCSV