Changes between Initial Version and Version 1 of IMF-GEC11-QSR

09/10/11 10:50:02 (10 years ago)
Rudra Dutta




    v1 v1  
     1= IMF GEC11 QSR =
     4= DRAFT =
     5= DRAFT =
     6= DRAFT =
     8= Overview =
     10This wiki page serves as the Status Report following GEC11 for the IMF project.
     12 * Draft created by Rudra Dutta on September 10, 2011
     13 * Modified by IMF team during ...
     14 * Final version released on ...
     16See post-GEC10 status report.  Between GEC10 and GEC11, the main focus of the IMF project was:
     18 * Streamlining and improving the pS IMFRealTime service developed before GEC10
     19 * Creating a standalone standard pS command-line client that can access IMF-obtained measurement data
     20 * Creating a standalone standard pS GUI client for the same purpose
     21 * Starting work on defining and developing the OMF-compatible interface to the IMF substrate control handler
     23The first three of these resulted in deliverables in this cycle; for the fourth, a deliverable is expected at a future GEC.
     25Due to shortage of demo and poster space at GEC11, the IMF team did not receive demo or poster space.  We separately demonstrated the new IMF capabilities to GPO at an individual meeting at the GEC, and a followup webex meeting post-GEC.
     27= Major Accomplishments =
     29== Milestones Achieved ==
     31 * IMF: S3.e Demonstration at GEC11 and Experimenter Outreach
     32 * IMF: S3.f Documentation and Code Release
     34== Deliverables made ==
     36 * GEC11 demo (see above)
     37 * [attachment:wiki:IMF:GEC11_imf_cluster-d_review.pptx Presentation at GEC11 Cluster D]
     38 * Code release (see below on this wiki)
     39 * Code documentation (see below on this wiki)
     41= Description of work performed during last quarter =
     43IMF project achievements for this first part of Spiral 3 consist of:
     45 * Streamlining and improving the pS IMFRealTime service developed before GEC10
     46 * Creating a standalone standard pS command-line client that can access IMF-obtained measurement data
     47 * Creating a standalone standard pS GUI client for the same purpose
     48 * Starting work on defining and developing the OMF-compatible interface to the IMF substrate control handler
     49 * Demonstrating new capabilities at GEC11
     51== Activities and Findings ==
     53At the GEC10 demo, IMF’s PSM and MH communicated using pS query (GEC8 demo and before – used proprietary (arbitrary) XML-RPC interaction).  The MH integrated new pS service “IMFRealTime” (based on “skeleton” service of pS) – it fields pS queries from the outside world, then triggers measurements.
     55In the GEC11 version, the “IMFRealTime” service was updated to subsume true MP+MA (with database) functions: this enables the IMF MH to serve more versatile pS queries.  pS queries can be sent by clients without going through the rest of IMF.  All measurements, triggered as well as scheduled, are stored in the database.
     57The GEC11 view of the IMF code is shown below in terms of the system architecture at GEC8 in the diagram below.
     59[[Image(GEC11_setup_on_arch.png, 90%)]] [[BR]]
     61The Java subscriber to the XMPP pub-sub service is not packaged in this release; the previously supplied code can, without change, be used (as part of a SILO service or standalone, as before), to extract the measurement data stored in the XMPP server.  The difference in this version is in how the IMF Pub-Sub Module acquires that data - it can now use a perfSONAR query to access the Measurement Handler (the IMF PSM can continue to reach the MH using an XML-RPC query as before as another alternative).
     63In the MH, the pS capability is introduced by including a new pS service that we have called the "IMFRealTime" pS service.  (''Note:'' this name may be somewhat misleading and may change in subsequent releases; see discussion below.)  The IMFRealTime service is different from a normal Perfsonar service (MA or MP) in the sense that it implements a bit of both. It uses the mh Perl module and expands Perfsonar's configuration file to provide the MP service. However, instead of sending the results to an MA, the collected data are saved locally.
     65The main reason behind this solution is that we want to avoid the delay introduced by adding an extra layer of database between the users and the MP. Consequently, the IMFRealTime service also needs to provide a user interface for real-time user queries in the absence of a standard Perfsonar MA.  In essence, the IMFRealTime service acts as a pS MP service (that uses previously developed IMF Measurement Handler capability to gather measurement data), but also looks like a pS MA service to external users who issue queries to obtain measurement data from the Measurement Archive.  In fact, internally, it uses shared memory as inter-process communication to access the most fresh data in response to user queries.  The shared memory acts as the rendezvous instead of the usual MA database.
     67'''About "!RealTime" Nomenclature: ''' This behavior is obviously not actually "real-time" in terms of the time when measurement is made.  In contrast, the original XML-RPC approach is "real-time" in that measurements are "triggered", or undertaken (by the MH) in response to incoming user queries.  Rather, the behavior is "real-time" in only the sense that the IMFRealTime service looks for the most recent available data in response to user queries.  To avoid this potential confusion, we are considering re-naming the IMFRealTime pS service to something else in future releases, and use the term "real-time" exclusively to refer to cases where actual fresh measurements are undertaken once user queries are received.  In the future, we hope to investigate the possibility of triggered real-time measurements using the pS pathway as well; at this time, it appears this may not be a suitable match, since the pS architecture relies on static configuration file for measurements, and thus clearly separates the data query from collection.
     69The developer notes, and other documentation included in the code below, provide more details regarding specifics.
     71== Project Participants ==
     73 * Rudra Dutta, George Rouskas (NCSU)
     74 * Shu Huang, Ilia Baldine (RENCI)
     75 * Keren Bergman, Michael Wang, Bala Bathula, Cathy Chen (Columbia U)
     77== Publications ==
     79Integrated quarterly report and code documentation (this wiki)
     81== Outreach ==
     83Participants of the IMF team attended GECs as well as meetings of the Instrumentation and Measurement Working Group as possible.  We have previously collaborated with the GENI LEARN project team, and progressing toward deploying IMF in LEARN.
     85== Collaborations ==
     87The IMF team continued to collaborate with Deniz Gurkan of the GENI project LEARN.
     89The IMF team collaborated with Martin Swany of the perfSONAR team to define and discuss the requirements of IMF as a pS usecase, resulting in the articulation of an alternate type of pS client and functionality, one that more nearly matches the needs of real-time oriented systems such as IMF.
     91== Other contributions ==
     95= Code Release =
     97== Code Snapshot ==
     99The GEC10 version of the IMF system can be built from source in the following tarballs:
     101 * [attachment:mh.tar.gz Measurement Handler]
     102 * [attachment:pubsub_for_perfsonar.tar.gz PubSub for perfSONAR]
     103 * [attachment:perfsonar_imf_realtime.tar.gz perfSONAR IMF Realtime service]
     105== Documentation and Release Notes ==
     107The diagram above (in "Activities/Findings" section) shows where these tarballs fit in.  The pubsub_for_perfsonar.tar.gz file (!PubSub for perfSONAR) replaces the imf.tar.gz file of previous releases: the internal file heirarchy is the same, but the top-level name is different: "imf_pfsr" instead of "imf"; this is a kluge that will be fixed in the next release.  As a consequence, this version of the code can '''''only''''' use pS query to reach the MH, the previous XML-RPC option is not available in this cut (the MH is still capable of it).
     109The following developer's notes on the IMFRealTime pS service provide more detail which should be helpful in using this code.  This file is also included as "doc/README.imf" in the perfsonar_imf_realtime.tar.gz tarball.  In addition, the following diagram of the code organization for the "!PubSub for perfSONAR" module (pubsub_for_perfsonar.tar.gz) may be helpful.
     111 * [attachment:release_notes.txt Developer Release Notes]
     113[[Image(PubSub_PerfSONAR_Source_Code_Folder_Structure.png, 50%)]] [[BR]][[BR]][[BR]]