= IMF GEC11 QSR = ---- = Overview = This wiki page serves as the Status Report following GEC11 for the IMF project. * Draft created by Rudra Dutta on September 10, 2011 * Modified by IMF team during September 10 - October 2, 2011 * Final version released on October 2, 2011 See post-GEC10 status report. Between GEC10 and GEC11, the main focus of the IMF project was: * Streamlining and improving the pS IMFRealTime service developed before GEC10 * Creating a standalone standard pS command-line client that can access IMF-obtained measurement data * Creating a standalone standard pS GUI client for the same purpose * Starting work on defining and developing the OMF-compatible interface to the IMF substrate control handler The first three of these resulted in deliverables in this cycle; for the fourth, a deliverable is expected at a future GEC. Due to shortage of demo and poster space at GEC11, the IMF team did not receive demo or poster space. We separately demonstrated the new IMF capabilities to GPO at an individual meeting at the GEC, and a followup webex meeting post-GEC. = Major Accomplishments = == Milestones Achieved == * IMF: S3.e Demonstration at GEC11 and Experimenter Outreach * IMF: S3.f Documentation and Code Release == Deliverables made == * GEC11 demo (see above) * [attachment:wiki:IMF:GEC11_imf_cluster-d_review.pptx Presentation at GEC11 Cluster D] * Code release (see below on this wiki) * Code documentation (see below on this wiki) = Description of work performed during last quarter = IMF project achievements for this first part of Spiral 3 consist of: * Streamlining and improving the pS IMFRealTime service developed before GEC10 * Creating a standalone standard pS command-line client that can access IMF-obtained measurement data * Creating a standalone standard pS GUI client for the same purpose * Starting work on defining and developing the OMF-compatible interface to the IMF substrate control handler * Demonstrating new capabilities at GEC11 == Activities and Findings == At the GEC10 demo, IMF’s PSM and MH communicated using pS query (GEC8 demo and before – used proprietary (arbitrary) XML-RPC interaction). The MH integrated new pS service “IMFRealTime” (based on “skeleton” service of pS) – it fields pS queries from the outside world, then triggers measurements. In the GEC11 version, the “IMFRealTime” service was updated to subsume true MP+MA (with database) functions: this enables the IMF MH to serve more versatile pS queries. pS queries can be sent by clients without going through the rest of IMF. All measurements, triggered as well as scheduled, are stored in the database. The GEC11 view of the IMF code is shown below in terms of the system architecture at GEC8 in the diagram below. [[Image(GEC11_setup_on_arch.png, 90%)]] [[BR]] The IMF Pub-Sub Module was also updated to work with the new version of the IMFRealTime service. The XML-RPC interface is still not re-enabled, we propose to do that at a future version (see post-GEC10 status report for discussion). Some of our thinking during this cycle served to clarify "pS-compatibility" and "OMF compatibility" in the context of the IMF project. Currently our operating definitions are as follows, followed by more detailed discussion. * "pS-compatibility" means "any measurement resource which is IMF-readable (i.e. for which an IMF Measurement Handler module has been developed) can be queried by a standard pS query client, without having to go through the rest of IMF, or having to use IMF semantics." * "OMF compatibility" means "any controllable substrate which is IMF-controllable (i.e. for which an IMF Substrate Control Handler module has been developed) can be viewed as an OMF controllable resource by an OMF experimenter, and controlled using an OMF experiment, without having to go through the rest of IMF, or having to use IMF semantics." '''''pS''''': As we have discussed previously, using pS compatible channels for sensing does not realistically allow enabling many cross-layer experiments. We took our goal to be defined by the following: "pS-compatibility" means "any measurement resource which is IMF-readable - for which an IMF Measurement Handler module has been developed - can be queried by a standard pS query client, without having to go through the rest of IMF, or having to use IMF semantics." Since GEC10, our goal was that (i) the IMFpSRealTime service should be indistinguishable from a reasonable MA service for existing pS query clients, and (ii) well-packaged ready code would be available for anybody who wants it. We have effectively achieved these goals, we are now working on a test deployment on LEARN. '''''OMF''''': We spent a significant time thinking about what it meant "to use OMF semantics for the substrate control channel". We settled on a similar definition as for pS: "OMF compatibility" means "any controllable substrate which is IMF-controllable - for which an IMF Substrate Control Handler module has been developed - can be viewed as an OMF controllable resource by an OMF experimenter, and controlled using an OMF experiment, without having to go through the rest of IMF, or having to use IMF semantics." We do not believe using OMF saves any effort in the IMF development; as before, we see the benefit as the accessibility to those used to OMF semantics. There are multiple potential ways of interfacing the IMF semantics with OMF, and we have articulated them. Basically the IMF Substrate Control Handler will behave as an OMF resource controller, and allow control through OMF. This does not realistically support timescales suitable for cross-layer control. The IMF system can provide access to the resource controller through a "long-standing" OMF experiment that basically acts as a slave, or repeater, for cross-layer control commands. However, though we might investigate this option as a value-add, with the GPO's advice, we have settled on our primary goal as being simply to incorporate an OMF resource controller interface to the IMF substrate control handler. Neither pS nor OMF allows timescales which are required for many cross-layer automated protocols. But different experiments may utilize measurement readings of different freshness or timescales. It would be meaningful to build in semantics for multiple measurement characteristics in IMF, and be able to provide experimenters with information about the timescales of available measurement and actuation sources. '''''Metadata''''': For cross-layer adaptive and responsive protocols within the slice, such as IMF should enable, it is important for the experimenter to know a few things about the available measurements: at what frequency are fresh measurements available, what is the latency with which the system can deliver the fresh data to the consumption point, whether the sensing can be adjusted to some extent by the consumer - scheduled ahead of time, or frequency changed? The same are also true of actuation capabilities. Earlier in the project, we simplified measurement into "realtime" and "not realtime" categories, i.e. those suitable for cross-layer control use, and those not. Actually there is a continuous spectrum between them, and different ones can be useful for dealing with different cross-layer control issues. The measurement metadata, such as is being standardized in the I&M group now, should contain some "pointer" to RSPECs that connect it to the capabilities of the resource that produced the data, which would address the above issues. The measurement consumer can refer to the RSPEC for that resource if they want to. The MDOD should also contain some metadata that helps the cross-layer algorithm decide whether it is "usable" or not, or how usable. A fine-grained timestamp, as envisaged in Section 4.1.2 of the GENI-SE-IM-ARCH-1.0 document, would certainly be useful there. Other potentially useful metadata includes a reference to a "stream" of measurements of which this particular measurement is a part. Incorporating these potential extensions into the MDOD and implementing the MDOD into IMF-sourced measurement data is part of the envisaged future of the IMF project. == Project Participants == * Rudra Dutta, George Rouskas (NCSU) * Shu Huang, Ilia Baldine (RENCI) * Keren Bergman, Michael Wang, Bala Bathula, Cathy Chen (Columbia U) == Publications == Integrated post-GEC report and code documentation (this wiki) == Outreach == Participants of the IMF team attended GECs as well as meetings of the Instrumentation and Measurement Working Group as possible. We have previously collaborated with the GENI LEARN project team, and progressing toward deploying IMF in LEARN. We have started communicating with the TuNIE team from Tsinguah U, with a view to possibly collaborating on adapting IMF to provide measurement capabilities to TuNIE. == Collaborations == The IMF team continued to collaborate with Deniz Gurkan of the GENI project LEARN. == Other contributions == None. = Code Release = == Code Snapshot == The GEC11 version of the IMF system can be built the following. Documentation is included in the code. * [attachment:ezclient.pl perfSONAR standalone command-line client for IMF measurements] * [attachment:perfsonar_imf_gui_v2.1.jar perfSONAR standalone GUI client for IMF measurements] * [attachment:mh.tar.gz Measurement Handler] * [attachment:PubSub_for_PerfSONAR.tar.gz PubSub for perfSONAR] * [attachment:perfsonar_imf_realtime.tar.gz perfSONAR IMF Realtime service] The pS clients are configured to try to access data from the IMF Measurement Handlers running on the BEN testbed, to retrieve port power and BER measurements made on Polatis and Infinera switches. To successfully run this, you need VPN access to the management plane network of BEN. Get in touch with the IMF team to obtain this access, if you need it. == Documentation and Release Notes == See post-GEC10 status report. The only changes to that code are the tarballs above, and the new pS client (command line and GUI) code.