[[PageOutline]]
= GENI / FIRE Collaboration Workshop, September 2015, Session 6: Monitoring =

== Session chairs ==

 * Scott Kirkpatrick
 * Jim Griffioen

== Session Summary ==

The session was separated into three parts.  We had thought of them as first, monitoring the testbed networks we had heard about in the rest of the program, studying clouds, clusters and slices connecting them, then second monitoring issues in wireless, but new technology testbeds and high bandwidth smartphones serving to observe networks, and third, monitoring the internet from end to end.  To accommodate some speakers’ travel schedules, we held the session in the order, topics 2, 3, then 1.

The discussion of wireless covered experimental design and protocols as seen in the iMinds testbeds, where the Wishfull project will make possible much more flexible experimental control, but the classic issue of the need for very controlled experiments to reproducibly characterize a technology in its very early stages remains.  5G, although an important goal, remains imperfectly specified and lacking in hardware to test.  Interference effects that will become important as spectrum is increasingly shared among different technologies are observed and will be of interest.  The second talk in the session, about the FCC’s comittment to transparency and efforts to put as many as 100,000 mobile applications performing standard edge internet performance efforts in the field to attain transparency, was disappointing.  The FCC is still not completely through their process of determining robustness of their results and a privacy exposure-free means of disseminating them.  They still hope to release a report of results by city and raw data sets for public inspection “soon,” but this would appear to not be likely until next year.  

In the end-to end session,  AMIS, a hardware and software platform for measurement of longer flows (>5 min) with modern privacy primitives ranging from full encryption to differential privacy (controlled noise). was described.  Capturing and sharing information from sensor networks was covered as a component that will be architected and implemented as part of a standard substrate built with a city’s cooperation in a new project, EMBERS, and made available through open source mechanisms for more general use. Congestion in the internet has been claimed to be a non-problem, but studies done at M-Lab which compare RTTs and download performance from content sources in multiple eyeball ISP over extended periods of time show that in fact there are frequent and extended anomalies.  Their methods are simple, and can be easily extended to include GENI clusters as participants in the measurement constellation.  Finally, data captured for a month at a time in several cities from a widely distributed smartphone app was shown  to provide the most extensive coverage of the internet’s edge to date.  For example, 50,000 unique (anonymized) Android phones contributed measurements in the LA basin, and these could be interpreted to show difference in loading or contention at different times of day between applications deriving their data over distinct paths in the Internet.  The discussion diverged briefly into a very animated discussion of the ethics of using so many people’s personal activities as part of an aggregated measurement tool.


The first session (which occurred last) began by looking at the problem of
monitoring power usage information for testbeds networks and clouds.  The
session began with a description of the Cloudlab power monitoring system
which uses the Intelligent Platform Management Interface (IPMI) to collect
power measurements over time and make them available to experimenters via a
web page.  A related talk described the power monitoring service supported by
the Kwapi system which was designed to offer high frequency data collection,
allowing users to correlate and associate power measurements with other
network behaviors (e.g., bandwidth usage). 
The federation of testbeds presents additional monitoring challenges
ranging from the complexities of dealing with heterogeneous infrastructure,
software systems, and data formats, to scalable data collection and
aggregation, to trust/reliability issues.  Example systems such as
MODACLOUDS, SPECS, and DICE were described that use monitoring architectures
consisting of many distinct components, each with a specific purpose, to
address the complexity of these systems.  The FELIX monitoring service
handles the scalability challenge by arranging measurement services
hierarchically at the highest level, but then using a distributed approach
within testbed islands. The Federated Trust and User Experience framework
address the problem of assessing a testbed's reliability and trustworthiness
by combining conventional monitoring information (objective information) with
reputation-based trust  management systems (subjective information) to
define a reputation value that can be used to compare testbeds.
While much work has focused on monitoring testbed infrastructure,
relatively few tools are available to users to measure their
specific experiment or slice.  Looking at experimenter mailing lists and
discussion groups shows that the majority of problems that experimenters
face would not be detected by monitoring the infrastructure, but rather
require slice/experiment-specific measurement and validation tools designed to
test and evaluate the performance of a user's slice or experiment.



== Presentations ==

=== Session 2: Monitoring Wireless Infrastructure ===
Discussion leader: Ivan Seskar

|| '''Author(s)''' || '''Title''' || '''Presentation''' ||
|| Ingrid Moerman || Emphasizing 5G Systems Measurement || [attachment:"FIRE-GENI_Emphasizing5GSystemMeasurement_IngridMoerman.pptx" slides] ||
|| James Miller || Measuring Broadband to the Sidewalk || [attachment:"MBA Fixed and Mobile Speed Test App - GENI.pptx" slides] ||

=== Session 3: Monitoring the Internet as a Whole ===
Discussion leader: Scott Kirkpatrick

|| '''Author(s)''' || '''Title''' || '''Presentation''' ||
|| Yan Luo  || Privacy Preserving Network Measurement || [attachment:"AMIS-GENI-FIRE-091815.pdf" slides] ||
|| Timur Friedman || Smart Cities || [attachment:"Smart Cities.pdf" slides] ||
|| Collin Anderson || Internet Misbehavior as Seen from M-Lab || [attachment:"GENI-Internet_Misbehavior.pdf" slides] ||
|| Scott Kirkpatrick || Crowd-sourced Data Demo || [attachment:"GENI-FIRE_talks.pdf" slides] ||


=== Session 1: Monitoring Testbed Networks, Clouds, Clusters, and Slices ===
Discussion leader: Jim Griffioen

|| '''Author(s)''' || '''Title''' || '''Presentation''' ||
|| Mike Zink || Power and Temperature Measurement Infrastructure for !CloudLab || [attachment:"2015-09-17 CloudLab Power Monitoring.pptx" slides] ||
|| Yahya Al-Hazmi || Monitoring Ontologies -- Demo || [attachment:"OMN4ExpMonitoringDemo.pdf" slides] ||
|| Brecht Vermeulen & Chrysa Papagianni || Facility Monitoring for Federations and Measuring Reputation in Testbeds || [attachment:"ReputationGENI-FIRE.pptx" slides] ||
|| Dana Petcu || Monitoring in Multi-clouds || [attachment:"Petcu-Monitoring.pdf" slides] ||
|| Bartek Belter || Monitoring of Multi-domain Slices in FELIX || [attachment:"FELIX-monitoring-v1.pptx" slides] ||
|| Lucas Nussbaum || Kwapi Monitor for GRID 5000 || [attachment:"kwapi.pdf" slides] ||
|| Jim Griffioen || Monitoring Experiments vs Testbeds || [attachment:"Griffioen-FIRE-GENI-Workshop-2015.pdf" slides] ||