| 15 | | At this working session, we will review and aim to approve an updated GENI monitoring architecture that satisfies the monitoring requirements discussed at the last GEC. In addition, we will discuss extensions to the design of the monitoring API for relational, event, and time series data. Finally, we will review the status of solutions to previously identified "pain points" of concern to the monitoring and operations community. Campus IT staff, aggregate operators and software developers are encouraged to attend. |
| | 19 | * Monitoring Architecture |
| | 20 | * Sarah Edwards, GPO |
| | 21 | * Monitoring Privacy |
| | 22 | * Heidi Picher Dempsey, GPO |
| | 23 | * Monitoring APIs + Experimenter/Operator Portal |
| | 24 | * Chaos Golubitsky, GPO |
| | 25 | * Status of Monitoring Pains |
| | 26 | * Sarah Edwards, GPO |
| | 27 | |
| | 28 | == Notes == |
| | 29 | |
| | 30 | '''Sarah Edwards''' described major elements of a proposed monitoring architecture including the major actors in the monitoring architecture, the interfaces between them, use cases, and the data to be collected. (See slides 4-22) |
| | 31 | |
| | 32 | The major actors are: |
| | 33 | * ''Meta-operations'', which collects and makes available operational data and also generates cross-GENI monitoring data |
| | 34 | * The future ''GENI Clearinghouse'', which is the authoritative repository for project, user, and slice information |
| | 35 | * ''Aggregates'', which contain resources |
| | 36 | * ''Campuses'', which host resources |
| | 37 | * ''Experimenters'', which under limited circumstances might provide some monitoring information to meta-operations |
| | 38 | * ''Regional Networks'' and ''Backbone Networks'' (I2, NLR) |
| | 39 | |
| | 40 | The major monitoring interfaces allow: |
| | 41 | * Aggregates/Campuses/Experimenters/Meta-operations to submit data to Meta-operations |
| | 42 | * Anyone to query data from Meta-operations |
| | 43 | * Out-of-band communication to experimenters, aggregates and campuses |
| | 44 | |
| | 45 | In addition, GENI monitoring relies on other interfaces which allow: |
| | 46 | * Access to definitive data about slices, projects, and users from the future GENI Clearinghouse (via interfaces defined by the GENI Architecture group) |
| | 47 | * Resolution of problems via out-of-band access between Campuses and their Regional Networks |
| | 48 | * Resolution of problems via out-of-band access between GMOC and GENI backbone networks (I2, NLR) |
| | 49 | |
| | 50 | The structure of the monitoring data falls in one of three categories: |
| | 51 | * ''Time series'' data is a particular piece of data measured over time. |
| | 52 | * ''Relational'' data conveys entities and the relationship between them as observed at time T. |
| | 53 | * ''Event'' data denotes something happening or being noticed at a particular moment in time. |
| | 54 | |
| | 55 | The precision of the data can be described as either: |
| | 56 | * ''Definitive'' data is known to be definitive because it was generated by the source that created it. The future GENI clearinghouse will contain a lot of definitive data. |
| | 57 | * ''Sampled'' data is collected by measuring the state of the world periodically. The Meta-operations database contains a lot of sampled data. |
| | 58 | |
| | 59 | Several uses cases were reviewed with notes about the data required to support them: |
| | 60 | * An operator responds to a request from an experimenter who is unable to create a sliver at an aggregate. The operator needs to see the current and recent status of that aggregate (eg is the AM responding to queries?) |
| | 61 | * Assess the availability/utilization of GENI resources over a four month period. |
| | 62 | * A core !OpenFlow switch misbehaves leading to intermittent failures. Various operators need information about whether L2 network resources are usable and the utilization of relevant network resources in order to debug the failure. |
| | 63 | * Assess the historic usage of network resources in order to plan for future expansion of those resources. |
| | 64 | * An experimenter wants to access basic information about their sliver to assess the general health and activity of their experiment. |
| | 65 | * An aggregate operator receives a complaint from campus IT about unusual traffic on the campus network. The aggregate operator needs to looks at the current state of resources at their campus in order to demonstrate the problem is not caused by his aggregate. |
| | 66 | |
| | 67 | The data needed to be collected for the above use cases was also reviewed. |
| | 68 | |
| | 69 | Discussion: |
| | 70 | |
| | 71 | Heidi Picher Dempsey asked about the scope of the use cases. Sarah Edwards responded that use cases aren't meant to be exhaustive, but are based on real things we wanted to know over the fairly short time we've been monitoring GENI. If people know of data or use cases that they think are important, talk to Sarah about adding them. |
| | 72 | |
| | 73 | Victor Orlikowsky noted that I&M overlaps with monitoring. The group discussed the distinction for awhile. In particular, Heidi noted that "we don't want every experimenter to go out and collect all of the same data." |
| | 74 | |
| | 75 | '''Heidi Picher Dempsey''' described a proposal for privacy of monitoring data in GENI. (Slides 24-31) |
| | 76 | |
| | 77 | Heidi said that the goal of the proposal is to make some decisions. We've been collecting data, but we've given no guidance about what to do about it. This proposal is based on input from Adam Slagell. |
| | 78 | |
| | 79 | General Recommendation: |
| | 80 | * Data that CAN be shared publicly: existence of slice (urn, UUID); slice is active (has resources); slice name |
| | 81 | * Some data CAN be shared IF you have access controls. Aggregates should make sure they are not sharing user data. LLR requests are a special case. |
| | 82 | * Data that should NOT be shared: opt-in user data; data that identifies experimenters by username/real name; data that identifies experimenter contact info |
| | 83 | |
| | 84 | Privacy recommendations to experimenters |
| | 85 | * GENI needs to warn experimenters that slice name is public |
| | 86 | * Slice credential will (soon) include a slice e-mail. If you don't want it shared, then don't use a personal email. |
| | 87 | * Your personal email is part of your experimenter credential and will be shared with the aggregate and GMOC. Should not be shared outside GMOC. |
| | 88 | |
| | 89 | Discussion: |
| | 90 | |
| | 91 | Victor suggested that we create two documents: |
| | 92 | 1. An operator "Code of Ethics": Have a privacy policy of what we will or won't do with your data. (Chaos Golubitsky suggested looking at the [https://www.usenix.org/lisa/system-administrators-code-ethics sysadmin code of ethics] for guidance). |
| | 93 | 2. "Experimenter's privacy recommendations": Here's what you should do/not do as an experimenter. |
| | 94 | |
| | 95 | One person expressed surprise that slice name is public and that we should make the warning about slice name being public front and center. |
| | 96 | |
| | 97 | Sarah noted that basic information about an experiment (eg nodes used, interface stats) will be publicly available. |
| | 98 | |
| | 99 | '''Chaos Golubitsky''' presented on monitoring interfaces and the experimenter portal including updates done in conjunction with Mitch !McCracken. (Slides 33-43) |
| | 100 | |
| | 101 | GMOC has been collecting slice metadata, measurement data, and operational data from GENI aggregates and slice authorities. Starting in Spiral 4, they will run the Service Desk which will detect and respond to operational problems in GENI. Chaos introduced '''Mitch !McCracken''' who is GMOC's primary software developer who is working on GENI monitoring; Mitch started working on monitoring data submission and interfaces in January. |
| | 102 | |
| | 103 | GMOC tools allow for the collection, display and alerting on operational slice metadata and measurement data. |
| | 104 | |
| | 105 | Work to improve these tools has focused on: |
| | 106 | * standardizing data submission to make it easy for new projects (like racks) to submit data to GMOC and provide consistent data naming |
| | 107 | * Improving user interfaces to make it easy for data of interest to be easy for experimenters and operators to find. In particular, data should be tied together in standard ways and operational health data should be easy to find. |
| | 108 | |
| | 109 | Currently there are two APIs for submitting data: a measurement API for submitting time series data, and a relational API for submitting meta-data. In Spiral 3 GMOC and GPO tested the measurement API and collected a year of data improving reliability and tools for data submission. |
| | 110 | |
| | 111 | Chaos demonstrated progress with the GMOC Experimenter web interface which supports finding and viewing time series data for slices and nodes. In addition, there is now submission of slice data from the GPO's SA using the relational API. |
| | 112 | |
| | 113 | In spiral 4, work will continue to test and improve the relational data API, refine the time-series API, and collect data from the new racks. In addition, work will continue on improving the user interfaces including: improved documentation, improved health reports, tying together relational data with time-series data, use of URNs and UUIDs consistently with other GENI entities to distinguish slices over time, and to support specific use cases. |
| | 114 | |
| | 115 | Chaos then invited participation by others on the following items: |
| | 116 | * early adopters for the experimenter web portal |
| | 117 | * ideas for GENI health tests that people are running or would like to see run |
| | 118 | * feedback |
| | 119 | |
| | 120 | '''Sarah Edwards''' reviewed the status of the ongoing list of "monitoring pains". (Slides 45-47) |
| | 121 | |
| | 122 | Items which have made progress since the last GEC are listed on the slides with check marks: |
| | 123 | * Chaos and Mitch have made progress on packaging the plastic slices monitoring software |
| | 124 | * Nick Bastin and Josh Smift have released FOAM (the !OpenFlow Aggregate Manager) and installed it throughout the meso-scale which supports improved monitoring |
| | 125 | * The experimenter web interfaces improves the ability to find data by slice |
| | 126 | * Heidi (based on input from Adam Slagell) outlined a proposal regarding privacy of monitoring data at this session |
| | 127 | * Sarah presented some monitoring data that should be collected at this session |
| | 128 | |
| | 129 | |
| | 130 | For more information please contact one of the following people or attend the biweekly monitoring call... |
| | 131 | |
| | 132 | Contacts: |
| | 133 | * GENI operational monitoring list: monitoring@geni.net |
| | 134 | * GMOC: Mitch !McCracken (mrmccrac@grnoc.iu.edu) |
| | 135 | * GPO: Chaos Golubitsky (chaos@bbn.com), Sarah Edwards (sedwards@bbn.com) |
| | 136 | |
| | 137 | Biweekly monitoring conference call: |
| | 138 | * Every other Friday at 2pm Eastern |
| | 139 | * Information on GENI monitoring mailing list |
| | 140 | |
| | 141 | GENI operational monitoring list: monitoring@geni.net (Sign up at http://lists.geni.net) |
| | 142 | |
| | 143 | |
| | 144 | |
| | 145 | |
| | 146 | |
| | 147 | |
| | 148 | |
| | 149 | |
