[[PageOutline]] = Plastic Slices Outgages = This page capture any outages that were experienced during the Plastic Slices project. == Baseline 1 Outages == || '''source'''|| '''time started''' || reason || || gpo-infra ||05/09/11 06:37:00 || jbs@bbn.com: Unable to connect to the Expedient AM at NLR || || Ticket #2290 ||05/19/11 13:27:43 || jbs@bbn.com: Some stuff is down in the core right now, making it somewhat hard to debug things. || == Baseline 2 Outages == || '''source''' || '''time started''' || reason || || Ticket #2332 || 05/21/11 15:16:11 || jbs@bbn.com: Flow Visor crash, out of memory. || || Ticket #2312 || 05/21/11 17:35:15 || jbs@bbn.com: Aaron Rosen reports (at 16:20) that the Clemson network is down due to a power outage.|| || gpo-infra || 05/23/11 17:06:00 || jbs@bbn.com: OpenFlow resources down at Stanford (back up at 19:23)|| || gpo-infra || 05/25/11 11:13:00 || jbs@bbn.com: Wisconsin MyPLC down (back up at 12:55)|| == Baseline 3 Outages == || '''source'''|| '''time started''' || reason || || Ticket #2339 || 05/26/11 14:35:23 || jbs@bbn.com of-planet3, of-planet4 down, replaced these hosts in slices. || || Ticket #2339 ||05/26/11 15:23:30 || jbs@bbn.com reporting as down between 15:10 and 15:17, all but four of which are Stanford|| || Ticket #2399 || 05/27/11 16:22:37 || jbs@bbn.com: Stanford's FlowVisor has been down all day. || || Ticket #2399|| 05/31/11 18:10:47 || jbs@bbn.com:a bunch of stuff seems to have gone down about 30 seconds after I started running plastic-103|| == Baseline 4 Outages == || '''source'''|| '''time started''' ||'''time resolved'''|| reason || || Ticket #2372|| || No outages recorded in ticket|| || gpo-infra || 06/04/11 09:39:00 || chaos@bbn.com: myplc.stanford.edu down || == Baseline 5 Outages == || '''source'''|| '''time started''' ||'''time resolved'''|| reason || || Ticket #2373 ||06/09/11 11:02:18 || jbs@bbn.com: plastic-102 Rutgers not working also Stanford -> Washington not working|| || Ticket #2373 ||06/09/11 11:02:18 || jbs@bbn.com: plastic-103 Rutgers -> Stanford died again || || Ticket #2373 ||06/09/11 11:02:18 || jbs@bbn.com: plastic-104 Indiana -> BBN died again || || Ticket #2373 ||06/09/11 11:02:18 || jbs@bbn.com: plastic-108: Stanford -> Rutgers continues to fail || || Ticket #2373 ||06/09/11 11:02:18 || jbs@bbn.com: plastic-110: Stanford -> Rutgers continues to not work at all|| == Baseline 6 Outages == || '''source'''|| '''time started''' ||'''time resolved'''|| reason || || Ticket #2401|| 06/10/11 18:17:51 || jbs@bbn.com: plastic-101 Indiana -> Rutgers is down. (general Internet2 outage ) (20:22:26 both back up) || || Ticket #2401|| 06/10/11 18:17:51 || jbs@bbn.com: plastic-105 Wisconsin -> Clemson was stuck, after client& server restart || || Ticket #2401|| 06/10/11 21:02:04 || jbs@bbn.com: plastic-103 orbitplc2.orbit-lab.org -> of-planet3.stanford.edu, but that hasn't worked in a while. || || Ticket #2401|| 06/10/11 21:02:04 || jbs@bbn.com: plastic-105 Things that I hadn't stopped, were stuck, so I restarted them. || || Ticket #2401|| 06/11/11 00:11:09 || jbs@bbn.com: plastic-103 Washington -> BBN was stuck; restarted the client. Stanford -> Washington. Rutgers -> Stanford still doesn't work || || Ticket #2401|| 06/11/11 00:11:09 || jbs@bbn.com: plastic-105 BBN -> GT had died somewhere in here; I started up the client, and it's running fine now. || || Ticket #2401|| 06/11/11 00:11:09 || jbs@bbn.com: plastic-107 Clemson -> Indiana was stuck; || || Ticket #2401|| 06/11/11 00:11:09 || jbs@bbn.com: plastic-108 Rutgers -> Indiana had been retrying and was then stuck || || Ticket #2401|| 06/11/11 00:11:09 || jbs@bbn.com: plastic-109 Every client was stuck || || Ticket #2401|| 06/11/11 00:11:09 || jbs@bbn.com: plastic-110 Every client was stuck || || Ticket #2401|| 06/12/11 13:38:39 || jbs@bbn.com: plastic-101 Indiana -> Rutgers, no pings flowing. At Rutgers, orbitplc2 rebooted + Indiana FV outage. || || Ticket #2401|| 06/12/11 13:38:39 || jbs@bbn.com: plastic-103 Stanford -> BBN was stuck + Stanford -> Washington was stuck + Rutgers -> Stanford was down || || Ticket #2401|| 06/12/11 13:38:39 || jbs@bbn.com: plastic-104 BBN -> Rutgers was down || || Ticket #2401|| 06/12/11 13:38:39 || jbs@bbn.com: plastic-105 GT -> Wisconsin was down, the client at GT had died|| || Ticket #2401|| 06/12/11 13:38:39 || jbs@bbn.com: plastic-106 Indiana -> GT isn't working || || Ticket #2401|| 06/12/11 13:38:39 || jbs@bbn.com: plastic-108 Wisconsin -> Stanford was stuck; Stanford -> Rutgers was stuck; Rutgers -> Indiana is down; Indiana -> Wisconsin was stuck|| || Ticket #2401|| 06/12/11 13:38:39 || jbs@bbn.com: plastic-109 Every connection was stuck || || Ticket #2401|| 06/12/11 13:38:39 || jbs@bbn.com: plastic-110 Stanford -> Rutgers down; Stanford -> Rutgers down; Indiana -> GT down; Restart !FlowVisor on naxos. || || Ticket #2401|| 06/13/11 12:36:01 || jbs@bbn.com: plastic-101 Indiana -> Rutgers still has no pings flowing, because orbitplc2 still has no static ARP. || || Ticket #2401|| 06/13/11 12:36:01 || jbs@bbn.com: plastic-102 Rutgers -> Stanford still has no pings flowing, because orbitplc2 still has no static ARP. || || Ticket #2401|| 06/13/11 12:36:01 || jbs@bbn.com: plastic-103 Stanford -> BBN down; Stanford -> Washington down; Rutgers -> Stanford down || || Ticket #2401|| 06/13/11 12:36:01 || jbs@bbn.com: plastic-105 GT -> Wisconsin down || || Ticket #2401|| 06/13/11 12:36:01 || jbs@bbn.com: plastic-108 Wisconsin -> Stanford was stuck; Rutgers -> Indiana is down; Indiana -> Wisconsin was stuck || || Ticket #2401|| 06/13/11 12:36:01 || jbs@bbn.com: plastic-109 Everyone *except* Wisconsin -> BBN was stuck || || Ticket #2401|| 06/13/11 12:36:01 || jbs@bbn.com: plastic-110 Everyone was stuck or dead.|| || Ticket #2401|| 06/13/11 20:01:41 || jbs@bbn.com: plastic-103 Wisconsin -> Stanford was stuck || || Ticket #2401|| 06/13/11 20:01:41 || jbs@bbn.com: plastic-110 Washington -> Stanford was stuck|| || Ticket #2401|| 06/14/11 11:39:26 || jbs@bbn.com: No traffic to or from Stanford || || Ticket #2401|| 06/14/11 12:01:20 || jbs@bbn.com: missed one of the NOX controllers, which caused the FV on 33106 to crash || || Ticket #2401|| 06/14/11 14:03:01 || jbs@bbn.com: plastic-103 Everything was down || || Ticket #2401|| 06/14/11 14:03:01 || jbs@bbn.com: plastic-105 Everything was down || || Ticket #2401|| 06/14/11 14:03:01 || jbs@bbn.com: plastic-108 Wisconsin -> Stanford was down; Rutgers -> Indiana was down;|| || Ticket #2401|| 06/14/11 14:03:01 || jbs@bbn.com: plastic-109 Everything was down except Wisconsin -> BBN; || || Ticket #2401|| 06/14/11 14:03:01 || jbs@bbn.com: plastic-110 Everything was down except GT -> BBN; || || Ticket #2401|| 06/15/11 15:20:05 || jbs@bbn.com: derailed by a problem with the NLR core and then the I2 core || || Ticket #2401|| 06/15/11 15:20:05 || jbs@bbn.com: plastic-101 Rutgers -> Stanford and Stanford -> Washington are both down. || || Ticket #2401|| 06/15/11 15:20:05 || jbs@bbn.com: plastic-102 Rutgers -> Stanford and Stanford -> Washington are both down. || || Ticket #2401|| 06/15/11 15:20:05 || jbs@bbn.com: plastic-103 Stanford -> BBN was down; BBN -> Rutgers was down; Stanford -> Washington was down; and Rutgers -> Stanford ;|| || Ticket #2401|| 06/15/11 15:20:05 || jbs@bbn.com: plastic-108 Wisconsin -> Stanford was stuck; Stanford -> Rutgers couldn't connect|| || Ticket #2401|| 06/15/11 15:20:05 || jbs@bbn.com: plastic-109 Clemson -> Stanford was stuck; Stanford -> Wisconsin was stuck;|| || Ticket #2401|| 06/15/11 15:20:05 || jbs@bbn.com: plastic-110 Washington -> Stanford was stuck; Stanford -> Rutgers was stuck; Rutgers -> Indiana was stuck; Indiana -> GT was stuck; || == Baseline 7 Outages == == Baseline 8 Outages ==