wiki:PlasticSlices/Outages

Plastic Slices Outgages

This page capture any outages that were experienced during the Plastic Slices project.

Baseline 1 Outages

source time started reason
gpo-infra 05/09/11 06:37:00 Josh Smift: Unable to connect to the Expedient AM at NLR
Ticket #2290 05/19/11 13:27:43 Josh Smift: Some stuff is down in the core right now, making it somewhat hard to debug things.

Baseline 2 Outages

source time started reason
Ticket #2332 05/21/11 15:16:11 Josh Smift: Flow Visor crash, out of memory.
Ticket #2312 05/21/11 17:35:15 Josh Smift: Aaron Rosen reports (at 16:20) that the Clemson network is down due to a power outage.
gpo-infra 05/23/11 17:06:00 Josh Smift: OpenFlow resources down at Stanford (back up at 19:23)
gpo-infra 05/25/11 11:13:00 Josh Smift: Wisconsin MyPLC down (back up at 12:55)

Baseline 3 Outages

source time started reason
Ticket #2339 05/26/11 14:35:23 Josh Smift of-planet3, of-planet4 down, replaced these hosts in slices.
Ticket #2339 05/26/11 15:23:30 Josh Smift reporting as down between 15:10 and 15:17, all but four of which are Stanford
Ticket #2399 05/27/11 16:22:37 Josh Smift: Stanford's FlowVisor has been down all day.
Ticket #2399 05/31/11 18:10:47 Josh Smift:a bunch of stuff seems to have gone down about 30 seconds after I started running plastic-103

Baseline 4 Outages

source time started time resolved reason
Ticket #2372 No outages recorded in ticket
gpo-infra 06/04/11 09:39:00 chaos@bbn.com: myplc.stanford.edu down

Baseline 5 Outages

source time started time resolved
Ticket #2373 06/09/11 11:02:18 Josh Smift: plastic-102 Rutgers not working also Stanford -> Washington not working
Ticket #2373 06/09/11 11:02:18 Josh Smift: plastic-103 Rutgers -> Stanford died again
Ticket #2373 06/09/11 11:02:18 Josh Smift: plastic-104 Indiana -> BBN died again
Ticket #2373 06/09/11 11:02:18 Josh Smift: plastic-108: Stanford -> Rutgers continues to fail
Ticket #2373 06/09/11 11:02:18 Josh Smift: plastic-110: Stanford -> Rutgers continues to not work at all

Baseline 6 Outages

source time started time resolved reason
Ticket #2401 06/10/11 18:17:51 Josh Smift: plastic-101 Indiana -> Rutgers is down. (general Internet2 outage ) (20:22:26 both back up)
Ticket #2401 06/10/11 18:17:51 Josh Smift: plastic-105 Wisconsin -> Clemson was stuck, after client& server restart
Ticket #2401 06/10/11 21:02:04 Josh Smift: plastic-103 orbitplc2.orbit-lab.org -> of-planet3.stanford.edu, but that hasn't worked in a while.
Ticket #2401 06/10/11 21:02:04 Josh Smift: plastic-105 Things that I hadn't stopped, were stuck, so I restarted them.
Ticket #2401 06/11/11 00:11:09 Josh Smift: plastic-103 Washington -> BBN was stuck; restarted the client. Stanford -> Washington. Rutgers -> Stanford still doesn't work
Ticket #2401 06/11/11 00:11:09 Josh Smift: plastic-105 BBN -> GT had died somewhere in here; I started up the client, and it's running fine now.
Ticket #2401 06/11/11 00:11:09 Josh Smift: plastic-107 Clemson -> Indiana was stuck;
Ticket #2401 06/11/11 00:11:09 Josh Smift: plastic-108 Rutgers -> Indiana had been retrying and was then stuck
Ticket #2401 06/11/11 00:11:09 Josh Smift: plastic-109 Every client was stuck
Ticket #2401 06/11/11 00:11:09 Josh Smift: plastic-110 Every client was stuck
Ticket #2401 06/12/11 13:38:39 Josh Smift: plastic-101 Indiana -> Rutgers, no pings flowing. At Rutgers, orbitplc2 rebooted + Indiana FV outage.
Ticket #2401 06/12/11 13:38:39 Josh Smift: plastic-103 Stanford -> BBN was stuck + Stanford -> Washington was stuck + Rutgers -> Stanford was down
Ticket #2401 06/12/11 13:38:39 Josh Smift: plastic-104 BBN -> Rutgers was down
Ticket #2401 06/12/11 13:38:39 Josh Smift: plastic-105 GT -> Wisconsin was down, the client at GT had died
Ticket #2401 06/12/11 13:38:39 Josh Smift: plastic-106 Indiana -> GT isn't working
Ticket #2401 06/12/11 13:38:39 Josh Smift: plastic-108 Wisconsin -> Stanford was stuck; Stanford -> Rutgers was stuck; Rutgers -> Indiana is down; Indiana -> Wisconsin was stuck
Ticket #2401 06/12/11 13:38:39 Josh Smift: plastic-109 Every connection was stuck
Ticket #2401 06/12/11 13:38:39 Josh Smift: plastic-110 Stanford -> Rutgers down; Stanford -> Rutgers down; Indiana -> GT down; Restart FlowVisor on naxos.
Ticket #2401 06/13/11 12:36:01 Josh Smift: plastic-101 Indiana -> Rutgers still has no pings flowing, because orbitplc2 still has no static ARP.
Ticket #2401 06/13/11 12:36:01 Josh Smift: plastic-102 Rutgers -> Stanford still has no pings flowing, because orbitplc2 still has no static ARP.
Ticket #2401 06/13/11 12:36:01 Josh Smift: plastic-103 Stanford -> BBN down; Stanford -> Washington down; Rutgers -> Stanford down
Ticket #2401 06/13/11 12:36:01 Josh Smift: plastic-105 GT -> Wisconsin down
Ticket #2401 06/13/11 12:36:01 Josh Smift: plastic-108 Wisconsin -> Stanford was stuck; Rutgers -> Indiana is down; Indiana -> Wisconsin was stuck
Ticket #2401 06/13/11 12:36:01 Josh Smift: plastic-109 Everyone *except* Wisconsin -> BBN was stuck
Ticket #2401 06/13/11 12:36:01 Josh Smift: plastic-110 Everyone was stuck or dead.
Ticket #2401 06/13/11 20:01:41 Josh Smift: plastic-103 Wisconsin -> Stanford was stuck
Ticket #2401 06/13/11 20:01:41 Josh Smift: plastic-110 Washington -> Stanford was stuck
Ticket #2401 06/14/11 11:39:26 Josh Smift: No traffic to or from Stanford
Ticket #2401 06/14/11 12:01:20 Josh Smift: missed one of the NOX controllers, which caused the FV on 33106 to crash
Ticket #2401 06/14/11 14:03:01 Josh Smift: plastic-103 Everything was down
Ticket #2401 06/14/11 14:03:01 Josh Smift: plastic-105 Everything was down
Ticket #2401 06/14/11 14:03:01 Josh Smift: plastic-108 Wisconsin -> Stanford was down; Rutgers -> Indiana was down;
Ticket #2401 06/14/11 14:03:01 Josh Smift: plastic-109 Everything was down except Wisconsin -> BBN;
Ticket #2401 06/14/11 14:03:01 Josh Smift: plastic-110 Everything was down except GT -> BBN;
Ticket #2401 06/15/11 15:20:05 Josh Smift: derailed by a problem with the NLR core and then the I2 core
Ticket #2401 06/15/11 15:20:05 Josh Smift: plastic-101 Rutgers -> Stanford and Stanford -> Washington are both down.
Ticket #2401 06/15/11 15:20:05 Josh Smift: plastic-102 Rutgers -> Stanford and Stanford -> Washington are both down.
Ticket #2401 06/15/11 15:20:05 Josh Smift: plastic-103 Stanford -> BBN was down; BBN -> Rutgers was down; Stanford -> Washington was down; and Rutgers -> Stanford ;
Ticket #2401 06/15/11 15:20:05 Josh Smift: plastic-108 Wisconsin -> Stanford was stuck; Stanford -> Rutgers couldn't connect
Ticket #2401 06/15/11 15:20:05 Josh Smift: plastic-109 Clemson -> Stanford was stuck; Stanford -> Wisconsin was stuck;
Ticket #2401 06/15/11 15:20:05 Josh Smift: plastic-110 Washington -> Stanford was stuck; Stanford -> Rutgers was stuck; Rutgers -> Indiana was stuck; Indiana -> GT was stuck;

Baseline 7 Outages

Baseline 8 Outages

Last modified 13 years ago Last modified on 06/17/11 11:16:57