Opened 9 years ago

Last modified 9 years ago

#673 new

Move SPP node in Washington D.C. from current rack to new rack

Reported by: hdempsey@bbn.com Owned by: matt@internet2.edu
Priority: major Milestone:
Component: SPP Version: SPIRAL2
Keywords: Cc: jdd@arl.wustl.edu, Vic Thomas, ricci@cs.utah.edu, jwilliams@bbn.com
Dependencies: #564

Description

The Internet2 engineers want to move the SPP in Washington D.C. to a new rack from the rack it was originally installed in, which also housed the now-defunct Ciena connection to I2.

This work is currently scheduled for 10/19. John Dehart of WUSTL has asked I2 to notify him when the move is done and to stay onsite until John finishes testing to verify that the node still works.

Hans Addleman addlema@grnoc.iu.edu is coordinating the install from the GMOC side.

Dependency on OpenFlow install ticket because both activities are scheduled for the same I2 team, but there is no technical dependencies to OpenFlow.

Change History (4)

comment:1 Changed 9 years ago by hdempsey@bbn.com

Cc: jdd@arl.wustl.edu added

comment:2 Changed 9 years ago by hdempsey@bbn.com

John Dehart reported problems with performance through the interfaces that go over connections to the I2 IP network (not through ProtoGENI switches). He is working with Rob Ricci of Utah and Matt Zekauskas of I2 to isolate the cause and resolve it. I2 install team is still in D.C.

From John:

"For the sftp that works well from SALT Src Addr (data transfered from this address): 128.252.202.128 Dst Addr (data received on this address): 64.57.23.210

For the sftp that does not work well from KANS Src Addr (data transfered from this address): 128.252.202.128 Dst Addr (data received on this address): 64.57.23.178

I don't have any baseline tests from the past for a similar transfer. My recollection is that the transfer rates always varied among the sites but it wasn't painful until recently. Sorry, I can't define recently very precisely."

From Matt:

And to make sure I understand the whole thread...

  • you're testing IP-IP using the connections you have the Internet2 IP

network

  • but Rob also has some packet drop counts on his switches, but they

are not in this path

Rob mentioned that the IP transfers use the same path as the ProtoGENI backbone-backbone link; it's probably true that the same routers are used, but the intermediate backbone links could be different (we have parallel 10G links between cities), and the endpoints are definitely different -- the ProtoGENI switches are directly connected to the router with their own 1G interfaces (currently); the IP connections are through a 10GE connection to the router via a different HP switch (the "Observatory" or "racklan" switch).

And just in case... do you have any baseline tests that worked earlier?

(I just want to check that this behavior isn't old, if we know...)

Assuming the same WASH interface is used for both transfers, the results would indicate that the problems are likely not at the WASH end...

comment:3 Changed 9 years ago by hdempsey@bbn.com

More from John:

All:

I think I have been able to convince myself that there is something wrong inside the SPP chassis at KANS. I don't know what yet but I now believe that every packet that leaves its interfaces is indeed reaching the other end. The packet drops for the tests I am running are inside the SPP chassis. Also, it is pkt rate related. At VERY low rates I can get 100% of my packets through.

I'll keep you posted.

John

On 10/20/2010 11:04 AM, John Dehart wrote: All,

This discussion has gotten a little tangled so I wanted to try to clarify where things are right now.

I am tackling two issues right now and it is still uncertain if they are related or not.

  1. The WASH SPP chassis was moved yesterday and I have been testing all the interfaces

to verify that they work properly. In testing the WASH to KANS links which go through the ProtoGENI network, I had 95% packet loss. That is the only issue I have come up with so far that pertains to WASH. I have not discovered any issues with the SPP interfaces at WASH that go directly to I2. And the same test over the WASH to SALT links had 0 pkt loss.

  1. In our preparation for GEC-9, Jon Turner and I have noticed that sftp transfers to and from KANS are at least 10 times slower than to/from SALT.

My immediate plans are to

  1. Re-check the configuration of our line card at KANS to make sure something didn't get mis-configured when we made the changes for the new I2 bandwidth.
  2. Call Camille at the WASH node and ask her to re-seat the cables for the WASH to KANS links

John

comment:4 Changed 9 years ago by matt@internet2.edu

More from John:

Mea culpa.

KANS is now back to normal.

In a recent round of bug fixes the wrong image got installed on the line card at KANS. Unfortunately it was only when I started re-testing WASH this week that it really came to light that there was a problem.

My apologies for suggesting there might be an I2 problem before checking every detail on my side.

The interface between WASH and KANS now also works fine. I have no outstanding issues with the move of the WASH SPP.

John

Note: See TracTickets for help on using tickets.