Notes from July 21, 2010 ad hoc meeting on backbone migration at GEC8

Kristin Rauschenbach: The GPO has been relying on donations from NLR and I2 for backbone services for the past two years. Those MOUs are expiring, and we're now in discussions with I2 about how to migrate, and what the next MOU will be. Both NLR and I2 are committed to GENI. The GPO is not in a position to pay for backbone services, we completely rely on donations during these prototyping years. We hope that the project will succeed and convert into being able to pay for services in the future, but we're not there yet. We also want to make sure that what NLR and I2 supply is affordable to them, while being useful and used by the community, and by that process, give feedback to I2 and NLR about what kinds of services GENI will want to use in the future. This is a good chance to talk about that.

KR: There was a point raised about migration steps, and requirements for the next couple months. There was a need that I2 had to get us off equipment that was dedicated to the first MOU, and is now being repurposed. So there is a near term and a long term. We're still discussing the long-term shape of things, so this is a key time to talk about the needs of the community.

KR: We'll open it up first to GENI users, to talk about needs; then maybe we'll hear from Randy of I2 and Glenn of NLR to talk a little about what they have, and their reactions to what they've heard.

Jon Turner: Was taken aback when we first learned of the backbone being reduced from 10G to 1G, as is being discussed, as one of the current users of the 10G capability. Both because it directly impacts what we're trying to do now and because it's important for the future of people doing development on GENI to know that they have resources that will be able to support large-scale deployment. A large part of why we are building GENI is to be able to support experimental networks that scale up to tens of thousands of users. 10G sounds like a lot, but it's not a lot of bandwidth for that kind of scale -- it's the low end for that, not the high end. The real risk of cutting back is that we make it less attractive for developers to build and deploy things, which will actually attract users. We have a long way to go, and we understand that resources aren't being used much at this point, but that's just a natural product of the fact that we're still in development mode.

Rob Ricci: My perspective is similar to but slightly different from John's. Building a bunch of disconnected facilities is not the goal, building something that's a research facility that's unified, in some sense, is the goal. An inability to connect these things at relatively low latencies in new non-IP networks, and at high bandwidths to attract users is bad. We want to make GENI a scientific facility on which you can get at least some level of understanding of what is the performance you can expect on the network. The 10G wave that was provided by I2, and the current resources from NLR, meet some of those goals. We appreciate that I2 has looked at using ION to get connections onto more campuses -- this is extremely important, because we have to connect this out to the users and the resources. The question is whether a medium amount of bandwidth is reserved for these kinds of slices; maybe it doesn't need to be reserved for slices all the time, maybe best-effort plus reservable bandwidth might be more economical and still suitable.

KR: High speeds can be important; there's always a chicken-and-egg problem. It's important for GENI to be at the edge of what's possible and what's viable. Moving away from dedicated bandwidth may make that more affordable. Also important is to think about last mile, how many people can we connect, and how does that affect utilization of high speeds in the core.

James Sterbenz: 10G is not particularly high-bandwidth any more. It is low for the core; 1G will be completely inadequate even for the small number of current users. If we don't have dedicated bandwidth, we won't be able to support the community; if we only have 1G, we might as well not even bother with GENI racks.

Randy Frank: The issue today is that what GENI has from I2, it's a dedicated wave that can't be used by anyone else. There's room to discuss continuing 10G access to the core. That's on the table; we want to discuss what the needs are. The key issue is that the current dedicated 10G waves were provisioned out of spare capacity, and that capacity isn't spare any more, that now needs to go back into the core. Continuing with that static reservation that is only for GENI use, and idle when GENI isn't using, is not on the table. What is on the table is what are the issues we'll face if we can get 10G interfaces into the backbone and which will be shared by other competing uses of I2. One of the services included in that is a reservable service. You're still coexisting with other uses, and you may not be able to get a reservation all the time, and they're only 2 week reservations, but they are reservable.

JS: I thought "we were Internet2".

RF: You are.

JS: I've heard rumors that NOAA is why educational usage is losing bandwidth.

RF: in the planning for the current 10x10 footprint, they made assumptions about how they'd get used. Included in those assumptions was the idea that some percentage of those waves would be used for the core I2 backbone, other waves would be used by other science communities. A major source of early funding was ESnet -- we could not have gotten that 10x10 footprint just on I2 funding. Early on, very little of the bandwidth was being used, so the footprint was very expensive in the early years; and the assumption was that it would eventually convert to 100% paying clients. One of those clients is NOAA, who is using this in support of the research communities, e.g. to connect the NOAA datacenter, and then gateway to I2. So instead of NOAA being a member of I2, their buying dedicated bandwidth removes their datacenter-to-datanceter traffic off of the backbone, but puts it onto the 10x10 waves. The same is true for ESnet. Having those on the backbone means that they don't compete with other traffic; even though some of their traffic ends up in research labs.

JS: So NOAA is now taking a dedicated 10 G wave?

RF: Yes -- but they're paying for that bandwidth. Part of the original assumption was that the 10x10 wave would be used by paying customers.

Jon DeHart: How much would the GENI donation cost?

RF: It's complicated because of amortization and stuff, but in the ballpark of 1.8 million per year. This isn't a real figure of how much a new circuit would cost, our bill to Level 3 has been constant for ten years, so there are opportunity costs -- it's complicated.

Rick McGeer: How long a wave would GENI need today and, and how many POPs?

JT and RR: SPP in 5 POPs, ProtoGENI in 8. SLC, KC, DC, HOU, ATL, CHI, SEA, LA/NY. A single 10G wave across the entire infrastructure.

KR: So to sum up: There are needs for 10 G services; RF says that 10 G is on the table, backbone access needs to be worked out; NLR has been providing 10G services. The GPO will try to get more details about what's needed, and match that with what's available.

JT: We know what's needed -- we know where the interfaces are and wher the network is going to be -- we don't need to gather that information, we already have that information.

KR: We still need to negotiate with I2, and based on that decide whether I2 or NLR services being provided to GENI is the right way to connect those locations.

RF: If we connect at 10G, that would enable bursting up to that speed. You can't sustain 10G on those interfaces for arbitrary lengths of time. 10G interfaces isn't the same as sustained bandwidth. ION can give you reservations, but that's different.

Glenn Ricart: NLR is in a slightly more fortunate position because we have more capacity on our own fiber, and aren't up against the limit at this time. We have dedicated 10 G on the network for GENI, and are in the cities Jon and Rob mentioned, and are ready to help. We're not trying to take anything away from anyone else, just trying to make the right things happen. We do have a problem in higher education network research, where no one wants to pay for bandwidth. As nice as it would be for NLR to always provide free service, we can't do that. We're not at the pinch point that I2 is now, but in the long range, we need some federal GENI project funding to make these things happen. It's just not a stable model to provide free service for networking. If no one pays them, how do we afford the next-generation technology? We just can't. We're dedicated to research and to education, we're happy to have GENI, but there's a serious problem in the community that we need to fix.

KR: We certainly don't view this as a long-term proposition. This is a period of time in the development of GENI, in which the GPO is very interested in continuing the donation of the backbone services, as one component of a very complex system. Backbone service isn't even the highest risk part of GENI, and we're trying to do the best we can to prove GENI to the community with the component resources that we have at hand. The major issue at hand is in Jon Turner's area, and we need to discuss this with their group.

KR: Heidi, Anything else for the overarching community?

HPD: no.

Matt Zekauskas: Glenn, can NLR provide 10G in all of those POPs with no additional funding?

GR: We've aggreed to provide 10G on our backbone, and if we have to add space at POPs, we'd have to get additional funding for that. The transponders for the GENI wave are deployed and in. There might be equipment like ports, OpenFlow routers, and if e.g. they need to put a new open flow router in LA, where there isn't one currently, that would be an additional expense that isn't funded now.

JS: How long is the NLR agreement for OF?

KR: Two more years.

Jim Griffioen: ShadowNet is in the same boat as Jon, and has the same constraints. They've been trying to put in 10G links to the reserved I2 wave, but now it sounds like they'd be better off going to the main I2 backbone.

RF: Many campuses now have 10G to regionals, and thus to I2. Putting equipment in POPs is tremendously expensive. It's the biggest profit center for Level 3, and is also logistically difficult. So unless there's a compelling reason to put stuff there, as a long-term strategy, getting stuff out of POPs and onto campuses makes sense. (GR agrees.) We're increasingly having trouble getting any space at any cost. GENI racks in POPs is not cheap. Where there is an argument for keeping equipment in POPs, from that point of view, we think there are discussions that could take place, and we can talk about how it could connect at 10G, just like from campuses.

GR: I agree about all of that. One of the things we're trying to do is work with regionals, and try to take that bandwidth through to the campus, to locate gear in either regional or campus facilities.

KR: The Quilt will be having a discussion Thu/Fri to talk about those issues. We'll include ShadowNet as a near-term concern for this backbone migration.

Joe Mambretti: It'd be good if we could sort these topics, since there are many different topics here. (1) Regionals haven't been mentioned enough, they have a big role to play. (2) Policies: It would've been nice if I2 bought NLR capacity instead of L3, that's a policy type thing. (3) "We are I2"; what are the policy decisions that led to this, is a communications topic. There should be a process where this community has an ear in I2 discussions. (RF: Or we need CxOs with better ears.) (4) Tech is driving down costs, so we should be seeing lower costs. In Japan, we have 1G to the home -- why does GENI have to struggle to get that in the backbone? (5) Economics: Right now 10G is common, we're ready to go to 100G, Starlight doesn't even count 10G connections any more, so there's something wrong if we're told that GENI can't afford 10G (JS and JT applaud). (6) 1.8 million as a cost of something -- that's a small number, compared to astrophysics, there has to be unity in this community, saying in unison that we're a valid area of research and need to be funded. Network research has boosted GNP more than any effort in the US.

Rick McGeer: I heard two extreme numbers here -- one is zero, which is nice, but unsustainable; one is 1.8 million, which is a substantial chunk of Chip's budget. On the other hand, I have a 1/3 K wave (?) of my own.

JM: There's something funny about the numbers.

RF: The 1.8 is not a today's-cost number, it includes amortization and stuff over 10 years. There's also an new VTPO (?) award to build more stuff. What we're talking about is what we can do is on August 1st. It may be that as we transition the core to 100G, we'll gain the ability to do a lot more. The 1.8 mil is the valuation that GENI used, but that's an amortized cost over 10 years, not the marginal cost of providing a new wave.

RM: I'm not asking about bookkeeping, I'm saying that if someone said today "we're going to solve this problem by writing a check", how large is that check?

RF: We'd need to know the POPs, and how many waves --

JT: What we have right now!

RF: You're not using what you have right now, and that may be overkill. JS: NOAA is getting these waves and we're losing them.

RF: No, but the other big use is on the some of the critical national paths, there is bandwidth needs on certain paths in the GENI footprint.

Peter O'Neil: It's more complicated than that. (Talks about why.) RF: Well, it's even more complicated than that. (Various people discuss about numbers and providers and ten-packs of waves.)

Chip: If you're going to have TB bandwidth in 6 - 12 months, can we talk about a forward-loking MOU?

RF: We’re not prepared to talk about an MOU on a network we haven't even begun to build yet.

KR: But you will get there, and we will be discussing it. Joe's points are good; some of them are global to this community and beyond and GENI may not be able to address them all. Most importantly 1.8 million is well beyond our budget now.

RF: That's not the number.

JD: Let's find out what that number would be. What if we wanted to buy the footprint we have right now. Not a quote right here and now, but let's find that out.

RF: Ok.

JD: There are two decisions: What are we doing for August 1st, and what's the long-term plan. We can't keep doing this with ad hoc meetings every two years.

KR: It's definitely important, and the disruption from this migration was part of why I2 and the GPO said let's talk about the future. We don't want this to be the end of the discussion for even the next two years, much less beyond that. I'm eyeing that TB of bandwidth, too.

JS: How much is NOAA paying?

RF: I don't know the number.

JS: Can you find out?

RF: I'll find out, if it's public.

KR: We will continue to discuss this with I2. Near term will include those affected by the immediate migration, and also longer term possibilities. Thanks to everyone for coming.

Last modified 14 years ago Last modified on 07/29/10 11:21:49