placement of PDUs within rack should be documented on the exogeni wiki

[chaos@bbn.com]

The PDUs in the BBN rack are labelled on the cable end as PDU1 - PDU6. However, they are not labelled on the PDU end, and i could not find a rack diagram or text on the wiki showing which PDU is in which location. As a result, i can't map which item in the rack is plugged into which floor circuit. Document the generic and/or per-site locations of PDUs within the racks, so that sites can label and map power connections according to their site practices.

[viviano@renci.org]

Not sure I agree this is something ExoGENI ops should maintain. Power layouts are site specific and generally handled by each sites facilities coordinator. The BBN site was a special case by sending a RENCI staffer out for configuration but the remaining sites will be connected by the local team. Those sites will handle the power/network cabling to their site standards and should be documented in their facilities paperwork. I am open to discussion on how you want to handle this. We can create a page for each site where they can add their own documentation but I don't see that ExoGENI ops can reasonably maintain such site specific information without an understanding of the site.

[chaos@bbn.com]

Yeah, so, to be clear, this is the numbering of the PDUs within the rack (not the circuits to which they're connected on the floor or anything, which is definitely in our court).

You may convince me this is a specific problem rather than a general one: the specific problem i have is that when the rack was delivered and cabled, the PDU cables were labelled PDU1 - PDU6, and i don't in fact actually know which of those is which.

The general problem is that if you guys are naming the PDUs, labelling the PDUs, and putting the PDUs in consistent places in the racks, as a rule, those places should be documented. But if you're not doing those things, then i agree it doesn't make as much sense (and that if a given site wants that information on their site page on the exogeni wiki for whatever reason, they can put it there).

[viviano@renci.org]

Replying to chaos@bbn.com:

Yeah, so, to be clear, this is the numbering of the PDUs within the rack (not the circuits to which they're connected on the floor or anything, which is definitely in our court).

You may convince me this is a specific problem rather than a general one: the specific problem i have is that when the rack was delivered and cabled, the PDU cables were labelled PDU1 - PDU6, and i don't in fact actually know which of those is which.

The general problem is that if you guys are naming the PDUs, labelling the PDUs, and putting the PDUs in consistent places in the racks, as a rule, those places should be documented. But if you're not doing those things, then i agree it doesn't make as much sense (and that if a given site wants that information on their site page on the exogeni wiki for whatever reason, they can put it there).

The layout of the PDU's is site specific because of the multiple power options. All 208V PDU's for example are going to be mounted in 0U locations in the skins of the rack and not user accessible without removing the skins of the rack. BBN's site is unique because IBM doesn't offer a 0U UPS for 110V. Otherwise your PDU's would be buried inside the rack like RENCI's.

As far as ExoGENI OPS is concerned, we just need spark. We don't care where it comes from or how it breaks out at the facility. If we need a site to hard reset a piece of equipment, we ask them to unplug the specific component at RMU X not at the PDU itself. We provide the PDU's simply as a convenience to the site to make it easier to connect the ExoGENI hardware and consider them part of the sites facility.

[viviano@renci.org]

Replying to viviano@renci.org:

Replying to chaos@bbn.com:

Yeah, so, to be clear, this is the numbering of the PDUs within the rack (not the circuits to which they're connected on the floor or anything, which is definitely in our court).

You may convince me this is a specific problem rather than a general one: the specific problem i have is that when the rack was delivered and cabled, the PDU cables were labelled PDU1 - PDU6, and i don't in fact actually know which of those is which.

The general problem is that if you guys are naming the PDUs, labelling the PDUs, and putting the PDUs in consistent places in the racks, as a rule, those places should be documented. But if you're not doing those things, then i agree it doesn't make as much sense (and that if a given site wants that information on their site page on the exogeni wiki for whatever reason, they can put it there).

The layout of the PDU's is site specific because of the multiple power options. All 208V PDU's for example are going to be mounted in 0U locations in the skins of the rack and not user accessible without removing the skins of the rack. BBN's site is unique because IBM doesn't offer a 0U UPS for 110V. Otherwise your PDU's would be buried inside the rack like RENCI's.

As far as ExoGENI OPS is concerned, we just need spark. We don't care where it comes from or how it breaks out at the facility. If we need a site to hard reset a piece of equipment, we ask them to unplug the specific component at RMU X not at the PDU itself. We provide the PDU's simply as a convenience to the site to make it easier to connect the ExoGENI hardware and consider them part of the sites facility.

Correction

Meant to say "IBM doesn't offer a 0U PDU for 110V" not UPS :).

[chaos@bbn.com]

Brad, the "we just need spark" idea was never communicated to us, and we went to a great deal of trouble to implement the circuits needed for your specific PDUs, which were not compatible with any circuits existing in our lab.

I'm going to fork off the normative question (what should happen in the general case) into e-mail.

In the case of BBN's specific rack, there are six PDUs which were loaded into the rack, and, since the cables are numbered (1-6) using ExoGENI labels, it is pretty clear to me that someone had an ordering in mind of the six PDUs within the rack. However, the six PDUs within the rack are not numbered or otherwise labelled at all (that i can find). I need to know which PDU is which circuit. In order to figure that out from scratch, i have to rip up my floor. Can whoever came up with the 1-6 numbering and figured out which was which when you installed the rack, please tell me what those numbers are. The locations of the six PDUs within the rack are:

A  D
B  E
C  F

along the two edges of the rack, when you look at it from the rear. Please pass along the mapping of A-F to 1-6, so that we have that information.

[viviano@renci.org]

I talked with Jonathan who is the RENCI person that was on site during the BBN install. The IBM tech labeled the PDU's 1-6 when he was there. Jonathan did nothing with the PDU's as it was IBM's job to coordinate with the local facilities person to get power. Who at BBN was there when power was connected and ensured everything was plugged in to the right location in your facility? That person would be the one to have the PDU map. Again, ExoGENI just needs spark, we don't care where its coming from, we view the connection to the power grid as a facilities level activity not part of ExoGENI ops.

[chaos@bbn.com]

Brad, you guys delivered our rack two months before we were ready to accept it, and we scrambled to be at all ready to take it, at your request. Please be a human, don't make me waste 8 hours ripping up my floor for something i could have reasonably assumed based on what you were saying at the time you were tracking, and check with IBM about this.

[viviano@renci.org]

The discussion of when the rack was delivered and why it was delivered by that date is above my pay grade. I'll defer that to Ilia and Heidi on that. I did what I was told, which is fight IBM to get the rack delivered to BBN by a certain date. I wasn't on site during the install, I don't have any details to offer you in regards to how the PDU's where plugged into the floor at BBN. If I had the information, I'd give it you, but I don't. We had no plans to track PDU layouts in the facility because we don't care where they go. We provide PDU's in line with what power the site can provide and simply tell you the number of connections we need (in your case 6).

Do you guys have a circuit tracer? I have something like this, it saves me a lot of time when something gets missed on our documentation schedules:

​http://www.amazon.com/Reliance-Controls-THP110-Digital-Circuit/dp/B000IF9KBC/ref=sr_1_3?ie=UTF8&qid=1337008379&sr=8-3

You just plug it into each PDU in turn and point it at the outlet until you find the one that beeps.

[chaos@bbn.com]

Brad: thanks for the follow-up. Obviously, you can't give me information you don't have, regardless of how much time it would save me to have it. :>) I made the bad assumption that, because i saw the labels on the cable ends of the PDUs, the information i needed was already there for whenever i wanted to go look for it.

I agree that us getting a circuit tracer would be a nondestructive and fast way to solve this problem. We've never used one, but i'll see if we can get our hands on one and try it out.

Anyway, thanks for iterating, and i will take up the more general question (which is above my pay grade too) via e-mail.

[jonmills@renci.org]

Chaos,

Would it help if I simply apologize for not correctly labeling both ends of the PDUs? I am sorry. I don't want you to have to waste time lifting up the two floor boards between the plugs and the rack and tracing the cables back. But what can you reasonably expect us to do about it now? I cannot very well go back to BBN and fix it. What's done is done. It was a busy week and we all put in a lot of effort to pull the rack together. I'm sorry I missed this detail.

--Jonathan

[chaos@bbn.com]

Jonathan: thanks for the apology. On my end, i'm terribly sorry for getting annoyed beyond the initial inquiry. It isn't that big an issue. I think Brad's suggestion that we get our hands on a circuit tracer is a really good one, and will save everyone time. I've opened an internal ticket to do that.

[chaos@bbn.com]

Peter was able to trace the circuits by hand yesterday, by opening the floor. (Thanks, Peter!) He determined that the seven PDUs in the rack are connected as:

   N
3  6
2  5
1  4

(when you look at the rack from the back), where N is the seventh PDU which is not plugged into a circuit.

Mentioning this FYI, on the off chance that it is IBM's standard numbering scheme and the information helps you in the future.

[chaos@bbn.com]

Based on Peter's success in tracing the cables, and on our e-mail discussion of options for powering our racks under UPS, i am happy to close this ticket.

For what it's worth, i found Brad's two e-mails describing power and conversion options for putting a rack's management plane onto UPS, to be extremely useful. I think that information would be sufficiently useful to prospective site admins that you might consider simply copy/pasting it onto a wiki page if you don't have time to edit it further.