BBN worker nodes have inconsistent /etc/resolv.conf

[jbs@bbn.com]

On the BBN worker nodes, /etc/resolv.conf is inconsistent:

[11:48:31] jbs@bbn-hn:/home/jbs
+$ for i in {1..3} 5 6 8 ; do echo "--> bbn-w$i" ; ssh bbn-w$i grep -v \'#\' /etc/resolv.conf ; echo ; done
--> bbn-w1
domain exogeni.net
search exogeni.net renci.org exogeni.gpolab.bbn.com

--> bbn-w2
domain exogeni.net
search exogeni.net renci.org exogeni.gpolab.bbn.com

--> bbn-w3
domain exogeni.net
search exogeni.net renci.org exogeni.gpolab.bbn.com

--> bbn-w5
; generated by /sbin/dhclient-script
search local
nameserver 10.100.0.1

--> bbn-w6
; generated by /sbin/dhclient-script
search local
nameserver 10.100.0.1

--> bbn-w8
; generated by /sbin/dhclient-script
search local
nameserver 10.100.0.1

(I left out 4 and 7 because they're not accessible at the moment.)

[vjo@duke.edu]

This should be resolved; please verify.

[jonmills@renci.org]

Greetings,

/etc/resolv.conf on rack head nodes is now managed by Puppet, from control.exogeni.net. Not having this before was an oversight on my part, and part of my larger systemic DNS woes. I think I'm on the path to clean those up, however.

BTW, resolv.conf on worker nodes has always been puppet managed, from the first rack -- it just points back to the rack head nodes' persistent ip address for the 1007 vlan (10.100.0.1)

As of yesterday, every rack head node is running a named that is managed by Puppet. Please reference /etc/named.conf, /etc/named/*, and /etc/rndc.conf. These files configure named on a head node to reach out for zone transfers from ns1.exogeni.net. So we have real DNS replication now. Do note that on the public side, exogeni.net, all rack head nodes receive this zone. However, on the management side, a rack head node only pulls down the management zone for its rack. This is consistent with our prior discussed notions of rack segregation. Because why should one rack's head node's DNS contain a DNS record for, let's say, the dataplane switch at another rack? We don't want rack/site-specific admins reaching out to other racks anyway. And xoadmins know all the racks and can resolve DNS to any of them from control.exogeni.net.

So, with these new settings, every rack head node now has a named running locally, which has a copy of the public zone, and a copy of its own private zone, and via puppet-managed resolv.conf, is configured to use itself (127.0.0.1) as its resolver, and to fall back to 192.168.100.2 (control.renci.xo) just in case.

[jbs@bbn.com]

This looks good now:

[17:27:39] jbs@bbn-hn:/home/jbs
+$ for i in {1..8} ; do echo "--> bbn-w$i" ; ssh bbn-w$i grep -v \'#\' /etc/resolv.conf ; echo ; done
--> bbn-w1
search local exogeni.net
nameserver 10.100.0.1

--> bbn-w2
search local exogeni.net
nameserver 10.100.0.1

--> bbn-w3
search local exogeni.net
nameserver 10.100.0.1

--> bbn-w4
search local exogeni.net
nameserver 10.100.0.1

--> bbn-w5
search local exogeni.net
nameserver 10.100.0.1

--> bbn-w6
search local exogeni.net
nameserver 10.100.0.1

--> bbn-w7
search local exogeni.net
nameserver 10.100.0.1

--> bbn-w8
search local exogeni.net
nameserver 10.100.0.1

So, that's it for this one.