wiki:JBSsandbox/SliceNotes

Version 3 (modified by Josh Smift, 6 years ago) (diff)

--

Slice notes

Here are some notes about things I like to do when I set up a slice, many of which I learned from Plastic Slices.

Choose what to use

Slices

Use one of these to identify which slices to use:

slices=$(echo gpoI{15,16})
slices=$(echo ps{103..110})
slices=$(echo ps{101..102})

Logins

Use one of these to create a file and set some variables containing the logins you want to use:

logins cat
logins egrep -h bbn
logins egrep -h -- '(-ig-|-pg-)'

Those create ~/tmp/logins-$slicename.txt for each slice, and put all the logins into $logins. They rely on a shell function like this:

logins () { for slicename in $slices ; do loginfile=~/tmp/logins-$slicename.txt ; $* ~/slices/*/logins/logins-$slicename.txt >| $loginfile ; done ; logins=$(for slicename in $slices ; do loginfile=~/tmp/logins-$slicename.txt ; cat $loginfile ; done) ; }

rspecs

Always do this:

declare -A rspecs

Then use something like one of these to identify which rspecs to use:

for slicename in $slices ; do rspecs[$slicename]=$(ls -1 ~/rspecs/request/$slicename/*.rspec) ; done
for slicename in $slices ; do rspecs[$slicename]=$(ls -1 ~/rspecs/request/$slicename/*.rspec | grep -v openflow) ; done
for slicename in $slices ; do rspecs[$slicename]=$(ls -1 ~/rspecs/request/$slicename/*.rspec | egrep '(bbn|utah)') ; done

Check what you've got:

for slicename in $slices ; do echo ${rspecs[$slicename]} ; done

Credentials

Fetch my user and slice credentials:

(cd ~/.gcf ; omni getusercred -o ; for slicename in $slices ; do omni getslicecred $slicename -o ; done)

Slice and sliver stuff

Create and renew slices

for slicename in $slices ; do omni createslice $slicename ; done

renewdate='2014-05-15 23:00 UTC'
for slicename in $slices ; do omni renewslice $slicename "$renewdate" ; done

Create and renew slivers

for slicename in $slices ; do for rspec in ${rspecs[$slicename]} ; do somni $slicename $rspec ; omni --usercredfile=$HOME/.gcf/$USER-geni-usercred.xml --slicecredfile=$HOME/.gcf/$slicename-cred.xml -a $am createsliver $slicename $rspec & done ; sleep 30s ; done

renewdate='2014-05-15 23:00 UTC'
for slicename in $slices ; do for rspec in ${rspecs[$slicename]} ; do somni $slicename $rspec ; omni --usercredfile=$HOME/.gcf/$USER-geni-usercred.xml --slicecredfile=$HOME/.gcf/$slicename-cred.xml -a $am renewsliver $slicename "$renewdate" & done ; sleep 5s ; done

Utah-specific hackeration

slices=$(echo ps{103..110} gpoI{15,16})
(cd ~/.gcf ; omni getusercred -o ; for slicename in $slices ; do omni getslicecred $slicename -o ; done)
declare -A rspecs
for slicename in $slices ; do rspecs[$slicename]=$(ls -1 ~/rspecs/request/$slicename/*.rspec | grep utah | egrep -v '(openflow|vts)') ; done
for slicename in $slices ; do echo ${rspecs[$slicename]} ; done

renewdate="$(date +%Y-%m-%d -d 'now + 4 days') 23:00 UTC"
for slicename in $slices ; do for rspec in ${rspecs[$slicename]} ; do somni $slicename $rspec ; omni --usercredfile=$HOME/.gcf/$USER-geni-usercred.xml --slicecredfile=$HOME/.gcf/$slicename-cred.xml -a $am renewsliver $slicename "$renewdate" & done ; sleep 5s ; done

ExoGENI-specific hackeration

slices=$(echo ps{103..110} gpoI{15,16}) 
(cd ~/.gcf ; omni getusercred -o ; for slicename in $slices ; do omni getslicecred $slicename -o ; done)
declare -A rspecs
for slicename in $slices ; do rspecs[$slicename]=$(ls -1 ~/rspecs/request/$slicename/*.rspec | grep exogeni | grep -v openflow) ; done
for slicename in $slices ; do echo ${rspecs[$slicename]} ; done

renewdate="$(date +%Y-%m-%d -d 'now + 13 days') 23:00 UTC"
for slicename in $slices ; do for rspec in ${rspecs[$slicename]} ; do somni $slicename $rspec ; omni --usercredfile=$HOME/.gcf/$USER-geni-usercred.xml --slicecredfile=$HOME/.gcf/$slicename-cred.xml -a $am renewsliver $slicename "$renewdate" & done ; sleep 5s ; done

Check sliver expiration

For a few slivers, you can just run the commands and eyeball the results as they fly past. For a lot of slivers, you can stash the results in some files and then analyze them afterwards.

For a few slivers

for slicename in $slices ; do for rspec in ${rspecs[$slicename]} ; do somni $slicename $rspec ; omni --usercredfile=$HOME/.gcf/$USER-geni-usercred.xml --slicecredfile=$HOME/.gcf/$slicename-cred.xml -a $am sliverstatus $slicename |& grep _expir || echo "no sliver" ; echo "is for $slicename @ $am" ; done ; done

For lots of slivers

Gather up expiration information, and stuff it into a results file:

for slicename in $slices
do
  cd
  rm -rf ~/tmp/renewsliver/$slicename
  mkdir -p ~/tmp/renewsliver/$slicename
  cd ~/tmp/renewsliver/$slicename
  for rspec in ${rspecs[$slicename]} ; do outfile=$(echo $(basename $rspec) | sed -e 's/.rspec$//') ; somni $slicename $rspec ; omni --usercredfile=$HOME/.gcf/$USER-geni-usercred.xml --slicecredfile=$HOME/.gcf/$slicename-cred.xml -a $am sliverstatus $slicename >& $outfile ; done
  cd ~/tmp/renewsliver/$slicename
  grep -h _expires * >> results.txt
  for i in * ; do grep _expires $i > /dev/null || echo "no 'expires' lines in $i" ; done >> results.txt
done

Look for anomalies in the results files:

cd ~/tmp/renewsliver
for slicename in $slices ; do echo "==> $slicename" ; grep foam_expires $slicename/results.txt ; done
for slicename in $slices ; do echo "==> $slicename" ; grep orca_expires $slicename/results.txt ; done
for slicename in $slices ; do echo "==> $slicename" ; grep pg_expires $slicename/results.txt ; done
for slicename in $slices ; do echo "==> $slicename" ; grep "no 'expires' lines" $slicename/results.txt ; done

If you find anomalies, you'll probably need to go back to the original output files to figure out where they came from.

Check sliver status

for slicename in $slices ; do for rspec in ${rspecs[$slicename]} ; do somni $slicename $rspec ; omni --usercredfile=$HOME/.gcf/$USER-geni-usercred.xml --slicecredfile=$HOME/.gcf/$slicename-cred.xml -a $am sliverstatus $slicename |& grep _status || echo "no sliver" ; echo "is for $slicename @ $am" ; done ; done

Get VTS pseudowire IDs

for slicename in $slices ; do for rspec in ${rspecs[$slicename]} ; do somni $slicename $rspec ; omni --usercredfile=$HOME/.gcf/$USER-geni-usercred.xml --slicecredfile=$HOME/.gcf/$slicename-cred.xml -a $am listresources $slicename |& grep shared_lan || echo "no sliver" ; echo "is for $slicename @ $am" ; done ; done

Get login info

At this point, I'm storing these files in Subversion too, in .../ssh_config, so change into the relevant directory for the collection (jbsN, infra4, plastic-slices, etc) first, e.g.

cd ~/slices/plastic-slices/ssh_config

Then, get a file per slice with readyToLogin:

export PYTHONPATH=~/src/gcf-current/src
for slicename in $slices ; do ams="" ; for rspec in ${rspecs[$slicename]} ; do somni $slicename $rspec ; ams="$ams -a $am" ; done ; readyToLogin --no-keys --output --prefix=$slicename --usercredfile=$HOME/.gcf/$USER-geni-usercred.xml --slicecredfile=$HOME/.gcf/$slicename-cred.xml $ams $slicename ; done
for slicename in $slices ; do mv -f $slicename-sshconfig.txt $slicename ; rm -f $slicename*.xml $slicename*.json $slicename-logininfo.txt ; done

I should have symlinks in ~/.ssh/config.d pointing to these, so I then just run 'sshconf' to rebuild ~/.ssh/config.

Delete slivers

for slicename in $slices ; do for rspec in ${rspecs[$slicename]} ; do somni $slicename $rspec ; omni --usercredfile=$HOME/.gcf/$USER-geni-usercred.xml --slicecredfile=$HOME/.gcf/$slicename-cred.xml -a $am deletesliver $slicename & done ; sleep 5s ; done

Login stuff

Essential one-time steps for any login

Find old SSH keys, and print lines to remove them:

for login in $logins ; do ssh $login true |& grep ssh-keygen | sed -e 's/remove with://' ; done

(Repeat that until it returns with no output.

Make sure I can log in:

shmux -c "hostname" $logins

Copy in my dotfiles:

for slicename in $slices ; do loginfile=~/tmp/logins-$slicename.txt ; export PSSH_ERRDIR=~/tmp/prsync-errors/$slicename ; prsync -h $loginfile -a ~/.cfhome/ '' ; done

(FIXME: This next thing doesn't really belong here, but I haven't figured out where else to put it.)

Check for errors from a prsync command:

for slicename in $slices ; do export PSSH_ERRDIR=~/tmp/prsync-errors/$slicename ; for file in ~/tmp/prsync-errors/$slicename/* ; do test -s $file && echo "==> $file" && cat $file && echo "" ; done  ; done

Pingtest stuff

There are two ways to do this, one "fast" one that works ok for small numbers of hosts but not for lots of hosts at once, and one "reliable" one that works consistently even with large numbers of hosts.

Install files

The first of these is for both ways; the last two are only needed for "the reliable way".

for slicename in $slices ; do loginfile=~/tmp/logins-$slicename.txt ; export PSSH_ERRDIR=~/tmp/prsync-errors/$slicename ; prsync -h $loginfile -a ~/slices/*/reachability/addrs-$slicename.conf pingtest.conf ; done
shmux -c 'mkdir -p bin' $logins 
for slicename in $slices ; do loginfile=~/tmp/logins-$slicename.txt ; export PSSH_ERRDIR=~/tmp/prsync-errors/$slicename ; prsync -h $loginfile -a ~/subversion/syseng/geni/share/experiment-setup/plastic-slices/reachability/pingtest bin/pingtest ; done

The fast way

This way uses fping, which is very fast, but seems to be unreliable, more so when there are large numbers of hosts. (I think the parallelizing is overwhelming something -- something OpenFlow-related? A switch? My controller? With ARPs? Hard to tell without digging deeper than I have.)

It's still useful for a quick one-host-at-a-time check, but the comprehensive pingtest is pretty unlikely to work at any scale greater than a few hosts.

Run a fast pingtest on one host

fping -q -c 10 < pingtest.conf |& grep -v "ICMP Host Unreachable"

Run a comprehensive pingtest

cd
rm -rf ~/tmp/pingtest
mkdir -p ~/tmp/pingtest
cd ~/tmp/pingtest
for login in $logins ; do ssh -n $login 'fping -q -c 10 < pingtest.conf |& grep -v "ICMP Host Unreachable"' > $login && echo $login & done

Analyze the results

Show everything that isn't up (i.e. that doesn't have exactly 0% packet loss):

grep -v "/0%" *

Show only things that are 100% down:

grep "/100%" *

Show only things that have some packet loss other than 0% or 100%:

egrep -v "(/0%|/100%)" *

Show everything that isn't up, but exclude things that are known to be down (in this example, Washington and Stanford):

grep -v "/0%" * | egrep -v "(10.42.[0-9]+.(8[01]|9[01])|(washington|stanford))"

Show everything that is "up" (0% packet loss):

grep "/0%" *

The reliable way

This way uses a script that pings one host at a time, which takes longer, but seems to be very reliable.

Run a comprehensive pingtest

cd
rm -rf ~/tmp/pingtest
mkdir -p ~/tmp/pingtest
cd ~/tmp/pingtest
for login in $logins ; do ssh -n $login pingtest '$(cat pingtest.conf)' > $login && echo $login & done 

Analyze the results

Show everything that isn't "up" (0% packet loss):

grep -v "is up" * 

Show only things that are "down" (100% packet loss):

grep "is down" * 

Show only things that are "partial" (some packet packet loss, but not 100%):

grep "is partial" * 

Show everything that isn't "up', but exclude things that are known to be down (in this example, Washington and Stanford):

grep -v "is up" * | egrep -v "(10.42.[0-9]+.(8[01]|9[01])|(washington|stanford))" 

GEMINI stuff

Initialize:

for slicename in $slices ; do ./gdesktop-init.py --certificate=~/.ssl/jbs\@ch.geni.net.pem --project=JBS --slicename=$slicename ; done

Instrumentize:

for slicename in $slices ; do ./gdesktop-instrumentize.py --certificate=~/.ssl/jbs\@ch.geni.net.pem --project=JBS --slicename=$slicename ; done

Remote commands

Here are some examples of using shmux to run commands remotely on all logins.

Install fping and iperf

shmux -c 'sudo $PKGMGR -y install fping iperf' $logins

See what version of iperf is installed

shmux -c 'pkgversion iperf' $logins

Controller stuff

I generally do this stuff on naxos.

Run NOX 'switch' with a LAVI interface

This runs NOX with the 'switch' module on port 33xxx, and a LAVI interface on port 11XXX.

subnet=017
port=33$subnet ; (cd /usr/bin && /usr/bin/nox_core --info=/home/jbs/nox/nox-${port}.info -i ptcp:$port switch lavi_switches jsonmessenger=tcpport=11$subnet,sslport=0)

Get a list of DPIDs from NOX

This uses a script from Aaron Rosen (from http://paste.pocoo.org/show/555163/) to talk to LAVI and print a list of DPIDs (and pipes them to sort).

subnet=017 ; nox-console -n localhost -p 11$subnet getnodes | sort

Get a list of DPIDs from rspecs

If you have a directory full of rspecs, you can print a list of DPIDs that they include, in the same format as the output from nox-console. (Note that this is pretty naive, and doesn't try to do things like avoid commented-out things, etc.)

cd ~/rspecs/request/jbs15
grep -h component_id openflow-* | esed -e 's/.+datapath\+([^"]+).+$/\1/'|sort

Old stuff

Stuff I was doing at one point, but am not any more.

MyPLC stuff

Essential one-time steps for each MyPLC login

Set the hostname:

for login in $myplclogins ; do ssh $login sudo hostname $login ; done

Enable cron and start it up:

shmux -c 'sudo chkconfig crond on && sudo service crond status | grep running || sudo service crond start' $myplclogins

You should only have to do each of those once, but they're safe to repeat.

wapgnode stuff

Specify which VLAN to fiddle with

Do this before doing any of the other things in this section.

VLAN=<vlan-id>

Install and start using a new ifcfg-eth1.VLAN file

for host in $logins ; do rscp ~/tmp/emulab-wapg-vlan-testing/ifcfg-eth1-$host $host:ifcfg-eth1.$VLAN ; done
shmux -c "sudo cp ifcfg-eth1.$VLAN /etc/sysconfig/network-scripts" $logins
shmux -c "sudo ifup eth1.$VLAN" $logins
shmux -c "ifconfig eth1.$VLAN" $logins

Stop using and delete an old interface

shmux -c "sudo ifdown eth1.$VLAN" $logins
shmux -c "sudo rm /etc/sysconfig/network-scripts/ifcfg-eth1.$VLAN" $logins

Configure FlowVisor

slicename=jbs17
subnet=017

mkdir -p ~/$slicename/flowvisor
cd ~/$slicename/flowvisor
fvconfig generate flowvisor-$slicename.xml naxos $(cat /etc/flowvisor/fvpasswd) 33$subnet 8$subnet

cd
flowvisor ~/$slicename/flowvisor/flowvisor-$slicename.xml >> ~/$slicename/flowvisor/flowvisor-$slicename.log 2>&1 &
fvctl --passwd-file=/etc/flowvisor/fvpasswd --url=https://localhost:8$subnet setConfig 'flowvisor!log_ident' flowvisor$subnet
for slice in $(fvctl --url=https://localhost:8$subnet --passwd-file=/etc/flowvisor/fvpasswd listSlices |& grep Slice | grep -v root | awk '{ print $3; }') ; do fvctl --url=https://localhost:8$subnet --passwd-file=/etc/flowvisor/fvpasswd deleteSlice $slice ; done
fvctl --passwd-file=/etc/flowvisor/fvpasswd --url=https://localhost:8$subnet createSlice $slicename tcp:localhost:42$subnet jbs@bbn.com
fvctl --passwd-file=/etc/flowvisor/fvpasswd --url=https://localhost:8$subnet addFlowSpace any 100 dl_type=0x800,nw_dst=10.42.$subnet.0/24 "Slice:$slicename=4"
fvctl --passwd-file=/etc/flowvisor/fvpasswd --url=https://localhost:8$subnet addFlowSpace any 100 dl_type=0x800,nw_src=10.42.$subnet.0/24 "Slice:$slicename=4"

Run a FlowVisor on port 33xxx

flowvisor ~/$slicename/flowvisor/flowvisor-$slicename.xml >> ~/$slicename/flowvisor/flowvisor-$slicename.log 2>&1 &

Run NOX on port 42xxx

port=42017 ; (cd /usr/bin && /usr/bin/nox_core --info=/home/jbs/$slicename/nox/nox-${port}.info -i ptcp:$port switch)

Talk to a FlowVisor on a specific port

fvctl --passwd-file=/etc/flowvisor/fvpasswd --url=https://localhost:8$subnet listFlowSpace

Kill all the FlowVisors that you own

pkill -u $USER -f "org.flowvisor.FlowVisor"