wiki:Dingbot

Version 5 (modified by Josh Smift, 10 years ago) (diff)

--

Dingbot

Dingbot is a post-boot script for bridging the gaps between different GENI aggregate managers and/or different GENI node images. We expect those gaps to narrow over time, but meanwhile, it's an example of how to get a more uniform experience out of some not-entirely-uniform resources. At GEC 17, it was mentioned in the Uniform Experimenter Experience session, and described in more detail at the Experimenter Roundtable session; see those pages for slides.

It's named after Agatha Heterodyne's clank minion Dingbot Prime. (That page contains spoilers for Girl Genius, if you care about that sort of thing.)

Setup

Dingbot's purpose is to run on a newly-provisioned GENI compute node, and do setup tasks that aren't otherwise done for you automatically. To use it, you'd put something like

<services>
  <install url="CONFIG-URL" install_path="/opt" />
  <install url="DINGBOT-URL" install_path="/opt" />
  <execute shell="/bin/bash" command="sudo /opt/dingbot/dingbot CONFIG-FILE HOSTNAME"/>
</services>

into the <node> section(s) of your rspec, where:

  • DINGBOT-URL is the URL of the a tarball containing Dingbot
  • CONFIG-URL is the URL of a tarball containing a config file to download and use
  • CONFIG-FILE is the path to the config file (so this needs to match the filename in the tarball, and the location where you're unpacking it)
  • HOSTNAME is the hostname that you want this system to use

It'd be better if Dingbot could deduce the hostname from something that's already in the rspec, like the client_id attribute, but right now the node doesn't have reliable access to that at boot time. (You can use $self.Name() on ExoGENI, via their Velocity template engine, to avoid having to hardcode in a hostname in your rspec.)

Dingbot needs to run as root; you can (and may have to) leave out the sudo on AMs where that's already the case (e.g. ExoGENI).

Operation

The simplest way to see what it does is to look at the code, which is commented in astonishingly verbose detail.

...but which isn't yet available here; we plan to publish it soon, stay tuned.

Configuration

Dingbot uses a JSON configuration file to specify things that different experimenters might want to do differently, such as set up certain user accounts, or install certain software packages. You can also use it to do different things on different nodes in your experiment: If you always want to do the same things on all your nodes, you can use the same config file in all of your rspecs; or if you want different things on different nodes, you can have as many different config files as you want, just put the appropriate URL into the appropriate <node> section in each rspec. There's also a general pre- and post-execution hook where you can put an arbitrary command (which could run a script full of arbitrary commands).

Because the <install url=...> element expects a .tar.gz file, the config file also needs to be in a tarball.

A sample config file:

{
 "prehook": "touch -r /etc/issue /.dingbot-pretest",
 "posthook": "touch -r /etc/issue /.dingbot-posttest",
 "packages": [ "fping", "iperf", "rsync", "sudo" ],
 "users":
 {
  "jbs":
  {
   "username": "jbs",
   "fullname": "Josh Smift",
   "shell": "/bin/bash",
   "authorized_keys": "ssh-rsa [* Josh's key]"
  },
  "nriga":
  {
   "username": "nriga",
   "fullname": "Niky Riga",
   "shell": "/bin/bash",
   "authorized_keys": "ssh-rsa [* Niky's key]"
  },
  "mbrinn":
  {
   "username": "mbrinn",
   "fullname": "Marshall Brinn",
   "shell": "/bin/bash",
   "authorized_keys": "ssh-rsa [* Marshall's key]"
  }
 }
}

The names of the packages in the list are handed to YUM/APT/etc for installation. It doesn't gracefully handle packages with different names on different OSes/distributions, yet.

Dingbot creates accounts for the listed users if they don't exist already. The only change it makes to existing accounts is to set the user's shell as specified in the config file; it doesn't change anything else about existing accounts, even if the Dingbot config differs from the current state on the system.

Debugging

Dingbot redirects output and errors to log files in /var/log, named dingbot-output-<timestamp>.log and dingbot-errors-<timestamp>.log, and those are good places to look if something didn't work as expected. If you run it by hand on the command line, you shouldn't get any output whether it succeeds or fails, because it tries hard to direct all output and errors into those files, so that no information is lost when it runs non-interactively at boot time (and it's not yet smart enough to detect whether it's being run in a terminal or not).

Future work

Dingbot makes various simplifying assumptions about its environment; see the code for more details. (The word "FIXME" is a good indicator.)

One significant limitation of Dingbot is that if it encounters any transient failures (e.g. it can't install software packages because the node temporarily can't reach the package repo, for whatever reason), it doesn't automatically retry or anything. Using a real system configuration management system (like Cfengine, Puppet, Chef, Salt, etc) could make this more robust.

Other more comprehensive experimenter tools include post-boot scripts as well, and do everything Dingbot does and a whole lot more. It was originally written to make life easier for one casual amateur experimenter, and isn't really intended to be a comprehensive solution for a wide range of people. But if it helps you use GENI, or serves as a starting point for your own ideas, or just amuses and entertains you, all the better.