LabWiki crash: with bad file descriptor message

[divyashri.bhat@gmail.com]

The production LabWiki (emmy9.casa.umass.edu:4000) is currently being used by students to run experiments. We noticed a few crashes with the same error as below:

DEBUG development::LabWiki::LWWidget: Calling 'on_stop_experiment on 'LabWiki::Plugin::Experiment::ExperimentWidget' widget


DEBUG development::LabWiki::Plugin::Experiment::ExperimentWidget: STOP EXPERIMENT as requested>>> {:action=>"stop_experiment", :col=>"execute", :sid=>"s6751972_4585340"}


DEBUG development::LabWiki::Plugin::Experiment::Experiment: SEND job stop request to http://emmy9.casa.umass.edu:8003/jobs/841cc39d-58ef-471d-aad4-7df138b71e60>>>


DEBUG development::LabWiki::Plugin::Experiment::Util::RetryHandler: canceled - #<Proc:0x00000003cea5d8@/var/lib/omfwebapps/lw_gec19/plugins/labwiki_experiment_plugin/lib/labwiki/plugin/experiment/log_adapter.rb:31>


DEBUG development::LabWiki::Plugin::Experiment::Util::RetryHandler: canceled - #<Proc:0x00000003cea1a0@/var/lib/omfwebapps/lw_gec19/plugins/labwiki_experiment_plugin/lib/labwiki/plugin/experiment/ec_adapter.rb:32>

terminate called after throwing an instance of 'std::runtime_error'

  what():  unable to add new descriptor: Bad file descriptor

Aborted (core dumped)

The maximum file descriptor size for emmy9 seems to be quite high

gimiadmin@emmy9:~$ cat /proc/sys/fs/file-max
1212548

Is there a maximum limit defined in gems used by LabWiki??

[johren@bbn.com]

Has not happened in the last few days. Scaling issue. Try to reproduce with automated tests. Could be sockets that could not be created.

[johren@bbn.com]

Have not been able to reproduce this issue with automated tests.