Custom Query (87 matches)

Discussed at the 3/31/14 meeting. When something goes wrong in the job service, it is sometimes hard to tell what is happening without cleaning up some of the jobs. This should be a little easier to do.

While using the job service running at ​http://emmy9.casa.umass.edu:8003, I ran into the following problem:

1. When there are many "Running" processes, the job service service stalls and puts the jobs in "Pending" status. To try to identify the source of this problem, I looked at the logs of the "Running" Processes.

The EC tries to connect to an RC which is either not up or does not exist and stays in that state while still showing the job status as "Running".

STDOUT: 11:26:21 INFO  OmfEc::Experiment: Experiment: dbhat-2014-04-11T10-18-13-05-00 starts
STDOUT: 11:26:21 INFO  OmfEc::Experiment: Configure 'nodea-labwikicrashtest' to join 'Source1'
STDOUT: 11:26:21 INFO  OmfEc::Experiment: Configure 'nodeb-labwikicrashtest' to join 'Source2'
STDOUT: 11:26:21 INFO  OmfEc::Experiment: Configure 'nodec-labwikicrashtest' to join 'Source3'

To resolve this problem, I tried:

1. delete all jobs with status as "Running" but they were only waiting for an RC to connect.
   
2. restart the job service on emmy9.
   

After this the experiments were ran successfully.

I am not sure if all of these resources are listed in the AMQP database.

But, suppose these resources are listed in the AMQP database and are deleted by the experimenter or Aggregate Manager, and at a later time, the experimenter tries to connect to these resources that do not actually exist:

1. How long will the EC wait for these RCs to connect?
   
2. With several such jobs, will job service continue to block and thus, prevent other experiments from running?

This is a ticket to shadow the "FiberError?" Labwiki crashes tracked in ​http://mytestbed.net/issues/1626.