Thursday, January 27, 2011

Schedule a Periodic Task with a Watchdog

This demonstrates a simple mechanism to start up a thread that executes some task periodically, with a watchdog to ensure it keeps running. It is assumed that the periodic task involves some kind of I/O or remote communication that can result in an exception. The watchdog is needed because the java.util.concurrent class used for the task will not proceed with subsequent executions after one of its executions encounters an exception. If the task requires a restart in this event, additional steps are taken to restore its state to where it left off. The watchdog uses the same java.util.concurrent mechanism as does the task, but we don't worry about putting a watchdog on the watchdog because the steps taken there are trivial and not as likely to result in an error. The periodic task keeps a reference to the java.util.concurrent.Future returned when it executes, as a means of exposing some functionality to the watchdog, in particular so that the watchdog can detect when the task has "completed" (it never should unless it encounters an error), in which case it will cancel the task.

Here is the task:


public class PeriodicTask {

private int total, startTotal;
private ScheduledExecutorService executor;
private ScheduledFuture future;

public PeriodicTask() {
this(0);
}

/**
* Construct the task with the given total - useful to restore its state as needed.
*/
public PeriodicTask(int theTotal) {
total = theTotal;
startTotal = total;
start();
}

private void start() {
// start immediately, repeat every 300 ms
executor = Executors.newSingleThreadScheduledExecutor();
future = executor.scheduleAtFixedRate(new TheTask(), 0, 300, TimeUnit.MILLISECONDS);
}

/**
* Allow the watchdog to cancel the task
*/
public boolean cancel(boolean b) {
return future.cancel(b);
}

/**
* Allow the watchdog to determine if the task has completed (which it never should unless
* it has encountered an exception)
*/
public boolean isDone() {
return future.isDone();
}

/**
* Allow the watchdog to shutdown the executor
*/
public void shutdown() {
executor.shutdown();
}

private class TheTask implements Runnable {

public void run() {
// update object state on each execution
total++;
if (total > startTotal + 10) {
// generate an exception to simulate what might happen
throw new IllegalStateException();
}
}
}

public int getTotal() {
return total;
}
}

Next, the watchdog:


public class Watchdog {

private PeriodicTask task;

public Watchdog(PeriodicTask theTask) {
task = theTask;
Executors.newSingleThreadScheduledExecutor()
.scheduleAtFixedRate(new Watcher(), 0, 200, TimeUnit.MILLISECONDS);
}

public PeriodicTask getTask() {
return task;
}

private class Watcher implements Runnable {

public void run() {

while (true) {
if (task.isDone()) {
System.out.println("Task stopped - restarting...");
// cancel the task and shutdown the executor
task.cancel(true);
task.shutdown();
// restart from where we left off - state gets restored:
int lastKnownState = task.getTotal();
task = new PeriodicTask(lastKnownState);
}
}
}
}
}

Finally, a driver class to get the processes going:


public class TaskWatchdogTester {

public static void main(String[] args) throws Exception {

PeriodicTask task = new PeriodicTask();
Watchdog watchdog = new Watchdog(task);

while (true) {
Thread.sleep(500);
System.out.println("Total: " + watchdog.getTask().getTotal());
}
}
}

And, the output:


Total: 2
Total: 4
Total: 6
Total: 7
Total: 9
Task stopped - restarting...
Total: 12
Total: 13
Total: 15
Total: 17
Total: 18
Total: 20
Task stopped - restarting...
Total: 23
Total: 24
Total: 26
Total: 28
Total: 29
Total: 31
...and etc...

2 comments:

  1. Excellent, thank you!
    Just one note: you don't need the while(true) in Watcher.run(). If the task fails and cannot be restarted, it results in the Watcher frantically trying to restart it over and over and over again, non-stop, whereas the Watcher is also on its schedule anyway.

    ReplyDelete
  2. Another improvement. Here is my current version. It is very helpful to catch the exceptions here. Otherwise if something goes wrong when restarting the task, it is difficult to know what happens. (I am using Groovy, so even if there is no matching constructor, this is a runtime error.)

    private class Watcher implements Runnable {

    public void run() {
    //while (true) {
    if (task.isDone()) {
    System.out.println("Task stopped - restarting...");
    // cancel the task and shutdown the executor
    task.cancel(true);
    task.shutdown();
    try {
    // restart from where we left off - state gets restored:
    task = task.restartItself()
    } catch (Exception e) {
    System.err <<"ERROR. TASK RESTART FAILED " << e
    }
    }
    //}
    }
    }

    ReplyDelete