Upstart: report (email, log) on respawn and failure

Question

I would like to get an email when an upstart job fails and respawns, and when all respawns are depleted, and the job fails.

How can I do that?

Answer 1

Upstart automatically logs when jobs are respawning and when it gives up on them.

[266017.061509] init: foo main process (11582) terminated with status 1
[266017.061684] init: foo respawning too fast, stopped

So just watch the logs for these messages and send an email alert when they are seen.

Answer 2

You may use pre-start stanza to send email but I think it's better to use post-stop which has the needed variables from last run.

If the job was stopped normally, the RESULT environment variable will be ok, otherwise if the job was stopped because it has failed it will be failed.

When the job has failed, the process that failed will be given in the PROCESS environment variable. This may be pre-start, post-start, main, pre-stop or post-stop; it may also be the special value respawn to indicate that the job was stopped because it hit the respawn limit.

Source: man stopped

References:

• Upstart Intro, Cookbook and Best Practises: respawn