Sometimes services crash badly, in this case I was trying to stabilize an old server running nginx and PHP via Fast-cgi. It crashes quite often, so I set up monit on it – but I noticed monit has stopped doing it’s job due to the following batch of errors:
[06:02:24] error : monit: Error reading pid from file '/var/run/fastcgi-php.pid'
[06:02:25] error : monit: Error reading pid from file '/var/run/fastcgi-php.pid'
[06:02:26] error : monit: Error reading pid from file '/var/run/fastcgi-php.pid'
[06:02:27] error : monit: Error reading pid from file '/var/run/fastcgi-php.pid'
[06:02:28] error : monit: Error reading pid from file '/var/run/fastcgi-php.pid'
[06:02:29] error : monit: Error reading pid from file '/var/run/fastcgi-php.pid'
Now the problem was, the PID file was there, but it was 0 bytes for some reason – so monit couldn’t read from it. I have a fairly regular monit setup for the normal PHP monitoring, something like:
check process php-cgi with pidfile /var/run/fastcgi-php.pid
start program = "/etc/init.d/php5-fcgi start"
stop program = "/etc/init.d/php5-fcgi stop"
if totalcpu > 60% for 2 cycles then alert
if totalcpu > 80% for 5 cycles then restart
if totalmem > 1200.0 MB for 5 cycles then alert
if failed unixsocket /var/run/fcgi.sock then restart
if 3 restarts within 5 cycles then timeout
if failed host localhost port 80 protocol http
and request "/healtcheckfile.php"
then restart
And this worked fine in most cases, but when something happened that caused a 0 byte PID file, monit would totally ignore the process with the following error:
[06:02:50] error : 'php-cgi' failed to start
[06:04:50] error : 'php-cgi' service timed out and will not be checked anymore
The solution to this problem, is to delete the PID file if it’s 0 bytes and restart the service so it creates a new, valid PID file. I achieved this with the following:
check file fastcgi-php.pid with path /var/run/fastcgi-php.pid
if size = 0 then
exec "/bin/rm /var/run/fastcgi-php.pid"
start program = "/etc/init.d/php5-fcgi start"
stop program = "/etc/init.d/php5-fcgi stop"
If you encounter the 0 byte PID after a crash and this works, you should see the following:
[11:21:39] error : 'fastcgi-php.pid' size test failed for /var/run/fastcgi-php.pid -- current size is 0 B
[11:21:39] info : 'fastcgi-php.pid' exec: /bin/rm
[11:23:39] error : 'fastcgi-php.pid' file doesn't exist
[11:23:39] info : 'fastcgi-php.pid' trying to restart
[11:23:39] info : 'fastcgi-php.pid' stop: /etc/init.d/php5-fcgi
[11:23:39] info : 'fastcgi-php.pid' start: /etc/init.d/php5-fcgi
[11:25:39] info : 'fastcgi-php.pid' file exist
[11:25:39] info : 'fastcgi-php.pid' size succeeded
Good luck!
