I use apache to serve a few sites from the firewall box in the basement and for some reason it kept dying on a regular basis. This started fairly recently, some time after I set up separate access log files for each of the sites.
The error logs showed entries like
[Sun Jun 10 06:28:03 2007] [warn] child process 16516 still did not exit, sending a SIGTERM
which seemed bizarre and weren’t being spawned by anything obvious in the access logs. I eventually remembered that I had set up log rotation for each of the virtual hosts, and went searching through the error logs to see if it was related. Sure enough, the shut-down was happening at the same time, and so the log rotation was likely to be related to the cause. But what could be the real cause? Surely not just rotating logs…
A bit of poking around the web found that other people have had this problem. I went down a bunch of dead ends (virtual hosts? apache2ctl restart vs apache2ctl start?) and eventually just turned error reporting up to the maximum. The next week, after the server went dead again, I found the following email in my sysadmin inbox.
/etc/cron.daily/logrotate: (98)Address already in use: make_sock: could not bind to address 0.0.0.0:80 no listening sockets available, shutting down Unable to open logs error: error running shared postrotate script for /var/log/apache2/*.log run-parts: /etc/cron.daily/logrotate exited with return code 1
Hunting around more on the web, I found a recommendation to explicitly set the TMPDIR environment variable before running logrotate. Apache has now been up for a few weeks without falling over, problem solved! I’m still not sure why this only started to happen after setting up separate log files for the separate virtual hosts, or even if that was more than a proximate coincidence.
