Apache Hangs with Apache Error Logs saying 'Timeout'

Question

I'm running OpenBSD's chrooted httpd process and for the last few days I'm getting random hangups that clear themselves without any intervention on my part. At first I thought it might be a resource problem as SSH was hanging at the same time, but have left 'top' running on the console which shows nothing out of the ordinary (memory usage is low, no high load, etc).

I managed to get my users to call me exactly when this happens instead of 20-30 minutes later, and noticed in my error logs the following:

Timeout
^@Timeout
^@Timeout
^@Timeout

That's all that is on the line. Sometimes the number of ^@Timeout's vary, but that is all that is sitting there. I've looked at the access logs to see if some of the potentially hanging processes like curl calls might be doing it, but I didn't see any correlation.

The site that is running in a custom PHP app that has, up until lately, not had any problems. No new code changes were pushed up at the same time this started happening.

Anyone else come across a situation like this, or have any ideas?

[Edit - 06-11-2009]

Looks like something else is going on outside of this server. The Timeout prints seem to happen randomly, but I think I'm seeing this same thing happen on a network level were connections are hanging.

Thanks for the input though!

Answer 1

Once we had similar problem with SVN. It turned out that system entropy pool was occasionally becoming completely drained and then SVN and couple other things (can't remember what) hung. This might be a case in regard to SSH and if the PHP is utilizing entropy this might be a case with Apache as well.

You can check entropy leven on your OpenBSD box with following command:

sysctl kern.random

As an output you'll get quite a lot on numbers. These are described in /usr/include/dev/rndvar.h:

struct rndstats {
        quad_t rnd_total;       /* total bits of entropy generated */
        quad_t rnd_used;        /* strong data bits read so far */
        quad_t rnd_reads;       /* strong read calls */
        quad_t arc4_reads;      /* aRC4 data bytes read so far */
        quad_t arc4_nstirs;     /* arc4 pool stirs */
        quad_t arc4_stirs;      /* arc4 pool stirs (bits used) */

        quad_t rnd_pad[5];

        quad_t rnd_waits;       /* sleeps for data */
        quad_t rnd_enqs;        /* enqueue calls */
        quad_t rnd_deqs;        /* dequeue calls */
        quad_t rnd_drops;       /* queue-full drops */
        quad_t rnd_drople;      /* queue low watermark low entropy drops */

        quad_t rnd_ed[32];      /* entropy feed distribution */
        quad_t rnd_sc[RND_SRC_NUM]; /* add* calls */
        quad_t rnd_sb[RND_SRC_NUM]; /* add* bits */
};

If it turns out it really is an entropy issue you may consider using non-strong random generator, like /dev/urandom, for your PHP app.