httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rainer Jung <rainer.j...@kippdata.de>
Subject Re: Problem with file descriptor handling in httpd 2.3.1
Date Sun, 04 Jan 2009 16:57:33 GMT
On 04.01.2009 15:04, Ruediger Pluem wrote:
>
> On 01/04/2009 12:49 AM, Rainer Jung wrote:
>> On 04.01.2009 00:36, Paul Querna wrote:
>>> Rainer Jung wrote:
>>>> During testing 2.3.1 I noticed a lot of errors of type EMFILE: "Too
>>>> many open files". I used strace and the problem looks like this:
>>>>
>>>> - The test case is using ab with HTTP keep alive, concurrency 20 and a
>>>> small file, so doing about 2000 requests per second.
>>>> MaxKeepAliveRequests=100 (Default)
>>>>
>>>> - the file leading to EMFILE is the static content file, which can be
>>>> observed to be open more than 1000 times in parallel although ab
>>>> concurrency is only 20
>>>>
>>>> - From looking at the code it seems the file is closed during a
>>>> cleanup function associated to the request pool, which is triggered by
>>>> an EOR bucket
>>>>
>>>> Now what happens under KeepAlive is that the content files are kept
>>>> open longer than the handling of the request, more precisely until the
>>>> closing of the connection. So when MaxKeepAliveRequests*Concurrency>
>>>> MaxNumberOfFDs we run out of file descriptors.
>>>>
>>>> I observed the behaviour with 2.3.1 on Linux (SLES10 64Bit) with
>>>> Event, Worker and Prefork. I didn't yet have the time to retest with
>>>> 2.2.
>>> It should only happen in 2.3.x/trunk because the EOR bucket is a new
>>> feature to let MPMs do async writes once the handler has finished
>>> running.
>>>
>>> And yes, this sounds like a nasty bug.
>> I verified I can't reproduce with the same platform and 2.2.11.
>>
>> Not sure I understand the EOR asynchronicity good enough to analyze the
>> root cause.
>
> Can you try the following patch please?

Here's the gdb story:

When the content file gets opened, its cleanup is correctly registered 
with the request pool. Later in core_filters.c at the end of function 
ap_core_output_filter() line 528 we call setaside_remaining_output().

This goes down the stack via ap_save_brigade(), file_bucket_setaside() 
to apr_file_setaside(). This kills the cleanup for the request pool and 
adds it instead to the transaction (=connection) pool. There we are.

2.2.x has a different structure, although I can also see two calls to 
ap_save_brigade() in ap_core_output_filter(), but they use different 
pools as new targets, namely a deferred_write_pool resp. input_pool.

So now we know, how it happens, but I don't have an immediate idea how 
to solve it.

Regards,

Rainer

Mime
View raw message