httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Orton <jor...@redhat.com>
Subject Re: Cleanup/desctruction of connection pool and associated bucket_alloc
Date Fri, 05 Oct 2007 09:45:58 GMT
On Fri, Oct 05, 2007 at 03:37:57PM +1000, Bojan Smojver wrote:
> Now imagine someone (like yours truly :-) writing a handler/filter that
> sends many, many buckets inside a brigade down the filter chain. This
> causes the httpd process to start consuming many, many megabytes (in
> some instances I measured almost 500 MB in my tests), which are never
> returned. Then imagine multiple httpd processes doing the same thing and
> not releasing any of that memory back. The machine goes into DOS
> quickly, due to excessive swapping.

It sounds like that is the root cause.  If you create a brigade with N 
buckets in for arbitrary values of N, expect maximum memory consumption 
to be O(N).  The output filtering guide touches on this:

http://httpd.apache.org/docs/trunk/developer/output-filters.html

Filters need to be written to pass processed buckets down the filter 
chain ASAP rather than buffering them up into big brigades.  Likewise 
for a content generator - buffering up a hundred 1MB HEAP buckets in a 
brigade will obviously give you a maximum heap usage of ~100MB; instead, 
pass each HEAP bucket down the filter chain as its generated and you get 
maximum of ~1MB.

Most of the shipped filters do behave correctly in this respect (though 
there are problems with the handling of FLUSH buckets); see e.g. the 
AP_MIN_BYTES_TO_WRITE handling in mod_include.

joe

Mime
View raw message