httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bojan Smojver <bo...@rexursive.com>
Subject Re: Cleanup/desctruction of connection pool and associated bucket_alloc
Date Fri, 05 Oct 2007 11:08:05 GMT
Quoting Joe Orton <jorton@redhat.com>:

> It sounds like that is the root cause.  If you create a brigade with N
> buckets in for arbitrary values of N, expect maximum memory consumption
> to be O(N).  The output filtering guide touches on this:
>
> http://httpd.apache.org/docs/trunk/developer/output-filters.html
>
> Filters need to be written to pass processed buckets down the filter
> chain ASAP rather than buffering them up into big brigades.  Likewise
> for a content generator - buffering up a hundred 1MB HEAP buckets in a
> brigade will obviously give you a maximum heap usage of ~100MB; instead,
> pass each HEAP bucket down the filter chain as its generated and you get
> maximum of ~1MB.
>
> Most of the shipped filters do behave correctly in this respect (though
> there are problems with the handling of FLUSH buckets); see e.g. the
> AP_MIN_BYTES_TO_WRITE handling in mod_include.

All true. The problem in both cases was created by the number of  
buckets, rather than by the size of them. At first I was suspecting I  
was doing something stupid in my code, so I checked out mod_include to  
see what happens there and to my surprise, it was eating memory and  
not returning it to OS too.

What I was worried about here is that after the connection is closed,  
the memory used for all this is never reclaimed, which is what  
actually caused DOS (i.e. multiple huge processes started being  
swapped out). Not sure how far mod_include would push this (I can test  
with say 10,000+ includes and see what happens), but maybe it would be  
better to destroy the pool, just like the request pool gets destroyed,  
to be on the safe side. But then again, if a module is buggy, it's not  
Apache's problem...

Not sure why MaxMemFree never kicked in to saved the day. Gdb says  
that the value was passed in correctly and all and yet, with no  
effect...

-- 
Bojan

Mime
View raw message