apr-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sander Striker <stri...@apache.org>
Subject Re: filtering huge request bodies (like 650MB files)
Date Thu, 11 Dec 2003 09:01:12 GMT
On Thu, 2003-12-11 at 09:54, Stas Bekman wrote:
> Sander Striker wrote:
> > On Thu, 2003-12-11 at 07:44, Stas Bekman wrote:
> > 
> > 
> >>I now know what the problem is. It is not a problem in httpd or its filters,

> >>but mod_perl, allocated filter struct from the pool. With many bucket brigades

> >>there were many filter invocations during the same request, resulting in 
> >>multiple memory allocation. So I have to move to the good-old malloc/free to

> >>solve this problem.
> >>
> >>Though it looks like I've found a problem in apr_pcalloc.
> >>
> >>modperl_filter_t *filter = apr_pcalloc(p, sizeof(*filter));
> >>
> >>was constantly allocating 40k of memory, whereas sizeof(*filter) == 16464
> >>
> >>replacing apr_pcalloc with apr_palloc reduced the memory allocations to 16k.
> >>
> >>Could it be a bug in APR_ALIGN_DEFAULT? apr_pcalloc calls APR_ALIGN_DEFAULT 
> >>and then it calls apr_palloc which calls APR_ALIGN_DEFAULT again, and probably

> >>doubling the memory usage.
> > 
> > 
> > Woah!  I'll look into this.
> 
> I think pcalloc shouldn't call the alignment function, but let palloc align 
> the size and return the updated size to pcalloc, so it could memset the right 
> size.

APR_ALIGN_DEFAULT only bumps the size to the next 8 byte boundary.  It
should definitely not induce the allocation of the amount of memory you
report.

> On the other hand, the "right" size is the one requested by the caller, so the 
> caller really expects to zero only the amount it has requested. So may be it's 
> OK if palloc allocates more memory (because of the alignment), but calloc 
> zeroing only the requested amount. In which case all we need to do is nuke the 
> call to APR_ALIGN_DEFAULT in pcalloc.

apr_pcalloc is even a macro when not in debug mode:

#define apr_pcalloc(p, size) memset(apr_palloc(p, size), 0, size)

With that in mind, we should loose the APR_ALIGN_DEFAULT call, but
nevertheless, it shouldn't make a difference.

Sander

Mime
View raw message