httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nick Gearls <nickgea...@gmail.com>
Subject Re: mod_substitute memory problem
Date Wed, 10 Jun 2009 16:16:16 GMT
The problem is real.
I have an application which generates one line of about 300 KB :-(
In this case, Apache eats up all the memory (about 1 GB), and never 
gives it back to the OS - game over - reboot !

How could this be solved?

Thanks,

Nick


Dan Poirier wrote:
> Nick Gearls <nickgearls@gmail.com> writes:
> 
>> There seems to be a memory problem when substituing something on very
>> long lines (several hundreds KB). this problem is different from bug
>> 44948 (I applied this patch).
>>
>> When using something like "Substitute s/string1/string2/" on a 300 KB
>> line with 30 times "string1" on the line, there are 2 problems:
>>
>> 1. It uses a huge amount of memory, as - I think - it reallocates a
>> new buffer (300 KB) before each substitute without freeing the
>> previous one
> 
> I think you're right - most of the working memory is not released until
> the request is over.  This is kind of a worst-case for mod_substitute's
> temporary memory usage.
> 
> Is this a problem in real cases?  Or just in artificial testing?
> If these are real requests, maybe if you tell a little more about them,
> we can find a better solution.
> 
>> 2. The memory is not freeed at the end of the HTTP request.
>> Maybe is it due to Keep-alive?
> 
> If by "the memory is not freed", you mean the memory usage for the
> http process as reported by the OS doesn't go down, that's right.
> But at the end of the request, the memory should be released back
> to Apache to re-use.  You can test this by repeating your test several
> times - after the first time, the process's memory usage should not keep
> going up the way it does the first time.
> 
> If that's a problem in your case, you can set MaxMemFree to ask Apache
> to release memory by calling free() above a certain threshold.  Even
> then, whether the process releases the memory back to the OS depends on
> the OS, so it might not gain you anything.
> 
> But if this is going to happen very often, you might as well let Apache
> hold on to the memory; it's just going to grab it back again the next
> time this happens.
> 
>> If using the "q" switch to not flatten the buckets, it uses almost no
>> memory. Btw, it correctly handles more 60 substitutions on the same
>> line (some shorter, some longer) without the q flag !?! When exactly
>> is this flag needed?
> 
> Don't know about this one.
> 
>> In macro SEDSCAT, we have
>> 	s2 = apr_pstrmemdup(pool, buff, blen);
>>         s1 = apr_pstrcat(pool, s1, s2, NULL);
>> Shouldn't we free the previous s1 buffer?
> 
> Pools don't provide a way of freeing individual allocations; you have
> to destroy the whole pool.
> 
>> Anyway, I do not understand why the memory is not released, as the
>> pool is supposed to be destroyed.
> 
> See above.
> 

Mime
View raw message