httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan Poirier <poir...@pobox.com>
Subject Re: mod_substitute memory problem
Date Wed, 10 Jun 2009 12:17:41 GMT
Nick Gearls <nickgearls@gmail.com> writes:

> There seems to be a memory problem when substituing something on very
> long lines (several hundreds KB). this problem is different from bug
> 44948 (I applied this patch).
>
> When using something like "Substitute s/string1/string2/" on a 300 KB
> line with 30 times "string1" on the line, there are 2 problems:
>
> 1. It uses a huge amount of memory, as - I think - it reallocates a
> new buffer (300 KB) before each substitute without freeing the
> previous one

I think you're right - most of the working memory is not released until
the request is over.  This is kind of a worst-case for mod_substitute's
temporary memory usage.

Is this a problem in real cases?  Or just in artificial testing?
If these are real requests, maybe if you tell a little more about them,
we can find a better solution.

> 2. The memory is not freeed at the end of the HTTP request.
> Maybe is it due to Keep-alive?

If by "the memory is not freed", you mean the memory usage for the
http process as reported by the OS doesn't go down, that's right.
But at the end of the request, the memory should be released back
to Apache to re-use.  You can test this by repeating your test several
times - after the first time, the process's memory usage should not keep
going up the way it does the first time.

If that's a problem in your case, you can set MaxMemFree to ask Apache
to release memory by calling free() above a certain threshold.  Even
then, whether the process releases the memory back to the OS depends on
the OS, so it might not gain you anything.

But if this is going to happen very often, you might as well let Apache
hold on to the memory; it's just going to grab it back again the next
time this happens.

> If using the "q" switch to not flatten the buckets, it uses almost no
> memory. Btw, it correctly handles more 60 substitutions on the same
> line (some shorter, some longer) without the q flag !?! When exactly
> is this flag needed?

Don't know about this one.

> In macro SEDSCAT, we have
> 	s2 = apr_pstrmemdup(pool, buff, blen);
>         s1 = apr_pstrcat(pool, s1, s2, NULL);
> Shouldn't we free the previous s1 buffer?

Pools don't provide a way of freeing individual allocations; you have
to destroy the whole pool.

> Anyway, I do not understand why the memory is not released, as the
> pool is supposed to be destroyed.

See above.

-- 
Dan Poirier <poirier@pobox.com>


Mime
View raw message