httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Plüm, Rüdiger, VF-Group <>
Subject Memory consumption of mod_substitute
Date Wed, 05 Dec 2007 16:36:34 GMT
I noticed that mod_substitute can consume an insane amount of memory
in specific situations.

I used the following testcase:

One file with 10,000,000 lines of

Replace me

(about 105MB, yes this is an edge case :-)))

and the following configuration:

Substitute "s/Replace me/blah/"
SetOutputfilter SUBSTITUTE

Requesting this file resulted in a temporary memory consumption of nearly 1 GB
(rss not virtual size).

After applying my patch the temporary memory consumption decreased to about
20 MB. But as the patch changes quite a lot some remote eyes are due :-).

Basicly the patch does the following:

* Precreate all needed brigades, save them in the filter context and reuse them in
  order to avoid frequent recreations using the request pool.

* Use a temporary pool for all the needed copy stuff and clean it up every
  time we passed the passbb brigade down the chain. IMHO we can pass the brigade
  down the chain directly after we processed one bucket from the original brigade
  as buffering is done by the network filters.

* Use transient instead of pool buckets.

* My test case lead to the exceptional situation of a very large passbb bucket brigade
  (about 1,000,000 buckets) as a result of processing 4 MB of the file. So I add
  a flush bucket once I have more than MAX_BUCKET (1000) buckets in the brigade and pass it
  down the chain to get it send and the passbb bucket brigade cleaned up and its memory
  reusable again.

Comments, thoughts?



View raw message