Mailing-List: contact dev-help@httpd.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@httpd.apache.org
Received-SPF: pass (asf.osuosl.org: local policy)
Message-ID: <4369439C.2030400@wstoddard.com>
Date: Wed, 02 Nov 2005 17:54:20 -0500
From: Bill Stoddard <bill@wstoddard.com>
Reply-To: bill@wstoddard.com
User-Agent: Mozilla Thunderbird 1.0.5 (Windows/20050711)
MIME-Version: 1.0
To: dev@httpd.apache.org
Subject: Re: Potential memory leak in /modules/cache/mod_mem_cache.c
References: <dkb9eh$ue$1@sea.gmane.org> <43693392.6050002@apache.org>
 <4369358A.9050306@wstoddard.com> <436940D0.5040708@apache.org>
In-Reply-To: <436940D0.5040708@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8bit

Ruediger Pluem wrote:
> 
> On 11/02/2005 10:54 PM, Bill Stoddard wrote:
> 
>>Ruediger Pluem wrote:
> 
> 
> [..cut..]
> 
> 
>>>Finding calls to malloc and friends in the code of mod_mem_cache is a
>>>clear documentation
>>>that there is no development care of this code as they should not be
>>>used inside of
>>>httpd code. Throughout the httpd code memory pools will be used to
>>>aquire memory for storing
>>>data structures.
>>
>>
>>mod_mem_cache does not use pools to store objects because it is not
>>straightforward how to recover pool storage when stale cache objects are
>>removed from the cache. For example, if you have 10,000 objects in the
>>mem cache and you need to garbage collect 1837 of them, how do you do it
>>(and return the memory to the system) if all 10,000 objects are
>>allocated out of the same pool? There are ways of doing it, but not
>>straigtforward.
> 
> 
> Ok. You got me :-). I should have taken a closer look at the code before.
> I agree that this problem is not straightforward to solve, especially not
> for someone who is not that deep into pools like me and who still has a lot
> to learn in this area.
> 
> In order to improve my understanding of mod_mem_cache at which I had
> a (more) closer look than before I would like to ask some questions:
> 
> For whatever reason I firmly believed that mod_mem_cache has a common cache
> via shared memory over all httpd processes. This does not seem to be true, correct?

Correct. mod_mem_cache stores objects in the heap (not in shared memory). It can store file contents in heap 
or it can store open file descriptors.

> 
> But if each httpd process maintains its own local cache, this would mean
> that the same object needs to be cached multiple times if clients request
> the same object from different httpd processes. This would
> 
> - increase the load on the backend
Yes

> - increase the thundering herd problem for stale cache entries
Yes

> - increase memory usage per object as each httpd process does have a local
>   copy of it in memory.
Yes

> 
> Do I see this correct?

Yep, you see it correctly. Mod_mem_cache is most interesting to me personally as a benchmark toy. It should be 
possible to extend mod_mem_cache to cache headers (rather than rebuilding them for each response sent out of 
the cache), align storage on processor cache-lines and other performance tweaks to serve static pages really 
really fast (or cache dynamically generated pages until they become stale). You should be able to use 
mod_mem_cache to explore the static page performance limits of Apache HTTPD 2.0.

> 
> 
> Regards
> 
> R�diger

Bill
> 
> 
>