apr-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefan Fritsch ...@sfritsch.de>
Subject reducing memory fragmentation
Date Sun, 06 Feb 2011 22:34:44 GMT
On Friday 04 February 2011, William A. Rowe Jr. wrote:
> >> My thought is that this might not be the only thing to 'refresh'
> >> when you have an app which needs current state.  It would be a
> >> chance to flush out all caches and set the state basically to
> >> apr_initialize(), modulo any open resources or currently
> >> allocated pools.  It could even go so far as to free() all the
> >> unused pool memory, giving us a really fresh start on heap
> >> pages that have been fragmented to heck.
> > 
> > ++1 for some way of handling the fragmentation... of course,
> > we could just switch to pocore for the memory stuff in APR.  :)
> Which helps for apr_allocator but not for the internal
> fragmentation within individual pools.  An optional pocore feature
> for APR 2 sounds great, patches welcome :)

The pools give their memory back to the allocator when they are 
cleared. Therefore at least the transaction pools and their children 
should not be a major problem for fragmentation.

However, one major problem are the allocators not giving back memory, 
or if configured with max_mem_free, giving it back in random order so 
that it is very unlikely that the malloc implementation can give 
memory back to the OS.

A second problem I have noticed with glibc: Its malloc allocates APR's 
typical multiple-of-4k chunks in a normal linked list. This means that 
there is some N bytes header allocated before each chunk. Now if APR 
has allocated say 16k, and frees it again, the hole is 16k+N. However 
this is not big enough to allocate two 8k chunks, which would need 
(8k+N)*2 == 16k+2N bytes. I think this may contribute to the 
fragmentation seen under Linux. In addition to that, glibc's malloc 
allocates smaller chunks in the same linked list, so in practice we 
have something like this:

8K pool chunk
8K pool chunk
some bytes from a third party lib
12K pool chunk
some more bytes from normal malloc
8K pool chunk

This doesn't help with fragmentation either.

Therefore I propose to use mmap/mumap in the apr_allocator instead of 
malloc, either as a compile time option or whenever max_mem_free is 
set for the allocator. This would then completely separate the 
allocations from apr_allocator and normal malloc and reduce 
fragmentation. And it would make apr_allocator_max_free_set actually 
give memory back to the OS.

This would be a relatively simple change (compared to replacing the 
whole allocator). Sounds ok?

View raw message