apr-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jim Jagielski <...@jaguNET.com>
Subject Re: reducing memory fragmentation
Date Wed, 16 Feb 2011 23:01:53 GMT
Please also look at Greg's pocore memory allocation stuff...
his hash stuff is also quite nice. Would be useful to be
able to use that as well....

On Feb 6, 2011, at 5:34 PM, Stefan Fritsch wrote:

> On Friday 04 February 2011, William A. Rowe Jr. wrote:
>>>> My thought is that this might not be the only thing to 'refresh'
>>>> when you have an app which needs current state.  It would be a
>>>> chance to flush out all caches and set the state basically to
>>>> apr_initialize(), modulo any open resources or currently
>>>> allocated pools.  It could even go so far as to free() all the
>>>> unused pool memory, giving us a really fresh start on heap
>>>> pages that have been fragmented to heck.
>>> ++1 for some way of handling the fragmentation... of course,
>>> we could just switch to pocore for the memory stuff in APR.  :)
>> Which helps for apr_allocator but not for the internal
>> fragmentation within individual pools.  An optional pocore feature
>> for APR 2 sounds great, patches welcome :)
> The pools give their memory back to the allocator when they are 
> cleared. Therefore at least the transaction pools and their children 
> should not be a major problem for fragmentation.
> However, one major problem are the allocators not giving back memory, 
> or if configured with max_mem_free, giving it back in random order so 
> that it is very unlikely that the malloc implementation can give 
> memory back to the OS.
> A second problem I have noticed with glibc: Its malloc allocates APR's 
> typical multiple-of-4k chunks in a normal linked list. This means that 
> there is some N bytes header allocated before each chunk. Now if APR 
> has allocated say 16k, and frees it again, the hole is 16k+N. However 
> this is not big enough to allocate two 8k chunks, which would need 
> (8k+N)*2 == 16k+2N bytes. I think this may contribute to the 
> fragmentation seen under Linux. In addition to that, glibc's malloc 
> allocates smaller chunks in the same linked list, so in practice we 
> have something like this:
> 8K pool chunk
> 8K pool chunk
> some bytes from a third party lib
> 12K pool chunk
> some more bytes from normal malloc
> 8K pool chunk
> This doesn't help with fragmentation either.
> Therefore I propose to use mmap/mumap in the apr_allocator instead of 
> malloc, either as a compile time option or whenever max_mem_free is 
> set for the allocator. This would then completely separate the 
> allocations from apr_allocator and normal malloc and reduce 
> fragmentation. And it would make apr_allocator_max_free_set actually 
> give memory back to the OS.
> This would be a relatively simple change (compared to replacing the 
> whole allocator). Sounds ok?

View raw message