httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sander Striker" <stri...@apache.org>
Subject RE: Allocating a buffer efficiently...?
Date Mon, 04 Mar 2002 12:05:10 GMT
> From: Igor Sysoev [mailto:is@rambler-co.ru]
> Sent: 02 March 2002 16:41

> On Sat, 2 Mar 2002, Sander Striker wrote:
> 
> > > In a recent patch to mod_proxy, a static buffer used to store data read
> > > from backend before it was given to frontend was changed to be allocated
> > > dynamically from a pool like so:
> > > 
> > > +    /* allocate a buffer to store the bytes in */
> > > +    /* make sure it is at least IOBUFSIZE, as recv_buffer_size may be
> > > zero for
> > > system default */
> > > +    buf_size = MAX(recv_buffer_size, IOBUFSIZE);
> > > +    buf = ap_palloc(r->pool, buf_size);
> > > 
> > > This change allows for a dynamically configurable buffer size, and fixes
> > > the code to be thread safe.
> > > 
> > > However: it has been pointed out that this new code makes the Apache
> > > footprint significantly larger like so:
> > > 
> > > > There is one drawback in this code. ap_palloc() is not good for
> > > > big allocations (I think > 16K) because it stores data and meta-data
> > > > together. I had found this when try to allocate memory from pool
> > > > for zlib in mod_deflate. zlib needs about 390K - 2*128K + 2*64K + 6K.
> > > > After this change Apache had grown up about 2M after about hour
> > > > with 50 requests/s. I'm not sure that this growing could continue but
> > > > I did not want additional 2M on each Apache.
> > 
> > Can you point me to the original post?  I'd like to see the context.
> > Specifically which pool is being used.
> 
> You see all context - Graham have quoted almost whole my email.
> As to pool I had tried to make big allocation from
> r->connection->client->pool. Keep-alives were off.

What you are seeing is the big allocation being pushed on the freelist.
You can prevent this by creating a subpool of the pool you were using
with a new allocator.  When the subpool is destroyed (and the allocator
to go with it) the memory is handed back to the system.

> > > > I use malloc for big allocations, store addresses in array
> > > > allocated from pool and set cleanup for this array.
> > > > In cleanup I free addresses if they is not free already.
> > > 
> > > Comments...?

We discussed this and came up with apr_tracker.  This basically
does malloc/free and can free all mallocs in one go aswell.  We (*cough* I)
never came round to finishing it.
 
> Igor Sysoev

Sander

Mime
View raw message