httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eli Marmor <>
Subject Buckets vs. Re-allocations
Date Wed, 17 Oct 2001 19:07:33 GMT
With more and more developers, modules and filter programmers, etc.,
join the 2.0 effort, I see the following dilemma as an important issue;

Are there "rules"?

Any time that the issue of "realloc()" is raised, the automatic answer
is that instead of resizing your buffer to include the extended
content using realloc(), it is better to just add one more bucket.

So instead of having a big buffer, that must be resized any time a new
content is added to it, we have a "brigade" of "buckets".

However, things are not "black or white":

Sometimes, realloc() may be used wisely, so it is not called any time,
but allocates more than needed (e.g., when new_size > old_size, instead
of realloc(buf, new_size), it is possible to realloc(buf, new_size*3/2)
so next times "even_newer_size" may be higher than "new_size", but not
than 1.5 times of it).

Sometimes, realloc() doesn't have to move memory areas, since the space
after the current buffer is not in use, and only heaps and malloc maps
should be fixed.

Sometimes, one of the receiving filters, down the stream, is not wise,
and does I/O per each bucket.

Sometimes, there are *MANY* small chunks (e.g. lines of content), that
may cause the method of brigades/buckets to be much heavier, especially
when realloc() allocates each time much more than needed, so a real re-
allocation is done only once per many chunks. Just try to think, for
example, about a parser filter, which processes one tag/comment/element
by one; It may end up with hundreds buckets or even more. Isn't it
better to just realloc a buffer in steps of, for example, 20KB in a
time, and pass it to the next filter?  And where is the exact "border"
between realloc() and splitted buckets?

I don't try to argue for apr_realloc() (although many will thank you
for doing it, even if a warning "isn't recommended for use!" will be
attached to it...);
I just want to know if it is possible to guide developers when the
brigades/buckets method is the right way, and when it is better to just
realloc(), bypassing the apr_ functions (assuming that people know what
they are doing, and don't leave memory leaks...).

Eli Marmor
CTO, Founder
Netmask (El-Mar) Internet Technologies Ltd.
Tel.:   +972-9-766-1020          8 Yad-Harutzim St.
Fax.:   +972-9-766-1314          P.O.B. 7004
Mobile: +972-50-23-7338          Kfar-Saba 44641, Israel

View raw message