apr-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cliff Woolley <cliffwool...@yahoo.com>
Subject Bucket memory allocation (was Re: SMS parm to bucket_foo_create())
Date Sun, 26 Aug 2001 01:13:42 GMT
On Sat, 25 Aug 2001, Ryan Bloom wrote:

> Woah!  Those patches were large, and I don't think I agree with them
> at all.

The fact that it's a big patch is why I've put it off this long.  I had
that first phase written months ago and never committed it because it was
large.  Note that much of that first patch was stuff that would have come
right back out again in phase 2... some temporary hacks were necessary as
while I made the basic changes and before changing the API so I could have
something I could test for breakage at several points along the way.

This is the approach we've been talking about doing for months and nobody
objected before as far as I can remember.  I posted this three-phase-plan
thingy on the list on July 8 and nobody voiced any concerns.  It just
hasn't got done because no one ever got around to it.  Phase 1 seemed
harmless enough (big though it was)... but now that I'm working on Phase
2, the more I see how greatly it affects the API, the less I like it, so
I'm willing to change.

> Plus, now we sometimes have both pools and sms's in the
> same structure. How are we synch'ing those two?

We're not.  It's annoying, I know, but we're just counting on the code not
to clean up the data before while we're still using the buckets.  But we
already do that, so that's not a change in the requirements.  Anyhow, the
idea is that the SMS used will live as long as the thread (which is
certainly at least as long as the data), and the caller just keeps giving
you back a pointer to that SMS so you don't have to look up your thread ID
all the time.

> Why can't we just create a free list for buckets?  That way, we take
> the hit for allocating buckets only when necessary, and at some point,
> we will reach a steady state.  The free list, could be per thread, and
> I believe I could have it coded tomorrow night.

That's what we were going to do originally when you and Greg and I and
the others started talking about this, what, 7 or 8 months ago?

But then when the SMS's came around they seemed ideal because we could
just write ourselves up a blocks SMS that gave us our free list tailored
the way we wanted it and pop it in.  The SMS is intended to be per-thread
and last as long as the thread though extra allocated memory might be
cleaned up now and then to avoid leaks, but because we get to choose when
that cleanup occurs, we can still reach our steady state.  As a fringe
benefit, it has been indicated on-list that some custom bucket writers
might want to use custom memory allocation systems, and this takes care of
that without us having to do anything special at all.

(Assume for the purposes of this paragraph that we ARE going to use SMS.)
It was debated whether the SMS to be used should be looked up by the
buckets code itself or whether it should be passed in by the caller.
Passing it in by the caller is what eventually came out seeming the best
because otherwise the buckets code (which has no way to know what its
thread ID is) has to go ask the OS for its thread ID and then look up the
SMS in a hash table or something based on that ID.  If the caller gives it
to us, the caller (which DOES know the thread ID at some point) can just
stash a pointer to that thread's SMS somewhere and pass it in to the
bucket functions and it's constant time instead of pseudo-constant or
logarithmic or linear or whatever.  On the other hand, now that I'm
actually implementing that as phase 2, I sort of dislike how cluttered it
makes the API seem.  So we could take a fork in the road from here and
leave it up to the buckets to find their thread ID.  Less performant, but
not terribly bad.  Probably still better than malloc, though the win is
less clear.  Or we could keep going.

(Assume for the purposes of this paragraph that we're NOT going to use
SMS, either because the group consensus is "SMS were a nice try, but they
couldn't beat pools, so it's time to get rid of them altogether" or
because we'd just rather do something custom here.)  The same thread ID
problem exists for a buckets-specific-non-SMS free list, so you'd end up
doing the call to find out what your thread ID is and then do your hash
lookup because the caller can't help you anymore.

So that was the logic behind it.



   Cliff Woolley
   Charlottesville, VA

View raw message