apr-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cliff Woolley <cliffwool...@yahoo.com>
Subject Re: Buckets destroy not cleaning up private structures?
Date Mon, 30 Apr 2001 19:20:17 GMT


  [Forgive me, Bill & Justin, for redirecting my answers to your questions
   back to the list... it's just that the questions are very good ones
   and others might be wondering the same things or have other opinions
   on the matter.]



On Mon, 30 Apr 2001, Justin Erenkrantz wrote:

> On Mon, Apr 30, 2001 at 02:16:52PM -0400, Cliff Woolley wrote:
> > bucket as long as refcount==1 (ie, there are no requests using that
> > file/mmap currently in progress) if it decides it has too many MMAPs
> > laying around.  It's possible that the file will be re-MMAPed by a later
>
> How do we decide that?  Doesn't that depend on your OS and the size of
> the MMAP extents (imagine 2 2GB files in your cache - which may be fine
> on certain OSes but not on others - imagine 64bit address spaces).  That
> is my inherent problem with using MMAP.

To be more precise, "too many MMAPs" is not just a factor of sheer number,
it's (size1+size2+size3+...+sizen > OVERALL_MMAP_BYTES_LIMIT).  I just
made that constant up, obviously, but apr_buckets_file.c defines
MMAP_LIMIT for its own purposes (if it's not already defined, that is), so
I see no reason that mod_file_cache couldn't similarly define a global
limit in the same vein.

The file buckets code has a default MMAP_LIMIT for a given file of 4MB,
just to give you an idea.  Any file bigger than that won't get MMAPed in
the first place, at least not by the buckets code.  Multiply that times
some sane number of files you wish to have MMAPed at one time, and voila,
you've got your global limit for mod_file_cache.

Right?

> And, in the case of mod_mbox, you could have extremely large files (some
> of the mailing lists I've used it with have file sizes of 750MB+).  I've
> thought about it some - I'm using MMAP when I create the SDBM indices
> with mod_mbox (the indices are creates offline - outside of Apache) -
> makes it substantially faster than just reading 4K or 8K at a time (and
> the code is so simple).
>
> I'd love to see SDBM use MMAP (and have coded it up - but I am unsure of
> its performance gains - the poor thing is reading the same file over and
> over again in 1K blocks - which I guess isn't a bad idea when you consider
> most OSes will cache it), but I haven't submitted a patch because MMAP
> gets scary when you could run out of address space.

Yeah... you're going to have to put some kind of arbitrary conservative
cap on it.  Above that threshold, you just have to live with the
read-performance penalty.

> Or, is there a common way to know when you might run out of address
> space?  -- justin

Not unless we come up with some kind of global constant to that effect in
APR, which I suppose might be doable.  I don't really have an informed
opinion on the feasability/desirability of that, though, so I'll leave
that as a question open for discussion amongst the group.  Opinions?

--Cliff


--------------------------------------------------------------
   Cliff Woolley
   cliffwoolley@yahoo.com
   Charlottesville, VA



Mime
View raw message