apr-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cliff Woolley <cliffwool...@yahoo.com>
Subject Re: apr_buckets_file.c:file_read + XTHREAD
Date Wed, 28 Nov 2001 03:35:17 GMT
On Tue, 27 Nov 2001, William A. Rowe, Jr. wrote:

> Bzzt... wrong.  There are plenty of non-zero copy operations in the
> world... this isn't the 'unique exception' ... it will be frequent
> enough to warrent redesign.
>
> > Perhaps a better solution would be to just create a read function
> > that accepts a pre-allocated buffer that we must copy into.  Then,
> > filters that know they are going to throw the data away would incur
> > the penalty of the copy in lieu of the penalty of the malloc().
> > With the file bucket, this has the advantage that the read from the
> > disk can be put directly into the pre-allocated buffer, and the copy
> > doesn't actually exist.
>
> Bingo.  That way you haven't tied it up with specific bucket types -
> but extended the api for all buckets.

Works for me.

> This is where I keep saying that the consumer often knows more about
> the mechanics of transfering or processing the data than the
> producer... until we add some graceful methods for handling those
> cases, kludges will abound.

Well, here's another kludge for you: the problem is that apr_bucket_read()
for file buckets falls into an abyss of performance as soon as the magic
APR_MMAP_LIMIT is exceeded.  That's because we give up on mmap entirely
and read 8kb at a time frome the whole bigass file, even if all we wanted
in the first place was the first 32 bytes or so (say for something like
mod_mime_magic) and we're never going to touch the other 12MB.  We still
have to pay the penalty anyway.  What if apr_bucket_read() on a file
bucket bigger than APR_MMAP_LIMIT would turn the file bucket into an mmap
bucket+file bucket pair, with the mmap bucket APR_MMAP_LIMIT in size, and
the file bucket gets the rest?  I think that would tackle this problem
nicely.  In a case like mod_mime_magic where we're only reading the
beginning of the file, we'd actually still be able to use sendfile()
with the first APR_MMAP_LIMIT's worth as a header.  :)  Of course, you
might run into a case where you're os-limited as to how much you can have
mmaped at one time per process if we buffer a large number of buckets for
some reason, but then the mmap will fail and we'll fall back on read()
[the code already does that].

It actually never occurred to me before that what happens with the code
now on a file bigger than APR_MMAP_LIMIT is that we get a bunch of 8KB
heap buckets until there is <= APR_MMAP_LIMIT _left_ in the file, and that
last bit of the file is mmaped.  So we have a bunch of 8KB heap buckets
followed by a 4MB-ish mmap.  Man, that's weird.  :)

--Cliff

--------------------------------------------------------------
   Cliff Woolley
   cliffwoolley@yahoo.com
   Charlottesville, VA



Mime
View raw message