httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Davi Arnaut <>
Subject Re: svn commit: r467655 - in /httpd/httpd/trunk: CHANGES docs/manual/mod/mod_cache.xml modules/cache/mod_cache.c modules/cache/mod_cache.h
Date Wed, 25 Oct 2006 21:48:31 GMT
Graham Leggett wrote:
> Davi Arnaut wrote:
>> Yes, first because size_t is 32 bits :). When you do a read like this on
>> a file bucket, the whole bucket is not read into memory. The read
>> function splits the bucket and changes the current bucket to refer to
>> data that was read.
> 32 bits is 4GB. A large number of webservers don't have memory that 
> size, thus the problem.
>> The problem lies that those new buckets keep accumulating in the
>> brigade! See my patch again.
> Where?

I was referring to "it will attempt to read all 4.7GB". Such a thing
does not exist!

> We start with one 4.7GB bucket in a brigade.

No, we start with a file bucket on the brigade, I will try to explain
what happens to see if we are talking about the same thing.

Suppose we have a brigade containing a a file bucket, and the file size
is 4.7GB. We want to read it fully.

When we call apr_bucket_read() on this bucket, we end-up calling the
bucket read function (file_bucket_read). What does the bucket file read do ?

If mmap is supported, it mmaps APR_MMAP_LIMIT bytes (4MB) by creating a
new mmap bucket and splits the bucket. So, after calling read on a file
bucket you have two buckets on the brigade. The first one is the mmap
bucket and last is file bucket with a updated offset.

The same thing happens if mmap is not supported, but the bucket type
will be a heap bucket. If we don't delete or flush those implicitly
created buckets they will keep the whole file in memory, but one single
read will not put the entire file on memory.

What Joe's patch does is remove this first implicitly created bucket
from the brigade, placing it on the brigade on a temporary brigade for
sending it to the client.

That's why splitting the brigade with magical values (16MB) is not such
a good idea, because the bucket type knows betters and will split the
bucket anyway.

Davi Arnaut

View raw message