httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brandon Fosdick <>
Subject Re: Large file support in 2.0.56?
Date Thu, 20 Apr 2006 03:20:46 GMT
Brandon Fosdick wrote:
 > It does seem like a rather small and arbitrary limit. I can't think of 
> what else besides apache would cause it, but I could be missing 
> something. The files are being dumped into mysql in 64K blocks. The 
> machine is an amd64, so that shouldn't be a problem, and 700MB isn't 
> near 2 or 4 GB anyway. Uploading from a cable modem doesn't go anywhere 
> near saturating the disk, cpu, or network. I've tried OSX, Win2k and 
> WinXP, all with the same result. I'm running out of things to check. Any 
> suggestions? I guess it could be a limit in mod_dav itself. I'm afraid 
> to go looks messy.

In the interest of posterity and curiosity I played with 2.0.55 a little bit more, and I thought
I'd document it here in case anyone cares.

My test setup includes a server running FreeBSD 6.1-PRE on a Sempron 3100+ and a PowerBookG4
(Tiger 10.4.5) acting as the client. I wrote a mod_dav storage provider that uses mysql as
a back end. Files are stored as a series of 64K records.

Previously I had reported upload failure of files greater than ~700MB. On a hunch I took half
the RAM out of the server, just to see if it's a memory problem. Sure enough, the limit appears
to be in the 300MB - 400MB range now (specifically, a win2k iso). But that still seemed odd,
since from previous testing I had determined that httpd/mod_dav streams the files to my provider
in 2K chunks, which are then assembled into 64K chunks and written to the database. It didn't
seem likely that I had enough bytes buffered at any one time to cause memory starvation. Plus,
I have more than enough swap configured.

So I redid the test with top running (for lack of a better idea) and a debug entry going to
the error log for each chunk of bytes given to the storage provider. Bytes are still being
provided in 2K chunks. I observed the following sequence:

1. OSX transfer window opens, progress bar proceeds rapidly to completion, and then says "closing
file", which it continues to say throughout the transfer. httpd and mysqld are using 0% of
the CPU and 0% of disk i/o. No log entries.

2. httpd jumps to using ~20% of the CPU and 100% of disk i/o. Debug log entries begin. Entries
report 2K chunks.

3. 1-2 minutes later, OSX reports "server disconnected". mod_dav is still streaming 2K chunks,
httpd is still using 100% disk i/o.

4. 1-2 minutes later, httpd client appears to crash and respawn. It appears to either restart
the transfer or re-stream the file to storage, but at this point the log file becomes corrupt.

5. If I cancel the transfer at this point at least one of the httpd processes starts using
100% CPU and 100% disk i/o. Then I killall httpd.

For completeness, I re-ran the above test without the debug logging and w/o mod_security and
observed the same behavior.

At this point I'm not sure if I should bother trying the large file hack for 2.0.55 or just
start migrating to 2.2.x. This no longer seems to be a large file problem, but I'm not sure
what kind of problem it is. Judging by the mod_security audit log, the client doesn't appear
to be doing anything odd. That is, it's just issuing a LOCK and a PUT for each file upload.
Naturally, my own code is suspect as well, but it never sees anything bigger than 64K, and
it can handle larger files when I haven't taken half the RAM out.

I'm stumped.

Unless anyone has a better idea I'll just give 2.2 a try and see what happens.

View raw message