httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Leif W" <warp-...@usa.net>
Subject Re: [users@httpd] serving millions of files
Date Fri, 04 Feb 2005 00:05:18 GMT
> "Eric Ladner" <eric.ladner@gmail.com>; 2005 February 03 Thursday 15:38
>
> On Thu, 3 Feb 2005 13:32:14 -0600, John Bohumil
> <jbohumil@mn-exch.tcfbank.com> wrote:
>> Should I consider one Linux file system over another?  Is there good 
>> reason to split these up into smaller directories vs. everything in 
>> one huge directory?
>>
>> Thanks for any feedback!
>> John
>
> Consider the reiserfs.  It's really good at storing large amounts (>
> 1million) of files in a directory and not taking forever to move,
> delete them.

I have absolutely no idea.  So when I have no idea, the best advice is 
to look at all the options, devise a fairly simple, standard test 
scenario, and generate some hard numbers on performance indicators, and 
make a decision that is based on some facts and growth projections.  One 
other option in Linux is ext3, another journalling file system.  60,000 
new and 60,000 removed files a day, and there are (24 h/d * 60 m/h * 60 
s/m) or 86,400 seconds per day, so it's over one file operation per 
second.  Then you have 30 days worth, 1.8 million files, and I assume 
they are going to be accessed during that 1-month period.  How often 
will each file be accessed on average?  How many file accesses will 
there be each day?  Understand the subtle difference of those questions? 
Will it always be 60,000/day?  Or will it grow insanely fast to 
600,000/day within 3 months?  Do some calculations on the threshhold 
times you will need in order to actually do everything (file creations, 
script running, web serving, file removals) and maintain acceptable 
performance levels.

In any case, you'll have at least 1.8 million files available each 
month.  If too many people are trying to access files simultaneously, 
then you may need to recompile or reconfigure your kernel to allow more 
simultaneously opened files (used to be like 1024, 2048, 4096 or 8192), 
and you may need to rebuild or reconfigure Apache with higher connection 
limits (Apache 1.3 used to have 512 connections as the default).

Leif



---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Mime
View raw message