httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michelle Konzack <linux4miche...@freenet.de>
Subject [users@httpd] Re: serving millions of files
Date Fri, 04 Feb 2005 18:27:08 GMT
Hello James,

I have a Server-Trio runing on the first my apache 1.3, on the second
Postgresql 7.1 (used/max 112GByte/240GByte) and on the third the
binaries (used/max 480GByte/800GByte, around 8 million files),
which can not be stored in the PostgreSQL. 

I have a directory struktur on the third Server too, but I have
encountered, if I make the Directories to deep, it will slow down
the filesystem.  I use only "/01/02/03/file_123" which give me
1 milliarde (am. billion) files on one partition (in theorie).

Having up to 1000 files per directory is acceptabel and will not
significant slowdown the Filesystem.

If you use "xfs" (I have my first experience since 2 weeks), you can
put 100.000 files in one directory and apache is fast as the heaven.

Am 2005-02-04 14:50:51, schrieb James Richardson:
> 
> In my experience ( I have done this before ), you really don't want 10's
> of thousands of files per directory. The reason is that directories are
> composed of limited size blocks, and to make huge directories requires
> many of these blocks chained together, and thus makes searching for files
> really slow. (By searching, I mean looking for the file to actually open
> it). This is because directory entries are stored unordered in directory
> blocks (even true for reiser4).
> 
> There are some physical limitations to the numbers of files on disk, I
> can't remember them offhand.

Are you sure ?  -  I have around 18 million on ONE ext3 partition
because it is a Maildir from courier-imap.  :-)

If reiserfs can not handel it, you need anoter fs like ext3.

Oh yes, some of the Directotries have more then 100.000 messages stored.

> So, then, best thing is to store a wide flat tree, using only a few
> directories at each level, then a bunch of files (10 or so) at the bottom.
> 
> Assuming that each file has a number, you can then create some algorithm
> to map the file number to a directory path.
> 
> e.g. 1 = /a/a/a/a/a/a/a/1
>      2 = /a/a/a/a/a/a/b/2
>     27 = /a/a/a/a/a/a/a/27
> 
> Or something like that. The depth of the tree gives you the number of
> files that it will store. Assuming 10 files per directory, and seven
> levels, you get about 80 billion files or something. ( Would the
> filesystem support this many files, I don't know ). 

I do not know, wheter you can create 27x27x27x27x27x27x27x27
(= 282.429.530.000 Dirs) but AFAIK does searching directories
slow down the Computer.

I have a 3Ware Raid with 3 x 400 GByte wnd ONE partition of around
800 GByte with 1 kByte blocksize. I make a test:

  __( '/home/michelle.konzack/bin/a_test' )_____________________________
 /
| #!/bin/bash
| 
| let COUNT=0
| for A in a b c d e f g h i j ; do
|   let COUNT=$COUNT+1
| for B in a b c d e f g h i j ; do
|   let COUNT=$COUNT+1
| for C in a b c d e f g h i j ; do
|   let COUNT=$COUNT+1
| for D in a b c d e f g h i j ; do
|   let COUNT=$COUNT+1
| for E in a b c d e f g h i j ; do
|   let COUNT=$COUNT+1
| for F in a b c d e f g h i j ; do
|   let COUNT=$COUNT+1
| for G in a b c d e f g h i j ; do
|   let COUNT=$COUNT+1
| for H in a b c d e f g h i j ; do
|   let COUNT=$COUNT+1
| for I in a b c d e f g h i j ; do
|   echo "$COUNT :  $A/$B/$C/$D/$E/$F/$G/$H/$I/{a,b,c,d,e,f,g,h,i,j}"
|   mkdir -p /home/michelle.konzack/tmp/a_test-1/$A/$B/$C/$D/$E/$F/$G/$H/$I/{a,b,c,d,e,f,g,h,i,j}
|   let COUNT=$COUNT+10
| done ; done ; done ; done ; done ; done ; done ; done ; done
 \______________________________________________________________________

I have stoped creating directories by 6.8 million...

> Hope that helps,
> 
> Best Regards,
> 
> James

Greetings
Michelle

-- 
Linux-User #280138 with the Linux Counter, http://counter.li.org/ 
Michelle Konzack   Apt. 917                  ICQ #328449886
                   50, rue de Soultz         MSM LinuxMichi
0033/3/88452356    67100 Strasbourg/France   IRC #Debian (irc.icq.com)

Mime
View raw message