jackrabbit-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefan Guggisberg <stefan.guggisb...@gmail.com>
Subject Re: the 1mn test: first jackrabbit performance and scalability tests
Date Fri, 12 Nov 2004 10:08:45 GMT
On Thu, 11 Nov 2004 18:44:13 +0100, Ugo Cei <ugo@apache.org> wrote:
> Il giorno 11/nov/04, alle 17:27, David Nuescheler ha scritto:
> 
> >> I ran the code you posted on my local box and the results
> >> are mostly comparable. However, I got a big performance
> >> boost when using the LocalFileSystem on a Reiser4
> >> filesystem (Linux 2.6.8): between 4 and 5 times faster
> >> then ext3!
> > cool. very interesting.

very good news indeed :)

> 
> I performed the load test comparing the
> ObjectPersistenceManager+CQFileSystem vs.
> ObjectPersistenceManager+LocalFileSystem on Reiser4. The surprising
> result is that they go at about the same speed! LFS on Reiser4 starts
> at about 5s/1000n and grows slowly from that. After about 50k nodes,
> the speed is around 9s. CQFS is comparable. I suspect the slowdown is
> due to the memory consumption of the JVM growing rather than to the
> number of files on disk.
> 
> Now, I find this very interesting, since I'd rather use a /real/
> filesystem than something like CQFileSystem, which puts everything
> inside one big file: I suspect this to be unmanageable when the size of
> your repository becomes quite large. Please correct me if I have the
> wrong impression.

the original motivation for writing our own unix-style file system in-a-file
was the need for a file system that can handle a very large number of 
very small files in a very efficient way. the windows file system in particular
really sucks in this department. in our product we use the cq file system 
for storing/maintaining search indexes and you can believe me that 
those can get pretty large ;) 

don't get me wrong, i am not promoting the use of our proprietary 
file system. the only reason why i brought this in was to show that
storing jackrabbit's state model in a file system need not necessarily
be inefficient/slow. i am very happy that you proved that with the 
reiser fs!

btw: the author of the cq file system is dominique pfister (the guy who
contributed the JTA support code).

> 
> I also suspect that it would be hard to beat these times with a JDBC
> persistence manager.

that's my bet too. i guess that the only thing that would be even harder 
to beat is a b-tree based persistence manager (e.g. using berkeley db).

on the other hand, the performance of the raw read/write operations 
doesn't seem to be the limiting factor anymore, now that we have 
those options. 

there's so much going on in a addNode/save cycle (transient changes,
versioning, observation, transaction support, validations, search
index updating,
etc. etc.) that propbably has a greater potential for performance optimization
than the raw read/write operations (where we now have reached a reasonable
performance). 

that's just my personal impression, i am not saying we shouldn't
tweak persistence managers/file systems if there's obvious potential for 
doing so.

> 
> Finally, maybe the LocalFileSystem class can be optimized further, what
> do you think?

absolutely, please feel free to tweak anything you wish.

cheers
stefan

> 
>        Ugo
> 
> 
> 
> --
> Ugo Cei - http://beblogging.com/
> 
> 
>

Mime
View raw message