lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ian Lea <...@digimem.net>
Subject Re: Stress/scalability testing Lucene
Date Thu, 21 Nov 2002 14:04:50 GMT
Otis


I could do some tests on a solaris and/or intel linux server
that have got about 10 to 15Gb disk space to play with.
Neither has huge amounts of RAM (512 and 256Mb respectively)
but that is probably enough.  Neither server is doing much
else most of the time.

If that would help would you like me to run your test code
to try and get consistent results, or shall I knock up my
own code?



--
Ian.
ian@digimem.net

> otis_gospodnetic@yahoo.com (Otis Gospodnetic) wrote 
>
> Hello,
> 
> Has anyone tested Lucene for scalability?
> I know that some peple have indices with 10M+ documents in it, but has
> anyone tried going beyond there, to 50M, 100M, 500M or more documents?
> (I know the size of the index and performance of searches depends on
> documents, number of fields, field types, query complexity, etc.)
> 
> Last night I wrote a simple class that creates a Lucene index of
> specified size with documents containing 2 fields, one Text with about
> 24 bytes, and one UnStored without about 16000 bytes.
> It took about 8 hours to index 100K documents, resulting in an index of
> 578 MB (optimized).  This was on 400MHz machine with about 384MB RAM,
> doing nothing else.
> 
> I then realized that I can't build a relaly big index to test Lucene's
> scalability properly, simply because I don't have a big enough disk :)
> 
> So my question is:
> Has anyone done this type of testing and can you share the results?
> Does anyone have a machine with sufficient amount of RAM and disk and
> wants to do this?
> 
> Thanks,
> Otis
> P.S.
> If anyone is wondering about those 8 hours - this was with a plain
> IndexWriter and mergeFactor set to 1000, and java -Xms50M and -Xmx80MB

----------------------------------------------------------------------
Searchable personal storage and archiving from http://www.digimem.net/


Mime
View raw message