lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karl Øie <k...@gan.no>
Subject Re: Stress/scalability testing Lucene
Date Thu, 21 Nov 2002 14:35:04 GMT
I have a index that is compiled each night that indexes 1,3gb with XML 
data that results into a 1,4gb index. The index takes about 11 hours to 
build on a dual 700mhz xeon processor with 768mb of ram. The index 
contains 4.388.730 documents and 953.632 terms.

Mvh karl øie


On Thursday, Nov 21, 2002, at 15:04 Europe/Oslo, Ian Lea wrote:

> Otis
>
>
> I could do some tests on a solaris and/or intel linux server
> that have got about 10 to 15Gb disk space to play with.
> Neither has huge amounts of RAM (512 and 256Mb respectively)
> but that is probably enough.  Neither server is doing much
> else most of the time.
>
> If that would help would you like me to run your test code
> to try and get consistent results, or shall I knock up my
> own code?
>
>
>
> --
> Ian.
> ian@digimem.net
>
>> otis_gospodnetic@yahoo.com (Otis Gospodnetic) wrote
>>
>> Hello,
>>
>> Has anyone tested Lucene for scalability?
>> I know that some peple have indices with 10M+ documents in it, but has
>> anyone tried going beyond there, to 50M, 100M, 500M or more documents?
>> (I know the size of the index and performance of searches depends on
>> documents, number of fields, field types, query complexity, etc.)
>>
>> Last night I wrote a simple class that creates a Lucene index of
>> specified size with documents containing 2 fields, one Text with about
>> 24 bytes, and one UnStored without about 16000 bytes.
>> It took about 8 hours to index 100K documents, resulting in an index 
>> of
>> 578 MB (optimized).  This was on 400MHz machine with about 384MB RAM,
>> doing nothing else.
>>
>> I then realized that I can't build a relaly big index to test Lucene's
>> scalability properly, simply because I don't have a big enough disk :)
>>
>> So my question is:
>> Has anyone done this type of testing and can you share the results?
>> Does anyone have a machine with sufficient amount of RAM and disk and
>> wants to do this?
>>
>> Thanks,
>> Otis
>> P.S.
>> If anyone is wondering about those 8 hours - this was with a plain
>> IndexWriter and mergeFactor set to 1000, and java -Xms50M and -Xmx80MB
>
> ----------------------------------------------------------------------
> Searchable personal storage and archiving from http://www.digimem.net/
>
> --
> To unsubscribe, e-mail:   
> <mailto:lucene-user-unsubscribe@jakarta.apache.org>
> For additional commands, e-mail: 
> <mailto:lucene-user-help@jakarta.apache.org>

--
To unsubscribe, e-mail:   <mailto:lucene-user-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-user-help@jakarta.apache.org>


Mime
View raw message