lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wenbo Zhao <zha...@gmail.com>
Subject Re: Can Lucene unite multiple instances run as one ?
Date Mon, 16 Nov 2009 04:39:26 GMT
My data is categorized by date.  About 14M+ docs per month, 37M+ terms.
When I use 1G heap size to do search of 10 month index, I got OOM.
The problem is I can't increase heap size in an easy way.
I have several machines, all 32bit windows, 4G ram.
And my goal is to index 10 year's data, plus more data every day !
If I put all of them together, I will need 8G+ ram to run search.
Maybe another 8G+ ram to run indexwriter.

I think to split large index into smaller indexes and use a group of
machines to work as one is more flexible and faster compare to one
huge ram machine.
Any suggestions ?  beside more rams.


2009/11/16 Jacob Rhoden <jrhoden@unimelb.edu.au>:
> Not sure how large your index is,  but it might be easier (if possible to
> increase your memory) than to develop a fairly complicated alternative
> strategy.
>
> On 16/11/2009, at 2:12 PM, Wenbo Zhao wrote:
>
>> Hi, all
>> I'm facing a large index, on a x86 win platform which may not have big
>> enough jvm heap space to hold the entire index.
>> So, I think it's possible to split the index into several smaller
>> indexes, run them in different jvm instances on different machine.
>> Then for each query, I can concurrently run it one every indexes and
>> merge the result together.
>> This can be a workaround of OutOfMemory issue.
>> But before I start to do this, I want to ask if Lucene already have a
>> solution for things like this.
>> Thanks.
>>
>> --
>>
>> Best Regards,
>> ZHAO, Wenbo
>>
>> =======================
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
> ____________________________________
> Information Technology Services,
> The University of Melbourne
>
> Email: jrhoden@unimelb.edu.au
> Phone: +61 3 8344 2884
> Mobile: +61 4 1095 7575
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>



-- 

Best Regards,
ZHAO, Wenbo

=======================

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message