lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacob Rhoden <jrho...@unimelb.edu.au>
Subject Re: Can Lucene unite multiple instances run as one ?
Date Mon, 16 Nov 2009 04:44:38 GMT
Sounds like you may need to have some sort of distributed system, I  
just wanted to make sure you were aware of the cost/benifits of just  
buying a big 62bit/8Gb ram machine, vs having to not only maintain and  
power several 32 bit machines, but also maintain and support your now  
more complicated code.

I have seen it too many times developers/companies spend so much money  
in not just the initial development, but long term support and  
maintenance that could have been simplified by just buying a bigger/ 
better more powerful machine in the first place.

I am interested to see what other people have to say about how to  
solve your problem.

Best regards,
Jacob

On 16/11/2009, at 3:39 PM, Wenbo Zhao wrote:

> My data is categorized by date.  About 14M+ docs per month, 37M+  
> terms.
> When I use 1G heap size to do search of 10 month index, I got OOM.
> The problem is I can't increase heap size in an easy way.
> I have several machines, all 32bit windows, 4G ram.
> And my goal is to index 10 year's data, plus more data every day !
> If I put all of them together, I will need 8G+ ram to run search.
> Maybe another 8G+ ram to run indexwriter.
>
> I think to split large index into smaller indexes and use a group of
> machines to work as one is more flexible and faster compare to one
> huge ram machine.
> Any suggestions ?  beside more rams.
>
>
> 2009/11/16 Jacob Rhoden <jrhoden@unimelb.edu.au>:
>> Not sure how large your index is,  but it might be easier (if  
>> possible to
>> increase your memory) than to develop a fairly complicated  
>> alternative
>> strategy.
>>
>> On 16/11/2009, at 2:12 PM, Wenbo Zhao wrote:
>>
>>> Hi, all
>>> I'm facing a large index, on a x86 win platform which may not have  
>>> big
>>> enough jvm heap space to hold the entire index.
>>> So, I think it's possible to split the index into several smaller
>>> indexes, run them in different jvm instances on different machine.
>>> Then for each query, I can concurrently run it one every indexes and
>>> merge the result together.
>>> This can be a workaround of OutOfMemory issue.
>>> But before I start to do this, I want to ask if Lucene already  
>>> have a
>>> solution for things like this.
>>> Thanks.
>>>
>>> --
>>>
>>> Best Regards,
>>> ZHAO, Wenbo
>>>
>>> =======================
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>
>> ____________________________________
>> Information Technology Services,
>> The University of Melbourne
>>
>> Email: jrhoden@unimelb.edu.au
>> Phone: +61 3 8344 2884
>> Mobile: +61 4 1095 7575
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
>
>
> -- 
>
> Best Regards,
> ZHAO, Wenbo
>
> =======================
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

____________________________________
Information Technology Services,
The University of Melbourne

Email: jrhoden@unimelb.edu.au
Phone: +61 3 8344 2884
Mobile: +61 4 1095 7575


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message