lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ganesh" <emailg...@yahoo.co.in>
Subject Re: Single searcher vs Multi Searcher
Date Mon, 06 Oct 2008 04:36:47 GMT
Hello Anshum,

My index is growing 1 million documents per day. Initially i planned to have 
a single database but the sorting of one or more fields consumes more RAM. 
Whether sharding the index would also consume the same.

My application should co-exist with other application of my product and my 
app could get 1 GB of RAM. Search speed is fine but i need to display the 
result in the sorted order.

I thought to keep 7 days of documents in one index and create one more after 
the 7 days. After 30 days the first index may get deleted. I need to keep 
the documents in the index DB for 30 days. My Index DB is in HDD.

I want to the pros and cons of sharding. I think maintance of the DB becomes 
easier.

It would be very much helpful, if you share some of your thoughts.

Regards
Ganesh


----- Original Message ----- 
From: "Anshum" <anshumg@gmail.com>
To: <java-user@lucene.apache.org>
Sent: Friday, October 03, 2008 9:48 PM
Subject: Re: Single searcher vs Multi Searcher


> Hi Ganesh,
>
> I have experimented with sharded indexes and they seem to benefit 
> me(atleast
> in my case). I would like to know a few things before I answer your
> question:
> 1. Do you have a reasonable criteria ( a calculated one) to shard the
> indexes?
> 2. How do you plan to split the index? Is it going to be document based
> (which I guess it should be as otherwise you would have to build a 
> complete
> distributed system)
> 3. Do you plan to put your indexes on the RAM or on (physically) seperate
> HDDs?
>
> Though all said and done, sharded indexes are a good approach, if done the
> right way.
> --
> Anshum Gupta
> Naukri Labs!
> http://ai-cafe.blogspot.com
>
> The facts expressed here belong to everybody, the opinions to me. The
> distinction is yours to draw............
>
>
> On Fri, Oct 3, 2008 at 3:01 PM, Ganesh <emailgane@yahoo.co.in> wrote:
>
>> Hello all,
>>
>> My indexing is growing by 1 million records per day and the memory
>> consumption of the searcher object is quite high.
>>
>> There are different opinion in the groups. Few suggest to use single
>> database and few to use sharding. My Database has 10 million records now 
>> and
>> it might go till 30 million or more. I plan to shard the index. but
>> Multisearcher will give me benifit.
>>
>> Regards
>> Ganesh
>>
>>
>> Send instant messages to your online friends 
>> http://in.messenger.yahoo.com
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
> 

Send instant messages to your online friends http://in.messenger.yahoo.com 

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message