lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aleksander M. Stensby" <aleksander.sten...@integrasco.no>
Subject Re: Sharding strategy
Date Wed, 10 Jun 2009 06:56:49 GMT
Hi Otis,
thanks for your reply!
You could say I'm lucky (and I totally agree since I've made the choice of  
ordering the data that way:p).
What you describe is what I've thought about doing and I'm happy to read  
that you approve. It is always nice to know that you are not doing things  
completely off - that's what I love about this mailing list!

I've implemented a sharded "yellow pages" that builds up the shard  
parameter and it will obviously be easy to search in two shards to  
overcome the beginning of the year situation, just thought it might be a  
bit stupid to search for 1% of the data in the "latest shard" and the rest  
in shard n-1. How much of a performance decrease do you recon I will get  
 from searching two shards instead of one?

Anyways, thanks for confirming things, Otis!

Cheers,
  Aleksander




On Wed, 10 Jun 2009 07:51:16 +0200, Otis Gospodnetic  
<otis_gospodnetic@yahoo.com> wrote:

>
> Aleksander,
>
> In a sense you are lucky you have time-ordered data.  That makes it very  
> easy to shard and cheaper to search - you know exactly which shards you  
> need to query.  The beginning of the year situation should also be  
> easy.  Do start with the latest shard for the current year, and go to  
> next shard only if you have to (e.g. if you don't get enough results  
> from the first shard).
>
>  Otis
> --
> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>
>
>
> ----- Original Message ----
>> From: Aleksander M. Stensby <aleksander.stensby@integrasco.no>
>> To: "solr-user@lucene.apache.org" <solr-user@lucene.apache.org>
>> Sent: Tuesday, June 9, 2009 7:07:47 AM
>> Subject: Sharding strategy
>>
>> Hi all,
>> I'm trying to figure out how to shard our index as it is growing  
>> rapidly and we
>> want to make our solution scalable.
>> So, we have documents that are most commonly sorted by their date. My  
>> initial
>> thought is to shard the index by date, but I wonder if you have any  
>> input on
>> this and how to best solve this...
>>
>> I know that the most frequent queries will be executed against the  
>> "latest"
>> shard, but then let's say we shard by year, how do we best solve the  
>> situation
>> that will occur in the beginning of a new year? (Some of the data will  
>> be in the
>> last shard, but most of it will be on the second last shard.)
>>
>> Would it be stupid to have a "latest" shard with duplicate data (always
>> consisting of the last 6 months or something like that) and maintain  
>> that index
>> in addition to the regular yearly shards? Any one else facing a similar
>> situation with a good solution?
>>
>> Any input would be greatly appreciated :)
>>
>> Cheers,
>> Aleksander
>>
>>
>>
>> --Aleksander M. Stensby
>> Lead software developer and system architect
>> Integrasco A/S
>> www.integrasco.no
>> http://twitter.com/Integrasco
>>
>> Please consider the environment before printing all or any of this  
>> e-mail
>
>



-- 
Aleksander M. Stensby
Lead software developer and system architect
Integrasco A/S
www.integrasco.no
http://twitter.com/Integrasco

Please consider the environment before printing all or any of this e-mail

Mime
View raw message