lucene-solr-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yonik Seeley <yo...@lucidimagination.com>
Subject Re: priority queue in query component
Date Tue, 09 Feb 2010 13:25:29 GMT
The SolrCloud branch now has load balancing and fail-over amongst
shard replicas.
Partial results aren't available yet (if there are no up replicas for
a shard), but that is planned.

-Yonik
http://www.lucidimagination.com


On Tue, Feb 9, 2010 at 8:21 AM, Jan Høydahl / Cominvent
<jan.asf@cominvent.com> wrote:
> Isn't that OK as long as there is the option of allowing partial results if you really
want?
> Keeping the logic simple has its benefits. Let client be responsible for query resubmit
strategy, and let load balancer (or shard manager) be responsible for marking a node/shard
as dead/inresponsive and choosing another for the next query.
>
> --
> Jan Høydahl  - search architect
> Cominvent AS - www.cominvent.com
>
> On 9. feb. 2010, at 04.36, Lance Norskog wrote:
>
>> At this point, Distributed Search does not support any recovery if
>> when one or more shards fail. If any fail or time out, the whole query
>> fails.
>>
>> On Sat, Feb 6, 2010 at 9:34 AM, mike anderson <saidtherobot@gmail.com> wrote:
>>> "so if we received the response from shard2 before shard1, we would just
>>> queue it up and wait for the response to shard1."
>>>
>>> This crossed my mind, but my concern was how to handle the case when shard1
>>> never responds. Is this something I need to worry about?
>>>
>>> -mike
>>>
>>> On Sat, Feb 6, 2010 at 11:33 AM, Yonik Seeley <yonik@lucidimagination.com>wrote:
>>>
>>>> It seems like changing an element in a priority queue breaks the
>>>> invariants, and hence it's not doable with a priority queue and with
>>>> the current strategy of adding sub-responses as they are received.
>>>>
>>>> One way to continue using a priority queue would be to add
>>>> sub-responses to the queue in the preferred order... so if we received
>>>> the response from shard2 before shard1, we would just queue it up and
>>>> wait for the response to shard1.
>>>>
>>>> -Yonik
>>>> http://www.lucidimagination.com
>>>>
>>>>
>>>> On Sat, Feb 6, 2010 at 10:35 AM, mike anderson <saidtherobot@gmail.com>
>>>> wrote:
>>>>> I have a need to favor documents from one shard over another when
>>>> duplicates
>>>>> occur. I found this code in the query component:
>>>>>
>>>>>          String prevShard = uniqueDoc.put(id, srsp.getShard());
>>>>>          if (prevShard != null) {
>>>>>            // duplicate detected
>>>>>            numFound--;
>>>>>
>>>>>            // For now, just always use the first encountered since
we
>>>> can't
>>>>> currently
>>>>>            // remove the previous one added to the priority queue.
 If we
>>>>> switched
>>>>>            // to the Java5 PriorityQueue, this would be easier.
>>>>>            continue;
>>>>>            // make which duplicate is used deterministic based
on shard
>>>>>            // if (prevShard.compareTo(srsp.shard) >= 0) {
>>>>>            //  TODO: remove previous from priority queue
>>>>>            //  continue;
>>>>>            // }
>>>>>          }
>>>>>
>>>>>
>>>>> Is there a ticket open for this issue? What would it take to fix?
>>>>>
>>>>> Thanks,
>>>>> Mike
>>>>>
>>>>
>>>
>>
>>
>>
>> --
>> Lance Norskog
>> goksron@gmail.com
>
>

Mime
View raw message