lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tedsolr <tsm...@sciquest.com>
Subject Re: Specify sorting of merged streams
Date Thu, 21 Jul 2016 21:31:45 GMT
The primary use case seems to require a SortStream. Ignoring the large join
for now...
1. search main collection with stream (a)
2. search other collection with stream (b)
3. hash join a & b (c)
4. full sort on c
5. aggregate c with reducer
6. apply user sort criteria with top

It's very likely that a field from 'b' will be in the user's search criteria
and the results list. Therefore the merge (3) must occur before the
aggregation (5), and the aggregation requires the sort (4). If this can't be
done without lots of hardware and memory, perhaps I'd be better off leaving
the data denormalized and increasing index speed by sharding (with many more
VMs). Will sharding increase re-indexing speed by a factor close to the # of
shards (collection with 10 shards indexes ~10x faster than same collection
with one shard)? 


Joel Bernstein wrote
> The tricky thing you have is a large join coupled with a reduce() group,
> which have different sorts.
> 
> ...
> 
>> Joel Bernstein wrote
>> > A few other things for you to consider:
>> >
>> > 1) How big are the joins?
>> > 2) How fast do they need to go?
>> > 3) How many queries need to run concurrently?
>> >
>> > #1 and 2# will dictate how many shards, replicas and parallel workers
>> are
>> > needed to perform the join. #3 needs to be carefully considered because
>> > MapReduce distributed joins are not going to scale like traditional
>> Solr
>> > queries.
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://lucene.472066.n3.nabble.com/Specify-sorting-of-merged-streams-tp4285026p4288194.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>





--
View this message in context: http://lucene.472066.n3.nabble.com/Specify-sorting-of-merged-streams-tp4285026p4288288.html
Sent from the Solr - User mailing list archive at Nabble.com.

Mime
View raw message