lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Simon Willnauer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-8757) Better Segment To Thread Mapping Algorithm
Date Thu, 02 May 2019 12:41:00 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-8757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16831591#comment-16831591
] 

Simon Willnauer commented on LUCENE-8757:
-----------------------------------------

Hey Atri,

thanks for putting up this patch, here is some additional feedback:
 - can we stick with an protected non-static method on IndexSearcher subclasses should be
able to override your impl. I think it's ok to have a static method like this:
{code:java}
 public static LeafSlice[] slices (List<LeafReaderContext> leaves, int maxDocsPerSlice,
int maxSegPerSlice){code}
that you can call from the protected method with your defaults?
 - you might want to change your sort to something like this: 
{code:java}
Collections.sort(leaves, Collections.reverseOrder(Comparator.comparingInt(l -> l.reader().maxDoc())));{code}

 - I think the _Leaves_ class is unnecessary we can just use _List<LeafReaderContext>_
instead?

> Better Segment To Thread Mapping Algorithm
> ------------------------------------------
>
>                 Key: LUCENE-8757
>                 URL: https://issues.apache.org/jira/browse/LUCENE-8757
>             Project: Lucene - Core
>          Issue Type: Improvement
>            Reporter: Atri Sharma
>            Priority: Major
>         Attachments: LUCENE-8757.patch
>
>
> The current segments to threads allocation algorithm always allocates one thread per
segment. This is detrimental to performance in case of skew in segment sizes since small segments
also get their dedicated thread. This can lead to performance degradation due to context switching
overheads.
>  
> A better algorithm which is cognizant of size skew would have better performance for
realistic scenarios



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message