lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Simon Willnauer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-8757) Better Segment To Thread Mapping Algorithm
Date Fri, 03 May 2019 08:29:00 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-8757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832343#comment-16832343
] 

Simon Willnauer commented on LUCENE-8757:
-----------------------------------------

Thanks [~atris], can you bring back the javadocs for 
{code:java}
protected LeafSlice[] slices(List<LeafReaderContext> leaves){code}

please don't reassign an argument like here:


{code:java}
leaves = new ArrayList<>(leaves);
{code}

The rest of the patch looks OK to me yet I am not so sure about the defaults. I do wonder
if we should look at this from a different perspective. Rather than using hard numbers can
we try to evenly balance the total number of documents across N threads and make N the variable?
[~mikemccand] WDYT?


> Better Segment To Thread Mapping Algorithm
> ------------------------------------------
>
>                 Key: LUCENE-8757
>                 URL: https://issues.apache.org/jira/browse/LUCENE-8757
>             Project: Lucene - Core
>          Issue Type: Improvement
>            Reporter: Atri Sharma
>            Priority: Major
>         Attachments: LUCENE-8757.patch
>
>
> The current segments to threads allocation algorithm always allocates one thread per
segment. This is detrimental to performance in case of skew in segment sizes since small segments
also get their dedicated thread. This can lead to performance degradation due to context switching
overheads.
>  
> A better algorithm which is cognizant of size skew would have better performance for
realistic scenarios



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message