lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Noble Paul (JIRA)" <>
Subject [jira] [Updated] (SOLR-10524) Explore in-memory partitioning for processing Overseer queue messages
Date Thu, 04 May 2017 21:36:04 GMT


Noble Paul updated SOLR-10524:
    Attachment: SOLR-10524.patch

[~caomanhdat] Looks good. 

This should be able to dramatically reduce the state update operations when overseer is under
heavy load

I have made the method {{Overseer#sortItems()}} as {{public static}} . Can you please write
simple JUnits to test the order of items produced by that method? 

[~dragonsinth] is it possible for u to take a look at the patch , especially the changes made
to {{DistributedQueue}}

> Explore in-memory partitioning for processing Overseer queue messages
> ---------------------------------------------------------------------
>                 Key: SOLR-10524
>                 URL:
>             Project: Solr
>          Issue Type: Improvement
>      Security Level: Public(Default Security Level. Issues are Public) 
>            Reporter: Erick Erickson
>         Attachments: SOLR-10524.patch, SOLR-10524.patch
> There are several JIRAs (I'll link in a second) about trying to be more efficient about
processing overseer messages as the overseer can become a bottleneck, especially with very
large numbers of replicas in a cluster. One of the approaches mentioned near the end of SOLR-5872
(15-Mar) was to "read large no:of items say 10000. put them into in memory buckets and feed
them into overseer....".
> This JIRA is to break out that part of the discussion as it might be an easy win whereas
"eliminating the Overseer queue" would be quite an undertaking.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message