lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pushkar Raste (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SOLR-9689) Process updates concurrently during PeerSync
Date Tue, 15 Nov 2016 20:23:58 GMT

     [ https://issues.apache.org/jira/browse/SOLR-9689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Pushkar Raste updated SOLR-9689:
--------------------------------
    Attachment: parallelize-peersync.patch

Attached working patch. 
For my tests I didn't see much improvement (in fact in some cases performance degraded) with
parallelization. I could not find any hotspot in the profile.

My theory is documents in test are so shorts and simple, that although parallelizing is working
functionally, we need to test this with more complex documents and verify performance gains.


Most of the parallelization parameters would be subjective and people need to verify which
ones work better for them.

It also seems performance would suffer if there are relatively high DBQs to  applied during
DBQs, since updates are applied out of order.

> Process updates concurrently during PeerSync
> --------------------------------------------
>
>                 Key: SOLR-9689
>                 URL: https://issues.apache.org/jira/browse/SOLR-9689
>             Project: Solr
>          Issue Type: Improvement
>      Security Level: Public(Default Security Level. Issues are Public) 
>            Reporter: Pushkar Raste
>         Attachments: SOLR-9689.patch, SOLR-9689.patch2, parallelize-peersync.patch
>
>
> This came up during discussion with [~shalinmangar]
> During {{PeerSync}}, updates are applied one a time by looping through the updates received
from the leader. This is slow and could keep node in recovery for a long time if number of
updates to apply were large. 
> We can apply updates concurrently, this should be no different than what could happen
during normal indexing (we can't really ensure that a replica will process updates in the
same order as the leader or other replicas).
> There are few corner cases around dbq we should be careful about. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message