lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mark Miller (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SOLR-12338) Replay buffering tlog in parallel
Date Fri, 11 May 2018 05:14:00 GMT

    [ https://issues.apache.org/jira/browse/SOLR-12338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16471502#comment-16471502
] 

Mark Miller commented on SOLR-12338:
------------------------------------

bq. Yeah, but we do not need that flag for the case of LogReplayer, right? Because we are
calling execute method in single-thread.

Technically that sounds right, but I'm not sure I read the contract explicitly promises that.
If we have good testing, it's not much of a concern.

bq. OrderedExecutor ensuring that tasks are kicked off in order for a same id. Yeah, task1
get taken off the queue only after it finishes.

Yeah, so I don't think I spot an open issue for a race.

bq. I think we will throttle the incoming updates properly by doing SOLR-12305.

Ah right, had been looking at that issue recently too and had it on my mind. That is more
where that comment belongs. I was thinking these queues would work with documents coming in
and getting buffered, but they won't get held up from dropping off the document to the tlog.
But anyway, I think that natural throttling is a good first step. I think at the end of the
day, we will want to end up with a Filter though that can do QOS and intelligent throttling
based on data, but I'm pro whatever gets us out of infinite tlog replay soonest short term.



> Replay buffering tlog in parallel
> ---------------------------------
>
>                 Key: SOLR-12338
>                 URL: https://issues.apache.org/jira/browse/SOLR-12338
>             Project: Solr
>          Issue Type: Improvement
>      Security Level: Public(Default Security Level. Issues are Public) 
>            Reporter: Cao Manh Dat
>            Assignee: Cao Manh Dat
>            Priority: Major
>         Attachments: SOLR-12338.patch, SOLR-12338.patch
>
>
> Since updates with different id are independent, therefore it is safe to replay them
in parallel. This will significantly reduce recovering time of replicas in high load indexing
environment. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message