lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cao Manh Dat (JIRA)" <>
Subject [jira] [Commented] (SOLR-12338) Replay buffering tlog in parallel
Date Fri, 11 May 2018 05:03:00 GMT


Cao Manh Dat commented on SOLR-12338:

Given that some machines these days have dozens of cores and you might have many SolrCores
recovering, we may want to cap the number of threads at some number or make it configurable
or something.
{{replayUpdatesExecutor }] is shared through all the SolrCores, therefore how many SolrCores
are recovering won't affect the max number of threads will be used. Although, make it configurable
is a good idea.

Yeah, you need that to ensure FIFO.
Yeah, but we do not need that flag for the case of LogReplayer, right? Because we are calling
execute method in single-thread.

I think what David is getting at is that you are ensuring that tasks are kicked off in order,
but once they are kicked off, you can't guarantee order. So task1 gets taken off the queue,
then task 2 is taken, now task 2 gets executed first when task 1 has it's thread unluckily
scheduled by the OS. At least that's how I read it. But that is not an issue right? Because
you don't run an item from the queue until the one in front of it is fully run right?
OrderedExecutor ensuring that tasks are kicked off in order *for a same id*. Yeah, task1 get
taken off the queue only after it finishes.

I like how this gives us some control to throttle, I wonder how efficient it is as documents
keep thundering in though - do we gobble up threads and connections waiting? That is where
it's a bummer it's hard to limit those resources. What are you going to do though? Those requests
have to wait somewhere or we have to start dropping them - and hopefully with NIO2 it's somewhat
efficient to wait on IO.
I think we will throttle the incoming updates properly by doing SOLR-12305. 

> Replay buffering tlog in parallel
> ---------------------------------
>                 Key: SOLR-12338
>                 URL:
>             Project: Solr
>          Issue Type: Improvement
>      Security Level: Public(Default Security Level. Issues are Public) 
>            Reporter: Cao Manh Dat
>            Assignee: Cao Manh Dat
>            Priority: Major
>         Attachments: SOLR-12338.patch, SOLR-12338.patch
> Since updates with different id are independent, therefore it is safe to replay them
in parallel. This will significantly reduce recovering time of replicas in high load indexing

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message