lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cao Manh Dat (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SOLR-9835) Create another replication mode for SolrCloud
Date Thu, 12 Jan 2017 03:50:16 GMT

     [ https://issues.apache.org/jira/browse/SOLR-9835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Cao Manh Dat updated SOLR-9835:
-------------------------------
    Attachment: SOLR-9835.patch

Updated patch for this issue :
- Remove {{UpdateCommand.NOTICE_INDEXWRITER}} flag by not replaying buffer log if the updates
is not made to IW. This changes help the patch more compact
- When replica recover, 
	- replica won't apply buffer updates
	- fix bug: RTG support for buffering updates ( along with test )
- {{ZkController.replicateFromLeaders}} is thread-safe now
- When cluster restart, only leaders should call {{ulog.recoverFromLog()}}, tested by TestCloudRecovery
- {{OnlyLeaderIndexesTest}}
	- test for RTG
	- test for replicas are consistent
	- more test for replica recovery
- More tests and fixed bugs

I will try to write more tests for all possible cases during Replica Recovery and Leader Election.
Hoping that I can find more bugs by doing this.

> Create another replication mode for SolrCloud
> ---------------------------------------------
>
>                 Key: SOLR-9835
>                 URL: https://issues.apache.org/jira/browse/SOLR-9835
>             Project: Solr
>          Issue Type: Bug
>      Security Level: Public(Default Security Level. Issues are Public) 
>            Reporter: Cao Manh Dat
>            Assignee: Shalin Shekhar Mangar
>         Attachments: SOLR-9835.patch, SOLR-9835.patch, SOLR-9835.patch, SOLR-9835.patch,
SOLR-9835.patch, SOLR-9835.patch, SOLR-9835.patch
>
>
> The current replication mechanism of SolrCloud is called state machine, which replicas
start in same initial state and for each input, the input is distributed across replicas so
all replicas will end up with same next state. 
> But this type of replication have some drawbacks
> - The commit (which costly) have to run on all replicas
> - Slow recovery, because if replica miss more than N updates on its down time, the replica
have to download entire index from its leader.
> So we create create another replication mode for SolrCloud called state transfer, which
acts like master/slave replication. In basically
> - Leader distribute the update to other replicas, but the leader only apply the update
to IW, other replicas just store the update to UpdateLog (act like replication).
> - Replicas frequently polling the latest segments from leader.
> Pros:
> - Lightweight for indexing, because only leader are running the commit, updates.
> - Very fast recovery, replicas just have to download the missing segments.
> To use this new replication mode, a new collection must be created with an additional
parameter {{liveReplicas=1}}
> {code}
> http://localhost:8983/solr/admin/collections?action=CREATE&name=newCollection&numShards=2&replicationFactor=1&liveReplicas=1
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message