Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@lucene.apache.org
Date: Fri, 7 Oct 2011 21:46:30 +0000 (UTC)
From: "Ted Dunning (Commented) (JIRA)" <jira@apache.org>
To: dev@lucene.apache.org
Message-ID: 
 <181432166.10152.1318023990131.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <46795933.8937.1297370218140.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (SOLR-2358) Distributing Indexing
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/SOLR-2358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123231#comment-13123231 ] 

Ted Dunning commented on SOLR-2358:
-----------------------------------

Mark,

How do you handle failure scenarios?

The failures I am curious about are:

- the leader fails, but a transaction is still sent to it because the client didn't get the memo in time

- the leader fails but has already written a transaction locally without having a chance to forward it to the followers

- the leader fails after writing locally and to the replicas but before sending an ACK

- a replica is partitioned from the cluster, a transaction is received and committed by all live replicas and then the failed index returns from the land of the living dead.

The bad behaviors that need to be avoided include

- document acked but not inserted

- document not acked, inserted again and two copies wind up in the index

- indexes diverge because some replicas commit a change while others do not

Two phase commit is not generally a viable solution for this in a cluster where failures can occur because it requires locks to be taken.  Once these locks are taken, the cluster cannot proceed until the locks are cleared and this cannot be done reliably in the presence of failures.

Zookeeper avoids this to a large degree by making updates idempotent before they are inserted into the update queue.  This means that if the updates are done more than once, most importantly during error recovery, that no error actually occurs.  This is what makes ZK able to take snapshots without stopping the world.  It does not entirely resolve the case of transactions that are committed but not acked.


> Distributing Indexing
> ---------------------
>
>                 Key: SOLR-2358
>                 URL: https://issues.apache.org/jira/browse/SOLR-2358
>             Project: Solr
>          Issue Type: New Feature
>          Components: SolrCloud, update
>            Reporter: William Mayor
>            Priority: Minor
>             Fix For: 4.0
>
>         Attachments: SOLR-2358.patch
>
>
> The first steps towards creating distributed indexing functionality in Solr

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org