lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ted Dunning (Commented) (JIRA)" <>
Subject [jira] [Commented] (SOLR-2358) Distributing Indexing
Date Fri, 07 Oct 2011 21:46:30 GMT


Ted Dunning commented on SOLR-2358:


How do you handle failure scenarios?

The failures I am curious about are:

- the leader fails, but a transaction is still sent to it because the client didn't get the
memo in time

- the leader fails but has already written a transaction locally without having a chance to
forward it to the followers

- the leader fails after writing locally and to the replicas but before sending an ACK

- a replica is partitioned from the cluster, a transaction is received and committed by all
live replicas and then the failed index returns from the land of the living dead.

The bad behaviors that need to be avoided include

- document acked but not inserted

- document not acked, inserted again and two copies wind up in the index

- indexes diverge because some replicas commit a change while others do not

Two phase commit is not generally a viable solution for this in a cluster where failures can
occur because it requires locks to be taken.  Once these locks are taken, the cluster cannot
proceed until the locks are cleared and this cannot be done reliably in the presence of failures.

Zookeeper avoids this to a large degree by making updates idempotent before they are inserted
into the update queue.  This means that if the updates are done more than once, most importantly
during error recovery, that no error actually occurs.  This is what makes ZK able to take
snapshots without stopping the world.  It does not entirely resolve the case of transactions
that are committed but not acked.

> Distributing Indexing
> ---------------------
>                 Key: SOLR-2358
>                 URL:
>             Project: Solr
>          Issue Type: New Feature
>          Components: SolrCloud, update
>            Reporter: William Mayor
>            Priority: Minor
>             Fix For: 4.0
>         Attachments: SOLR-2358.patch
> The first steps towards creating distributed indexing functionality in Solr

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message