lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ted Dunning (Commented) (JIRA)" <>
Subject [jira] [Commented] (SOLR-2765) Shard/Node states
Date Sun, 09 Oct 2011 17:33:29 GMT


Ted Dunning commented on SOLR-2765:

{We don't actually currently remove info about a shard when it goes away}
That is fine.

But there is a strong distinction between different information.  I see three kinds of information
about a shard that live in different places and are intended for different audiences.

- the content and status of collections.  This is intended as control input for the overseer
and contains information about which collections exist and what partitions they contain and
other policy information such as requested replication level.  This is the information that
should not be deleted

- the shard assignments for nodes.  This is generated by the overseer and intended for the
nodes.  When a node goes down, the assignments for that node *should* be deleted and other
nodes should get other assignments.

- the cluster state.  This is generated by the nodes and the overseer and read by the search
clients to let them know who to send queries to.  This is the file that I was referring to.
 I was suggesting that the clauses that state that a particular node is serving requests for
a particular shard should be deleted by the overseer when the node disappears.  I can't imagine
that there is any question about this because the clients have an urgent need to stop sending
queries to downed nodes as soon as possible.  Deleting a clause from this state doesn't forget
about the shard; it just gives an accurate picture of who is serving a shard.

I really think that separating the information like this makes it much simpler to keep track
of what is going on, especially for the clients.  They should only need to look one place
for what they need to know.  Besides, they can't really reconcile things correctly.  If shards
A,B,C were all served by node 3, then when node 3 goes down, the client can get things right.
 But if node 3 comes back and reloads A but not yet B and C, the client cannot know what the
current state is unless somebody updates the state correctly.

In my view, what should happen when node 3 goes down and then back up is this:

- the ephemeral for node 3 in live_nodes disappears

- the overseer removes the "node 3 has A, B, C" clause from the cluster state

- the overseer assigns shards A, B and C to other nodes to start the replication process

- node 6 registers that it is serving shard C

- node 5 registers that it is serving shard B

- node 3 comes back, creates the ephemeral in live_nodes and starts updating the indexes it
has from the transaction log.

- node 3 registers that it has shards A and B up-to-date because there have been no changes.

- the overseer looks at the cluster state and decides that node 3 is under-utilized and shard
B is over replicated.  It unassigns shard B from node 5 and assigns shard D to node 3.

- node 5 updates the cluster state to indicate it is no longer serving shard B

- node 3 downloads a copy of shard D and replays the log for D to catch up the archived index.

- node 3 registers that it is serving D

and so on.

The sequence of states for node 3 in this scenario is

- serving A,B,C
- down (not serving anything)
- up (not serving anything)
- serving A and B
- serving A, B and D

The client cannot derive this information accurately from simple liveness information.  Therefore
the node has to update the cluster state.

> Shard/Node states
> -----------------
>                 Key: SOLR-2765
>                 URL:
>             Project: Solr
>          Issue Type: Sub-task
>          Components: SolrCloud, update
>            Reporter: Yonik Seeley
>             Fix For: 4.0
>         Attachments: combined.patch, incremental_update.patch, scheduled_executors.patch,
> Need state for shards that indicate they are recovering, active/enabled, or disabled.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message