cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sam Overton (JIRA)" <>
Subject [jira] [Created] (CASSANDRA-2948) Nodetool move fails to stream out data from moved node to new endpoint.
Date Tue, 26 Jul 2011 15:43:11 GMT
Nodetool move fails to stream out data from moved node to new endpoint.

                 Key: CASSANDRA-2948
             Project: Cassandra
          Issue Type: Bug
          Components: Core
    Affects Versions: 0.8.2
            Reporter: Sam Overton
            Priority: Critical

When moving a node in the ring with nodetool move, that node streams its data to itself instead
of to the new endpoint responsible for its old range.

Steps to reproduce:
* Create a cluster (A,B,C,D) with tokens (0,4,8,C) using ByteOrderedPartitioner
* Create a keyspace and CF with RF=1
* Insert keys (2,6,A,E). This should put one key on each node.
* Move node A to token 7. This should cause:
  + node C streams key 6 to node A
  + node A streams key E to node B
  However instead, node A streams key E to itself.

Selected log messages from node A:
 INFO [RMI TCP Connection(6)-] 2011-07-26 16:29:17,075 (line
1878) Moving miles/ from Token(bytes[00]) to Token(bytes[07]).
DEBUG [RMI TCP Connection(6)-] 2011-07-26 16:29:17,080 (line
1941) Table ks: work map {/[(Token(bytes[04]),Token(bytes[07])]]}.
 INFO [RMI TCP Connection(6)-] 2011-07-26 16:29:17,080 (line
1946) Sleeping 30000 ms before start streaming/fetching ranges.
 INFO [RMI TCP Connection(6)-] 2011-07-26 16:29:46,728 (line
522) Moving: fetching new ranges and streaming old ranges
DEBUG [RMI TCP Connection(6)-] 2011-07-26 16:29:46,728 (line
1960) [Move->STREAMING] Work Map: {ks={(Token(bytes[0c]),Token(bytes[00])]=[miles/]}}
DEBUG [RMI TCP Connection(6)-] 2011-07-26 16:29:46,729 (line
1965) [Move->FETCHING] Work Map: {ks={/[(Token(bytes[04]),Token(bytes[07])]]}}
DEBUG [RMI TCP Connection(6)-] 2011-07-26 16:29:46,730 (line
2411) Requesting from / ranges (Token(bytes[04]),Token(bytes[07])]
 INFO [StreamStage:1] 2011-07-26 16:29:46,737 (line 90) Beginning transfer
to miles/
DEBUG [StreamStage:1] 2011-07-26 16:29:46,737 (line 91) Ranges are (Token(bytes[0c]),Token(bytes[00])]

This appears to be caused because in StorageService.move we call
    Gossiper.instance.addLocalApplicationState(ApplicationState.STATUS, valueFactory.moving(newToken));
and then get the new token metadata in order to calculate where the new endpoint is that we
should stream to
    TokenMetadata tokenMetaClone = tokenMetadata_.cloneAfterAllSettled();
however, in addLocalApplicationState there is no notification broadcast for the change in
local state, so tokenMetadata_ never updates the list of moving nodes, and the tokenMetaClone
is still the state of the ring from before the move.

This message is automatically generated by JIRA.
For more information on JIRA, see:


View raw message