cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ravi Prasad (JIRA)" <>
Subject [jira] [Created] (CASSANDRA-6622) Streaming session failures during node replace using replace_address
Date Sun, 26 Jan 2014 22:25:39 GMT
Ravi Prasad created CASSANDRA-6622:

             Summary: Streaming session failures during node replace using replace_address
                 Key: CASSANDRA-6622
             Project: Cassandra
          Issue Type: Bug
         Environment: RHEL6, cassandra-2.0.4
            Reporter: Ravi Prasad

When using replace_address, Gossiper ApplicationState is set to hibernate, which is a down
state. We are seeing that the peer nodes are seeing streaming plan request even before the
Gossiper on them marks the replacing node as dead. As a result, streaming on peer nodes convicts
the replacing node by closing the stream handler.  
I think, making the StorageService thread on the replacing node, sleep for BROADCAST_INTERVAL
before bootstrapping, would avoid this scenario.

Relevant logs from peer node (see that the Gossiper on peer node mark the replacing node as
down, 2 secs after  the streaming init request):

 INFO [STREAM-INIT-/x.x.x.x:46436] 2014-01-26 20:42:24,388 (line 116)
[Stream #5c6cd940-86ca-11e3-90a0-411b913c0e88] Received streaming plan for Bootstrap
 INFO [GossipTasks:1] 2014-01-26 20:42:25,240 (line 181) [Stream #5c6cd940-86ca-11e3-90a0-411b913c0e88]
Session with /x.x.x.x is complete
 WARN [GossipTasks:1] 2014-01-26 20:42:25,240 (line 210) [Stream #5c6cd940-86ca-11e3-90a0-411b913c0e88]
Stream failed
 INFO [GossipStage:1] 2014-01-26 20:42:25,242 (line 850) InetAddress /x.x.x.x
is now DOWN
ERROR [STREAM-IN-/x.x.x.x] 2014-01-26 20:42:25,766 (line 410) [Stream #5c6cd940-86ca-11e3-90a0-411b913c0e88]
Streaming error occurred
java.lang.RuntimeException: Outgoing stream handler has been closed
        at org.apache.cassandra.streaming.ConnectionHandler.sendMessage(
        at org.apache.cassandra.streaming.StreamSession.prepare(
        at org.apache.cassandra.streaming.StreamSession.messageReceived(
        at org.apache.cassandra.streaming.ConnectionHandler$
 INFO [STREAM-IN-/x.x.x.x] 2014-01-26 20:42:25,768 (line 181) [Stream
#5c6cd940-86ca-11e3-90a0-411b913c0e88] Session with /x.x.x.x is complete
 WARN [STREAM-IN-/x.x.x.x] 2014-01-26 20:42:25,768 (line 210) [Stream
#5c6cd940-86ca-11e3-90a0-411b913c0e88] Stream failed

This message was sent by Atlassian JIRA

View raw message