cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pas (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-6565) New node refuses to join the ring.
Date Wed, 12 Mar 2014 13:09:43 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-6565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13931732#comment-13931732
] 

Pas commented on CASSANDRA-6565:
--------------------------------

Hello!

Slightly similar issue, I didn't want to further pollute the issue tracker, so:

adding plus one node to a 2-node ring resulted in (all three on 2.0.6):

#  grep 192.168.36.171 system.log
{code}
 INFO [HANDSHAKE-/192.168.36.173] 2014-03-12 13:45:57,801 OutboundTcpConnection.java (line
386) Handshaking version with /192.168.36.173
 INFO [GossipStage:1] 2014-03-12 13:46:00,170 Gossiper.java (line 883) Node /192.168.36.173
is now part of the cluster
 INFO [HANDSHAKE-/192.168.36.173] 2014-03-12 13:46:00,172 OutboundTcpConnection.java (line
386) Handshaking version with /192.168.36.173
 INFO [RequestResponseStage:9] 2014-03-12 13:46:00,185 Gossiper.java (line 849) InetAddress
/192.168.36.173 is now UP
 INFO [STREAM-INIT-/192.168.36.173:34610] 2014-03-12 13:46:48,608 StreamResultFuture.java
(line 121) [Stream #2e0c6a10-a9e4-11e3-9c59-5340fa68c75f] Received streaming plan for Bootstrap
 INFO [STREAM-IN-/192.168.36.173] 2014-03-12 13:47:28,974 ColumnFamilyStore.java (line 785)
Enqueuing flush of Memtable-TStore@119522965(23215181/50011094 serialized/live bytes, 84365
ops)
 INFO [STREAM-IN-/192.168.36.173] 2014-03-12 13:47:52,156 StreamResultFuture.java (line 173)
[Stream #2e0c6a10-a9e4-11e3-9c59-5340fa68c75f] Prepare completed. Receiving 0 files(0 bytes),
sending 1971 files(6355271334 bytes)
ERROR [STREAM-OUT-/192.168.36.173] 2014-03-12 13:51:21,163 StreamSession.java (line 418) [Stream
#2e0c6a10-a9e4-11e3-9c59-5340fa68c75f] Streaming error occurred
 INFO [STREAM-OUT-/192.168.36.173] 2014-03-12 13:51:21,181 StreamResultFuture.java (line 186)
[Stream #2e0c6a10-a9e4-11e3-9c59-5340fa68c75f] Session with /192.168.36.173 is complete
 INFO [STREAM-IN-/192.168.36.173] 2014-03-12 13:51:21,181 StreamResultFuture.java (line 186)
[Stream #2e0c6a10-a9e4-11e3-9c59-5340fa68c75f] Session with /192.168.36.173 is complete
 WARN [STREAM-OUT-/192.168.36.173] 2014-03-12 13:51:21,184 StreamResultFuture.java (line 215)
[Stream #2e0c6a10-a9e4-11e3-9c59-5340fa68c75f] Stream failed
 WARN [STREAM-IN-/192.168.36.173] 2014-03-12 13:51:21,185 StreamResultFuture.java (line 215)
[Stream #2e0c6a10-a9e4-11e3-9c59-5340fa68c75f] Stream failed
ERROR [STREAM-OUT-/192.168.36.173] 2014-03-12 13:51:21,185 StreamSession.java (line 418) [Stream
#2e0c6a10-a9e4-11e3-9c59-5340fa68c75f] Streaming error occurred
[....]
 WARN [STREAM-OUT-/192.168.36.173] 2014-03-12 13:51:21,461 StreamResultFuture.java (line 215)
[Stream #2e0c6a10-a9e4-11e3-9c59-5340fa68c75f] Stream failed
ERROR [STREAM-OUT-/192.168.36.173] 2014-03-12 13:51:21,462 StreamSession.java (line 418) [Stream
#2e0c6a10-a9e4-11e3-9c59-5340fa68c75f] Streaming error occurred
 INFO [STREAM-OUT-/192.168.36.173] 2014-03-12 13:51:21,462 StreamResultFuture.java (line 186)
[Stream #2e0c6a10-a9e4-11e3-9c59-5340fa68c75f] Session with /192.168.36.173 is complete
 WARN [STREAM-OUT-/192.168.36.173] 2014-03-12 13:51:21,462 StreamResultFuture.java (line 215)
[Stream #2e0c6a10-a9e4-11e3-9c59-5340fa68c75f] Stream failed
 INFO [GossipStage:1] 2014-03-12 13:51:23,158 Gossiper.java (line 863) InetAddress /192.168.36.173
is now DOWN
 INFO [GossipTasks:1] 2014-03-12 13:51:51,522 Gossiper.java (line 633) FatClient /192.168.36.173
has been silent for 30000ms, removing from gossip
{code}

And there is always a Broken Pipe IOException after Streaming error occured
{code}
java.io.IOException: Broken pipe
        at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
        at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
        at sun.nio.ch.IOUtil.write(IOUtil.java:65)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487)
        at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:44)
        at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:383)
        at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:363)
        at java.lang.Thread.run(Thread.java:744)
{code}



Whereas on 192.168.36.173 (the new node)

{code}
 INFO [main] 2014-03-12 13:45:24,140 StorageService.java (line 985) JOINING: Starting to bootstrap...
 INFO [main] 2014-03-12 13:45:32,319 StreamResultFuture.java (line 87) [Stream #2e0c6a10-a9e4-11e3-9c59-5340fa68c75f]
Executing streaming plan for Bootstrap
 INFO [main] 2014-03-12 13:45:32,319 StreamResultFuture.java (line 91) [Stream #2e0c6a10-a9e4-11e3-9c59-5340fa68c75f]
Beginning stream session with /192.168.36.172
 INFO [main] 2014-03-12 13:45:32,319 StreamResultFuture.java (line 91) [Stream #2e0c6a10-a9e4-11e3-9c59-5340fa68c75f]
Beginning stream session with /192.168.36.171
 INFO [STREAM-IN-/192.168.36.172] 2014-03-12 13:45:32,370 StreamResultFuture.java (line 173)
[Stream #2e0c6a10-a9e4-11e3-9c59-5340fa68c75f] Prepare completed. Receiving 1 files(64 bytes),
send
ing 0 files(0 bytes)
[...]
 WARN [STREAM-IN-/192.168.36.171] 2014-03-12 13:50:04,882 StreamSession.java (line 532) [Stream
#2e0c6a10-a9e4-11e3-9c59-5340fa68c75f] Retrying for following error
java.lang.NullPointerException
        at org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:63)
        at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:47)
        at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:37)
        at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:55)
        at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:283)
        at java.lang.Thread.run(Thread.java:744)
ERROR [STREAM-IN-/192.168.36.171] 2014-03-12 13:50:04,944 StreamSession.java (line 418) [Stream
#2e0c6a10-a9e4-11e3-9c59-5340fa68c75f] Streaming error occurred
java.lang.IllegalArgumentException: Unknown type -128
        at org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:89)
        at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:54)
        at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:283)
        at java.lang.Thread.run(Thread.java:744)
 INFO [STREAM-IN-/192.168.36.171] 2014-03-12 13:50:04,946 StreamResultFuture.java (line 186)
[Stream #2e0c6a10-a9e4-11e3-9c59-5340fa68c75f] Session with /192.168.36.171 is complete
 WARN [STREAM-IN-/192.168.36.171] 2014-03-12 13:50:04,947 StreamResultFuture.java (line 215)
[Stream #2e0c6a10-a9e4-11e3-9c59-5340fa68c75f] Stream failed
ERROR [main] 2014-03-12 13:50:04,947 CassandraDaemon.java (line 488) Exception encountered
during startup
java.lang.RuntimeException: Error during boostrap: Stream failed
        at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:86)
        at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1013)
        at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:775)
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:610)
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:501)
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:353)
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:471)
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:560)
Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
        at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
        at com.google.common.util.concurrent.Futures$4.run(Futures.java:1160)
        at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
        at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
        at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
        at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
        at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:216)
        at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:191)
        at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:329)
        at org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:423)
        at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:300)
        at java.lang.Thread.run(Thread.java:744)
 INFO [StorageServiceShutdownHook] 2014-03-12 13:50:04,955 Gossiper.java (line 1251) Announcing
shutdown
 INFO [StorageServiceShutdownHook] 2014-03-12 13:50:06,955 MessagingService.java (line 667)
Waiting for messaging service to quiesce
 INFO [ACCEPT-/192.168.36.173] 2014-03-12 13:50:06,956 MessagingService.java (line 907) MessagingService
has terminated the accept() thread
{code}

Thanks.

> New node refuses to join the ring.
> ----------------------------------
>
>                 Key: CASSANDRA-6565
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6565
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Shao-Chuan Wang
>
> We have 30 nodes in one DC, 25 nodes in another. We are running 2.0.1.
> Two nodes are joining the ring, but one of them failed
> ARN [STREAM-IN-/10.4.197.53] 2014-01-09 19:41:40,418 StreamResultFuture.java (line 209)
[Stream #e515d6e0-795d-11e3-b74a-b72892248056] Stream failed
> ERROR [main] 2014-01-09 19:41:40,418 CassandraDaemon.java (line 459) Exception encountered
during startup
> java.lang.RuntimeException: Error during boostrap: Stream failed
>         at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:86)
>         at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:901)
>         at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:670)
>         at org.apache.cassandra.service.StorageService.initServer(StorageService.java:529)
>         at org.apache.cassandra.service.StorageService.initServer(StorageService.java:428)
>         at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:343)
>         at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:442)
>         at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485)
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
>         at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:210)
>         at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:185)
>         at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:321)
>         at org.apache.cassandra.streaming.StreamSession.complete(StreamSession.java:501)
       at org.apache.cassandra.streaming.StreamSession.messageReceived(Stre
> amSession.java:376)        at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:293)
>         at java.lang.Thread.run(Thread.java:744)
> ERROR [StorageServiceShutdownHook] 2014-01-09 19:41:40,428 CassandraDaemon.java (line
185) Exception in thread Thread[StorageServiceShutdownHook,5,main]
> java.lang.NullPointerException
>         at org.apache.cassandra.service.StorageService.stopRPCServer(StorageService.java:312)
>         at org.apache.cassandra.service.StorageService.shutdownClientServers(StorageService.java:361)
>         at org.apache.cassandra.service.StorageService.access$000(StorageService.java:96)
>         at org.apache.cassandra.service.StorageService$1.runMayThrow(StorageService.java:494)
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>         at java.lang.Thread.run(Thread.java:744)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message