cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shao-Chuan Wang (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-6565) New node refuses to join the ring.
Date Sat, 11 Jan 2014 07:12:52 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-6565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13868693#comment-13868693
] 

Shao-Chuan Wang edited comment on CASSANDRA-6565 at 1/11/14 7:11 AM:
---------------------------------------------------------------------

Noted that we have replication factor 3 in both data centers. So ideally, we have 6 replicas.
Also, we saw from *nodetool status*, some of nodes appear to be down but they are not. Could
it because the replicas owners are too busy or network is too slow such that it only saw one
available replica when this brand new node was trying to join. Noted that by retrying a lot
of times of adding this node node, it finally joined the ring. But the joining new nodes have
become a very painful process for us. 


was (Author: shaochuan):
Noted that we have replication factor 3 in both data centers. So ideally, we have 6 replicas.
Also, we saw from nodetool status, some of nodes appear to be down but they are not. Could
it because the replicas owners are too busy or network is too slow such that it only saw one
available replica when this brand new node was trying to join. Noted that by retrying a lot
of times of adding this node node, it finally joined the ring. But the joining new nodes have
become a very painful process for us. 

> New node refuses to join the ring.
> ----------------------------------
>
>                 Key: CASSANDRA-6565
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6565
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Shao-Chuan Wang
>
> We have 30 nodes in one DC, 25 nodes in another. We are running 2.0.1.
> Two nodes are joining the ring, but one of them failed
> ARN [STREAM-IN-/10.4.197.53] 2014-01-09 19:41:40,418 StreamResultFuture.java (line 209)
[Stream #e515d6e0-795d-11e3-b74a-b72892248056] Stream failed
> ERROR [main] 2014-01-09 19:41:40,418 CassandraDaemon.java (line 459) Exception encountered
during startup
> java.lang.RuntimeException: Error during boostrap: Stream failed
>         at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:86)
>         at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:901)
>         at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:670)
>         at org.apache.cassandra.service.StorageService.initServer(StorageService.java:529)
>         at org.apache.cassandra.service.StorageService.initServer(StorageService.java:428)
>         at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:343)
>         at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:442)
>         at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485)
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
>         at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:210)
>         at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:185)
>         at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:321)
>         at org.apache.cassandra.streaming.StreamSession.complete(StreamSession.java:501)
       at org.apache.cassandra.streaming.StreamSession.messageReceived(Stre
> amSession.java:376)        at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:293)
>         at java.lang.Thread.run(Thread.java:744)
> ERROR [StorageServiceShutdownHook] 2014-01-09 19:41:40,428 CassandraDaemon.java (line
185) Exception in thread Thread[StorageServiceShutdownHook,5,main]
> java.lang.NullPointerException
>         at org.apache.cassandra.service.StorageService.stopRPCServer(StorageService.java:312)
>         at org.apache.cassandra.service.StorageService.shutdownClientServers(StorageService.java:361)
>         at org.apache.cassandra.service.StorageService.access$000(StorageService.java:96)
>         at org.apache.cassandra.service.StorageService$1.runMayThrow(StorageService.java:494)
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>         at java.lang.Thread.run(Thread.java:744)



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message