Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Date: Thu, 3 Jul 2014 07:39:24 +0000 (UTC)
From: "Dmitry Mitko (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: <JIRA.12688122.1389297516555.95503.1404373164518@arcas>
In-Reply-To: <JIRA.12688122.1389297516555@arcas>
References: <JIRA.12688122.1389297516555@arcas>
Subject: [jira] [Comment Edited] (CASSANDRA-6565) New node refuses to join
 the ring.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/CASSANDRA-6565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14051148#comment-14051148 ] 

Dmitry Mitko edited comment on CASSANDRA-6565 at 7/3/14 7:38 AM:
-----------------------------------------------------------------

Sure, it is true. I've checked it then and have found, that source node has corrupted SSTable, that can't be streamed successfully.

Right now, I've fixed that problem - to remove corrupted SSTable. I saw similar problems before in the nodes' logs. Tried to fix it using cleanup and other nodetool modes - pity, but corrupted SSTables wasn't fixed or repaired.

Can you advice some nice solution for them?

For example, if we have LeveledCompaction mode in the CF settings, how can we remove all of SSTables, that belong to the same group?

P.S.: It will be so good, if failed to join node will write a lot of useful info into logs, such as: "Probably, you have corrupted SSTable(s) in your source nodes, please, remove them before joining".


was (Author: dv1983):
Sure, it is true. I've checked it then and have found, that source node has corrupted SSTable, that can't be streamed successfully.

Right now, I've fixed that problem - to remove corrupted SSTable. I saw similar problems before in the nodes' logs. Tried to fix it using cleanup and other nodetool modes - pity, but corrupted SSTables wasn't fixed or repaired.

Can you advice some nice solution for them?

For example, if we have LeveledCompaction mode in the CF settings, how can we remove all of SSTables, that belong to the same group?

> New node refuses to join the ring.
> ----------------------------------
>
>                 Key: CASSANDRA-6565
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6565
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Shao-Chuan Wang
>
> We have 30 nodes in one DC, 25 nodes in another. We are running 2.0.1.
> Two nodes are joining the ring, but one of them failed
> ARN [STREAM-IN-/10.4.197.53] 2014-01-09 19:41:40,418 StreamResultFuture.java (line 209) [Stream #e515d6e0-795d-11e3-b74a-b72892248056] Stream failed
> ERROR [main] 2014-01-09 19:41:40,418 CassandraDaemon.java (line 459) Exception encountered during startup
> java.lang.RuntimeException: Error during boostrap: Stream failed
>         at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:86)
>         at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:901)
>         at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:670)
>         at org.apache.cassandra.service.StorageService.initServer(StorageService.java:529)
>         at org.apache.cassandra.service.StorageService.initServer(StorageService.java:428)
>         at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:343)
>         at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:442)
>         at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485)
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
>         at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:210)
>         at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:185)
>         at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:321)
>         at org.apache.cassandra.streaming.StreamSession.complete(StreamSession.java:501)        at org.apache.cassandra.streaming.StreamSession.messageReceived(Stre
> amSession.java:376)        at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:293)
>         at java.lang.Thread.run(Thread.java:744)
> ERROR [StorageServiceShutdownHook] 2014-01-09 19:41:40,428 CassandraDaemon.java (line 185) Exception in thread Thread[StorageServiceShutdownHook,5,main]
> java.lang.NullPointerException
>         at org.apache.cassandra.service.StorageService.stopRPCServer(StorageService.java:312)
>         at org.apache.cassandra.service.StorageService.shutdownClientServers(StorageService.java:361)
>         at org.apache.cassandra.service.StorageService.access$000(StorageService.java:96)
>         at org.apache.cassandra.service.StorageService$1.runMayThrow(StorageService.java:494)
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>         at java.lang.Thread.run(Thread.java:744)


--
This message was sent by Atlassian JIRA
(v6.2#6252)