cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Victor Trac (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CASSANDRA-11982) Cassandra 3.5 cluster join fails
Date Wed, 08 Jun 2016 21:43:20 GMT
Victor Trac created CASSANDRA-11982:
---------------------------------------

             Summary: Cassandra 3.5 cluster join fails
                 Key: CASSANDRA-11982
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-11982
             Project: Cassandra
          Issue Type: Bug
         Environment: # java -version
openjdk version "1.8.0_91"
OpenJDK Runtime Environment (build 1.8.0_91-b14)
OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)

# uname -a
Linux cassandra-dfs-10-10-160-19 4.4.11-23.53.amzn1.x86_64 #1 SMP Wed Jun 1 22:22:50 UTC 2016
x86_64 x86_64 x86_64 GNU/Linux

# rpm -qa datastax-ddc
datastax-ddc-3.5.0-1.noarch

            Reporter: Victor Trac


In an effort to work around a failing upgradesstables/scrub (https://issues.apache.org/jira/browse/CASSANDRA-11981),
I force-removed a node from the cluster, deleted its data directory, and tried to join a node
to the cluster anew. After a few hours, the bootstrap thread failed:

cassandra.log
{code}
INFO  18:56:24 Redistributing index summaries
INFO  19:56:23 Saved KeyCache (17 items) in 386 ms
INFO  19:56:24 Redistributing index summaries
ERROR 20:40:27 [Stream #b1e4d290-2d91-11e6-8904-df8dbad02c2a] Remote peer 10.10.160.18 failed
stream session.
INFO  20:40:27 [Stream #b1e4d290-2d91-11e6-8904-df8dbad02c2a] Session with /10.10.160.18 is
complete
WARN  20:40:27 [Stream #b1e4d290-2d91-11e6-8904-df8dbad02c2a] Stream failed
ERROR 20:40:27 Error while waiting on bootstrap to complete. Bootstrap will have to be restarted.
java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException: Stream
failed
        at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
~[guava-18.0.jar:na]
        at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
~[guava-18.0.jar:na]
        at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-18.0.jar:na]
        at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1387)
[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:974)
[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:748)
[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:613)
[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:349) [apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:551)
[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:680) [apache-cassandra-3.5.0.jar:3.5.0]
Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
        at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) ~[guava-18.0.jar:na]
        at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
~[guava-18.0.jar:na]
        at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
~[guava-18.0.jar:na]
        at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
~[guava-18.0.jar:na]
        at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
~[guava-18.0.jar:na]
        at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:213)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:189)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:429)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.streaming.StreamSession.sessionFailed(StreamSession.java:636)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:489)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:274)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_91]
Jun 08, 2016 8:40:27 PM com.google.common.util.concurrent.ExecutionList executeListener
SEVERE: RuntimeException while executing runnable com.google.common.util.concurrent.Futures$6@3592d1b5
with executor INSTANCE
java.lang.NullPointerException
        at org.apache.cassandra.service.StorageService$2.onFailure(StorageService.java:1382)
        at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
        at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
        at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
        at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
        at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
        at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:213)
        at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:189)
        at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:429)
        at org.apache.cassandra.streaming.StreamSession.sessionFailed(StreamSession.java:636)
        at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:489)
        at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:274)
        at java.lang.Thread.run(Thread.java:745)

WARN  20:40:27 Some data streaming failed. Use nodetool to check bootstrap state and resume.
For more, see `nodetool help bootstrap`. IN_PROGRESS
INFO  20:40:27 Waiting for gossip to settle before accepting client requests...
INFO  20:40:35 No gossip backlog; proceeding
{code}

debug.log
{code}
DEBUG [GossipStage:1] 2016-06-08 20:40:27,370 FailureDetector.java:456 - Ignoring interval
time of 2001208211 for /10.240.0.4
DEBUG [STREAM-IN-/10.10.160.18] 2016-06-08 20:40:27,701 ConnectionHandler.java:269 - [Stream
#b1e4d290-2d91-11e6-8904-df8dbad02c2a] Received Session Failed
ERROR [STREAM-IN-/10.10.160.18] 2016-06-08 20:40:27,701 StreamSession.java:635 - [Stream #b1e4d290-2d91-11e6-8904-df8dbad02c2a]
Remote peer 10.10.160.18 failed stream session.
DEBUG [STREAM-IN-/10.10.160.18] 2016-06-08 20:40:27,708 ConnectionHandler.java:110 - [Stream
#b1e4d290-2d91-11e6-8904-df8dbad02c2a] Closing stream connection handler on /10.10.160.18
INFO  [STREAM-IN-/10.10.160.18] 2016-06-08 20:40:27,708 StreamResultFuture.java:185 - [Stream
#b1e4d290-2d91-11e6-8904-df8dbad02c2a] Session with /10.10.160.18 is complete
DEBUG [STREAM-IN-/10.10.160.18] 2016-06-08 20:40:27,709 StreamCoordinator.java:146 - Finished
connecting all sessions
WARN  [STREAM-IN-/10.10.160.18] 2016-06-08 20:40:27,710 StreamResultFuture.java:212 - [Stream
#b1e4d290-2d91-11e6-8904-df8dbad02c2a] Stream failed
ERROR [main] 2016-06-08 20:40:27,711 StorageService.java:1392 - Error while waiting on bootstrap
to complete. Bootstrap will have to be restarted.
java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException: Stream
failed
        at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
~[guava-18.0.jar:na]
        at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
~[guava-18.0.jar:na]
        at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-18.0.jar:na]
        at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1387)
[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:974)
[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:748)
[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:613)
[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:349) [apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:551)
[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:680) [apache-cassandra-3.5.0.jar:3.5.0]
Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
        at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) ~[guava-18.0.jar:na]
        at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
~[guava-18.0.jar:na]
        at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
~[guava-18.0.jar:na]
        at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
~[guava-18.0.jar:na]
        at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
~[guava-18.0.jar:na]
        at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:213)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:189)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:429)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.streaming.StreamSession.sessionFailed(StreamSession.java:636)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:489)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:274)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_91]
WARN  [main] 2016-06-08 20:40:27,777 StorageService.java:1029 - Some data streaming failed.
Use nodetool to check bootstrap state and resume. For more, see `nodetool help bootstrap`.
IN_PROGRESS
INFO  [main] 2016-06-08 20:40:27,779 CassandraDaemon.java:639 - Waiting for gossip to settle
before accepting client requests...
DEBUG [GossipStage:1] 2016-06-08 20:40:31,371 FailureDetector.java:456 - Ignoring interval
time of 2256483778 for /10.10.128.18
DEBUG [GossipStage:1] 2016-06-08 20:40:31,371 FailureDetector.java:456 - Ignoring interval
time of 2001387252 for /10.10.160.18
{code} 

system.log
{code}
INFO  [IndexSummaryManager:1] 2016-06-08 19:56:24,495 IndexSummaryRedistribution.java:74 -
Redistributing index summaries
ERROR [STREAM-IN-/10.10.160.18] 2016-06-08 20:40:27,701 StreamSession.java:635 - [Stream #b1e4d290-2d91-11e6-8904-df8dbad02c2a]
Remote peer 10.10.160.18 failed stream session.
INFO  [STREAM-IN-/10.10.160.18] 2016-06-08 20:40:27,708 StreamResultFuture.java:185 - [Stream
#b1e4d290-2d91-11e6-8904-df8dbad02c2a] Session with /10.10.160.18 is complete
WARN  [STREAM-IN-/10.10.160.18] 2016-06-08 20:40:27,710 StreamResultFuture.java:212 - [Stream
#b1e4d290-2d91-11e6-8904-df8dbad02c2a] Stream failed
ERROR [main] 2016-06-08 20:40:27,711 StorageService.java:1392 - Error while waiting on bootstrap
to complete. Bootstrap will have to be restarted.
java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException: Stream
failed
        at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
~[guava-18.0.jar:na]
        at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
~[guava-18.0.jar:na]
        at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-18.0.jar:na]
        at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1387)
[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:974)
[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:748)
[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:613)
[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:349) [apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:551)
[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:680) [apache-cassandra-3.5.0.jar:3.5.0]
Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
        at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) ~[guava-18.0.jar:na]
        at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
~[guava-18.0.jar:na]
        at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
~[guava-18.0.jar:na]
        at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
~[guava-18.0.jar:na]
        at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
~[guava-18.0.jar:na]
        at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:213)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:189)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:429)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.streaming.StreamSession.sessionFailed(StreamSession.java:636)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:489)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:274)
~[apache-cassandra-3.5.0.jar:3.5.0]
        at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_91]
WARN  [main] 2016-06-08 20:40:27,777 StorageService.java:1029 - Some data streaming failed.
Use nodetool to check bootstrap state and resume. For more, see `nodetool help bootstrap`.
IN_PROGRESS
INFO  [main] 2016-06-08 20:40:27,779 CassandraDaemon.java:639 - Waiting for gossip to settle
before accepting client requests...
INFO  [main] 2016-06-08 20:40:35,780 CassandraDaemon.java:670 - No gossip backlog; proceeding
INFO  [main] 2016-06-08 20:40:35,876 NativeTransportService.java:70 - Netty using native Epoll
event loop
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message