cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joshua McKenzie (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-10927) Stream failed during bootstrap
Date Fri, 08 Jan 2016 16:11:40 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-10927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Joshua McKenzie updated CASSANDRA-10927:
----------------------------------------
    Assignee: Paulo Motta

> Stream failed during bootstrap
> ------------------------------
>
>                 Key: CASSANDRA-10927
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10927
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Streaming and Messaging
>         Environment: CentOS 7 x64, Java 1.8.0_65
>            Reporter: Kai Wang
>            Assignee: Paulo Motta
>
> When I start the new node, I got this error:
> {noformat}
> ERROR [STREAM-IN-/192.168.0.10] 2015-12-22 15:37:56,302 StreamSession.java:524 - [Stream
#bfc4e100-a8eb-11e5-bec5-67d8099a8b91] Streaming error occurred
> java.nio.channels.ClosedChannelException: null
> 	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(Unknown Source) ~[na:1.8.0_65]
> 	at sun.nio.ch.SocketChannelImpl.read(Unknown Source) ~[na:1.8.0_65]
> 	at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:53)
~[apache-cassandra-2.2.4.jar:2.2.4]
> 	at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
~[apache-cassandra-2.2.4.jar:2.2.4]
> 	at java.lang.Thread.run(Unknown Source) [na:1.8.0_65]
> ERROR [Thread-22] 2015-12-22 15:37:56,302 CassandraDaemon.java:185 - Exception in thread
Thread[Thread-22,5,main]
> java.lang.RuntimeException: java.lang.InterruptedException
> 	at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.jar:na]
> 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) ~[apache-cassandra-2.2.4.jar:2.2.4]
> 	at java.lang.Thread.run(Unknown Source) ~[na:1.8.0_65]
> Caused by: java.lang.InterruptedException: null
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(Unknown
Source) ~[na:1.8.0_65]
> 	at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(Unknown Source) ~[na:1.8.0_65]
> 	at java.util.concurrent.ArrayBlockingQueue.put(Unknown Source) ~[na:1.8.0_65]
> 	at org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:176)
~[apache-cassandra-2.2.4.jar:2.2.4]
> 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.2.4.jar:2.2.4]
> 	... 1 common frames omitted
> INFO  [STREAM-IN-/192.168.0.10] 2015-12-22 15:37:56,345 StreamResultFuture.java:182 -
[Stream #bfc4e100-a8eb-11e5-bec5-67d8099a8b91] Session with /192.168.0.10 is complete
> WARN  [STREAM-IN-/192.168.0.10] 2015-12-22 15:37:56,346 StreamResultFuture.java:209 -
[Stream #bfc4e100-a8eb-11e5-bec5-67d8099a8b91] Stream failed
> ERROR [main] 2015-12-22 15:37:56,347 StorageService.java:1245 - Error while waiting on
bootstrap to complete. Bootstrap will have to be restarted.
> java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException:
Stream failed
> 	at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
~[guava-16.0.jar:na]
> 	at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
~[guava-16.0.jar:na]
> 	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-16.0.jar:na]
> 	at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1240) [apache-cassandra-2.2.4.jar:2.2.4]
> 	at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:920)
[apache-cassandra-2.2.4.jar:2.2.4]
> 	at org.apache.cassandra.service.StorageService.initServer(StorageService.java:705) [apache-cassandra-2.2.4.jar:2.2.4]
> 	at org.apache.cassandra.service.StorageService.initServer(StorageService.java:581) [apache-cassandra-2.2.4.jar:2.2.4]
> 	at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:315) [apache-cassandra-2.2.4.jar:2.2.4]
> 	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:529) [apache-cassandra-2.2.4.jar:2.2.4]
> 	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:638) [apache-cassandra-2.2.4.jar:2.2.4]
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
> 	at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
~[apache-cassandra-2.2.4.jar:2.2.4]
> 	at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172) ~[guava-16.0.jar:na]
> 	at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
~[guava-16.0.jar:na]
> 	at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
~[guava-16.0.jar:na]
> 	at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) ~[guava-16.0.jar:na]
> 	at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
~[guava-16.0.jar:na]
> 	at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:210)
~[apache-cassandra-2.2.4.jar:2.2.4]
> 	at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:186)
~[apache-cassandra-2.2.4.jar:2.2.4]
> 	at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:434)
~[apache-cassandra-2.2.4.jar:2.2.4]
> 	at org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:529) ~[apache-cassandra-2.2.4.jar:2.2.4]
> 	at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:279)
~[apache-cassandra-2.2.4.jar:2.2.4]
> 	at java.lang.Thread.run(Unknown Source) ~[na:1.8.0_65]
> WARN  [main] 2015-12-22 15:37:56,353 StorageService.java:975 - Some data streaming failed.
Use nodetool to check bootstrap state and resume. For more, see `nodetool help bootstrap`.
IN_PROGRESS
> INFO  [main] 2015-12-22 15:37:56,354 CassandraDaemon.java:596 - Waiting for gossip to
settle before accepting client requests...
> INFO  [main] 2015-12-22 15:38:04,355 CassandraDaemon.java:628 - No gossip backlog; proceeding
> INFO  [main] 2015-12-22 15:38:04,411 Server.java:146 - Netty using native Epoll event
loop
> INFO  [main] 2015-12-22 15:38:04,442 Server.java:184 - Using Netty Version: [netty-buffer=netty-buffer-4.0.23.Final.208198c,
netty-codec=netty-codec-4.0.23.Final.208198c, netty-codec-http=netty-codec-http-4.0.23.Final.208198c,
netty-codec-socks=netty-codec-socks-4.0.23.Final.208198c, netty-common=netty-common-4.0.23.Final.208198c,
netty-handler=netty-handler-4.0.23.Final.208198c, netty-transport=netty-transport-4.0.23.Final.208198c,
netty-transport-rxtx=netty-transport-rxtx-4.0.23.Final.208198c, netty-transport-sctp=netty-transport-sctp-4.0.23.Final.208198c,
netty-transport-udt=netty-transport-udt-4.0.23.Final.208198c]
> INFO  [main] 2015-12-22 15:38:04,443 Server.java:185 - Starting listening for CQL clients
on /192.168.0.12:9042...
> INFO  [main] 2015-12-22 15:38:04,495 ThriftServer.java:119 - Binding thrift service to
/192.168.0.12:9160
> INFO  [Thread-23] 2015-12-22 15:38:04,501 ThriftServer.java:136 - Listening for thrift
clients...
> ERROR [SharedPool-Worker-1] 2015-12-22 15:38:19,963 Message.java:611 - Unexpected exception
during request; channel = [id: 0x03e85563, /192.168.0.10:55681 => /192.168.0.12:9042]
> java.lang.NullPointerException: null
> 	at org.apache.cassandra.auth.PasswordAuthenticator.doAuthenticate(PasswordAuthenticator.java:143)
~[apache-cassandra-2.2.4.jar:2.2.4]
> 	at org.apache.cassandra.auth.PasswordAuthenticator.authenticate(PasswordAuthenticator.java:85)
~[apache-cassandra-2.2.4.jar:2.2.4]
> 	at org.apache.cassandra.auth.PasswordAuthenticator.access$100(PasswordAuthenticator.java:53)
~[apache-cassandra-2.2.4.jar:2.2.4]
> 	at org.apache.cassandra.auth.PasswordAuthenticator$PlainTextSaslAuthenticator.getAuthenticatedUser(PasswordAuthenticator.java:181)
~[apache-cassandra-2.2.4.jar:2.2.4]
> 	at org.apache.cassandra.transport.messages.AuthResponse.execute(AuthResponse.java:78)
~[apache-cassandra-2.2.4.jar:2.2.4]
> 	at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:507)
[apache-cassandra-2.2.4.jar:2.2.4]
> 	at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:401)
[apache-cassandra-2.2.4.jar:2.2.4]
> 	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
> 	at io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
> 	at io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [na:1.8.0_65]
> 	at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
[apache-cassandra-2.2.4.jar:2.2.4]
> 	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.2.4.jar:2.2.4]
> 	at java.lang.Thread.run(Unknown Source) [na:1.8.0_65]
> {noformat}
> I tried to resume bootstrap via nodetool a couple of times. It made some progress initially
(meaning some data was copied to the new node) but eventually all stopped with the same error.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message