cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Evans <john.eric.ev...@gmail.com>
Subject Re: Error while rebuilding a node: Stream failed
Date Fri, 27 May 2016 14:39:30 GMT
>From the various stacktraces in this thread, it's obvious you are
mixing versions 2.1.13 and 2.1.14.  Topology changes like this aren't
supported with mixed Cassandra versions.  Sometimes it will work,
sometimes it won't (and it will definitely not work in this instance).

You should either upgrade your 2.1.13 nodes to 2.1.14 first, or add
the new nodes using 2.1.13, and upgrade after.

On Fri, May 27, 2016 at 8:41 AM, George Sigletos <sigletos@textkernel.nl> wrote:

>>>> ERROR [STREAM-IN-/192.168.1.141] 2016-05-26 09:08:05,027
>>>> StreamSession.java:505 - [Stream #74c57bc0-231a-11e6-a698-1b05ac77baf9]
>>>> Streaming error occurred
>>>> java.lang.RuntimeException: Outgoing stream handler has been closed
>>>>         at
>>>> org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:138)
>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>>         at
>>>> org.apache.cassandra.streaming.StreamSession.receive(StreamSession.java:568)
>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>>         at
>>>> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:457)
>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>>         at
>>>> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:263)
>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>>         at java.lang.Thread.run(Unknown Source) [na:1.7.0_79]
>>>>
>>>> And this is from the source node:
>>>>
>>>> ERROR [STREAM-OUT-/172.31.22.104] 2016-05-26 11:08:05,097
>>>> StreamSession.java:505 - [Stream #74c57bc0-231a-11e6-a698-1b05ac77baf9]
>>>> Streaming error occurred
>>>> java.io.IOException: Broken pipe
>>>>         at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
>>>> ~[na:1.7.0_79]
>>>>         at sun.nio.ch.FileChannelImpl.transferToDirectly(Unknown Source)
>>>> ~[na:1.7.0_79]
>>>>         at sun.nio.ch.FileChannelImpl.transferTo(Unknown Source)
>>>> ~[na:1.7.0_79]
>>>>         at
>>>> org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:84)
>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>>         at
>>>> org.apache.cassandra.streaming.messages.OutgoingFileMessage.serialize(OutgoingFileMessage.java:88)
>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>>         at
>>>> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:49)
>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>>         at
>>>> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:41)
>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>>         at
>>>> org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:45)
>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>>         at
>>>> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:358)
>>>> [apache-cassandra-2.1.14.jar:2.1.14]
>>>>         at
>>>> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:330)
>>>> [apache-cassandra-2.1.14.jar:2.1.14]


>>>>>>>>>>> ERROR [STREAM-IN-/192.168.1.140] 2016-05-24 22:44:57,704
>>>>>>>>>>> StreamSession.java:620 - [Stream #2c290460-20d4-11e6-930f-1b05ac77baf9]
>>>>>>>>>>> Remote peer 192.168.1.140 failed stream session.
>>>>>>>>>>> ERROR [STREAM-OUT-/192.168.1.140] 2016-05-24
22:44:57,705
>>>>>>>>>>> StreamSession.java:505 - [Stream #2c290460-20d4-11e6-930f-1b05ac77baf9]
>>>>>>>>>>> Streaming error occurred
>>>>>>>>>>> java.io.IOException: Connection timed out
>>>>>>>>>>>         at sun.nio.ch.FileDispatcherImpl.write0(Native
Method)
>>>>>>>>>>> ~[na:1.7.0_79]
>>>>>>>>>>>         at sun.nio.ch.SocketDispatcher.write(Unknown
Source)
>>>>>>>>>>> ~[na:1.7.0_79]
>>>>>>>>>>>         at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown
>>>>>>>>>>> Source) ~[na:1.7.0_79]
>>>>>>>>>>>         at sun.nio.ch.IOUtil.write(Unknown Source)
~[na:1.7.0_79]
>>>>>>>>>>>         at sun.nio.ch.SocketChannelImpl.write(Unknown
Source)
>>>>>>>>>>> ~[na:1.7.0_79]
>>>>>>>>>>>         at
>>>>>>>>>>> org.apache.cassandra.io.util.DataOutputStreamAndChannel.write(DataOutputStreamAndChannel.java:48)
>>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13]
>>>>>>>>>>>         at
>>>>>>>>>>> org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:44)
>>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13]
>>>>>>>>>>>         at
>>>>>>>>>>> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:351)
>>>>>>>>>>> [apache-cassandra-2.1.13.jar:2.1.13]
>>>>>>>>>>>         at
>>>>>>>>>>> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:323)
>>>>>>>>>>> [apache-cassandra-2.1.13.jar:2.1.13]
>>>>>>>>>>>         at java.lang.Thread.run(Unknown Source)
[na:1.7.0_79]
>>>>>>>>>>> INFO  [STREAM-IN-/192.168.1.140] 2016-05-24 22:44:58,625
>>>>>>>>>>> StreamResultFuture.java:180 - [Stream #2c290460-20d4-11e6-930f-1b05ac77baf9]
>>>>>>>>>>> Session with /192.168.1.140 is complete
>>>>>>>>>>> WARN  [STREAM-IN-/192.168.1.140] 2016-05-24 22:44:58,627
>>>>>>>>>>> StreamResultFuture.java:207 - [Stream #2c290460-20d4-11e6-930f-1b05ac77baf9]
>>>>>>>>>>> Stream failed
>>>>>>>>>>> ERROR [RMI TCP Connection(24)-127.0.0.1] 2016-05-24
22:44:58,628
>>>>>>>>>>> StorageService.java:1075 - Error while rebuilding
node
>>>>>>>>>>> org.apache.cassandra.streaming.StreamException:
Stream failed
>>>>>>>>>>>         at
>>>>>>>>>>> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
>>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13]
>>>>>>>>>>>         at
>>>>>>>>>>> com.google.common.util.concurrent.Futures$4.run(Futures.java:1172)
>>>>>>>>>>> ~[guava-16.0.jar:na]
>>>>>>>>>>>         at
>>>>>>>>>>> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
>>>>>>>>>>> ~[guava-16.0.jar:na]
>>>>>>>>>>>         at
>>>>>>>>>>> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
>>>>>>>>>>> ~[guava-16.0.jar:na]
>>>>>>>>>>>         at
>>>>>>>>>>> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
>>>>>>>>>>> ~[guava-16.0.jar:na]
>>>>>>>>>>>         at
>>>>>>>>>>> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
>>>>>>>>>>> ~[guava-16.0.jar:na]
>>>>>>>>>>>         at
>>>>>>>>>>> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:208)
>>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13]
>>>>>>>>>>>         at
>>>>>>>>>>> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:184)
>>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13]
>>>>>>>>>>>         at
>>>>>>>>>>> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:415)
>>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13]
>>>>>>>>>>>         at
>>>>>>>>>>> org.apache.cassandra.streaming.StreamSession.sessionFailed(StreamSession.java:621)
>>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13]
>>>>>>>>>>>         at
>>>>>>>>>>> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:475)
>>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13]
>>>>>>>>>>>         at
>>>>>>>>>>> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:256)
>>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13]
>>>>>>>>>>>         at java.lang.Thread.run(Unknown Source)
~[na:1.7.0_79]
>>>>>>>>>>> ERROR [STREAM-OUT-/192.168.1.140] 2016-05-24
22:44:58,629
>>>>>>>>>>> StreamSession.java:505 - [Stream #2c290460-20d4-11e6-930f-1b05ac77baf9]
>>>>>>>>>>> Streaming error occurred
>>>>>>>>>>> java.io.IOException: Broken pipe
>>>>>>>>>>>         at sun.nio.ch.FileDispatcherImpl.write0(Native
Method)
>>>>>>>>>>> ~[na:1.7.0_79]
>>>>>>>>>>>         at sun.nio.ch.SocketDispatcher.write(Unknown
Source)
>>>>>>>>>>> ~[na:1.7.0_79]
>>>>>>>>>>>         at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown
>>>>>>>>>>> Source) ~[na:1.7.0_79]
>>>>>>>>>>>         at sun.nio.ch.IOUtil.write(Unknown Source)
~[na:1.7.0_79]
>>>>>>>>>>>         at sun.nio.ch.SocketChannelImpl.write(Unknown
Source)
>>>>>>>>>>> ~[na:1.7.0_79]
>>>>>>>>>>>         at
>>>>>>>>>>> org.apache.cassandra.io.util.DataOutputStreamAndChannel.write(DataOutputStreamAndChannel.java:48)
>>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13]
>>>>>>>>>>>         at
>>>>>>>>>>> org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:44)
>>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13]
>>>>>>>>>>>         at
>>>>>>>>>>> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:351)
>>>>>>>>>>> [apache-cassandra-2.1.13.jar:2.1.13]
>>>>>>>>>>>         at
>>>>>>>>>>> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:331)
>>>>>>>>>>> [apache-cassandra-2.1.13.jar:2.1.13]
>>>>>>>>>>>         at java.lang.Thread.run(Unknown Source)
[na:1.7.0_79]



-- 
Eric Evans
john.eric.evans@gmail.com

Mime
View raw message