cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sebastian Estevez <sebastian.este...@datastax.com>
Subject Re: Error while rebuilding a node: Stream failed
Date Fri, 27 May 2016 14:50:26 GMT
Check ifconfig for dripped tpc messages. Let's rule out your network.

all the best,

Sebastián
On May 27, 2016 10:45 AM, "George Sigletos" <sigletos@textkernel.nl> wrote:

> Hello,
>
> No there is no version mix. The first stack traces were indeed from
> 2.1.13. Then I upgraded all nodes to 2.1.14. Still getting the same errors
>
>
> On Fri, May 27, 2016 at 4:39 PM, Eric Evans <john.eric.evans@gmail.com>
> wrote:
>
>> From the various stacktraces in this thread, it's obvious you are
>> mixing versions 2.1.13 and 2.1.14.  Topology changes like this aren't
>> supported with mixed Cassandra versions.  Sometimes it will work,
>> sometimes it won't (and it will definitely not work in this instance).
>>
>> You should either upgrade your 2.1.13 nodes to 2.1.14 first, or add
>> the new nodes using 2.1.13, and upgrade after.
>>
>> On Fri, May 27, 2016 at 8:41 AM, George Sigletos <sigletos@textkernel.nl>
>> wrote:
>>
>> >>>> ERROR [STREAM-IN-/192.168.1.141] 2016-05-26 09:08:05,027
>> >>>> StreamSession.java:505 - [Stream
>> #74c57bc0-231a-11e6-a698-1b05ac77baf9]
>> >>>> Streaming error occurred
>> >>>> java.lang.RuntimeException: Outgoing stream handler has been closed
>> >>>>         at
>> >>>>
>> org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:138)
>> >>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>> >>>>         at
>> >>>>
>> org.apache.cassandra.streaming.StreamSession.receive(StreamSession.java:568)
>> >>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>> >>>>         at
>> >>>>
>> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:457)
>> >>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>> >>>>         at
>> >>>>
>> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:263)
>> >>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>> >>>>         at java.lang.Thread.run(Unknown Source) [na:1.7.0_79]
>> >>>>
>> >>>> And this is from the source node:
>> >>>>
>> >>>> ERROR [STREAM-OUT-/172.31.22.104] 2016-05-26 11:08:05,097
>> >>>> StreamSession.java:505 - [Stream
>> #74c57bc0-231a-11e6-a698-1b05ac77baf9]
>> >>>> Streaming error occurred
>> >>>> java.io.IOException: Broken pipe
>> >>>>         at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
>> >>>> ~[na:1.7.0_79]
>> >>>>         at sun.nio.ch.FileChannelImpl.transferToDirectly(Unknown
>> Source)
>> >>>> ~[na:1.7.0_79]
>> >>>>         at sun.nio.ch.FileChannelImpl.transferTo(Unknown Source)
>> >>>> ~[na:1.7.0_79]
>> >>>>         at
>> >>>>
>> org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:84)
>> >>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>> >>>>         at
>> >>>>
>> org.apache.cassandra.streaming.messages.OutgoingFileMessage.serialize(OutgoingFileMessage.java:88)
>> >>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>> >>>>         at
>> >>>>
>> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:49)
>> >>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>> >>>>         at
>> >>>>
>> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:41)
>> >>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>> >>>>         at
>> >>>>
>> org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:45)
>> >>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>> >>>>         at
>> >>>>
>> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:358)
>> >>>> [apache-cassandra-2.1.14.jar:2.1.14]
>> >>>>         at
>> >>>>
>> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:330)
>> >>>> [apache-cassandra-2.1.14.jar:2.1.14]
>>
>>
>> >>>>>>>>>>> ERROR [STREAM-IN-/192.168.1.140] 2016-05-24
22:44:57,704
>> >>>>>>>>>>> StreamSession.java:620 - [Stream
>> #2c290460-20d4-11e6-930f-1b05ac77baf9]
>> >>>>>>>>>>> Remote peer 192.168.1.140 failed stream
session.
>> >>>>>>>>>>> ERROR [STREAM-OUT-/192.168.1.140] 2016-05-24
22:44:57,705
>> >>>>>>>>>>> StreamSession.java:505 - [Stream
>> #2c290460-20d4-11e6-930f-1b05ac77baf9]
>> >>>>>>>>>>> Streaming error occurred
>> >>>>>>>>>>> java.io.IOException: Connection timed
out
>> >>>>>>>>>>>         at sun.nio.ch.FileDispatcherImpl.write0(Native
Method)
>> >>>>>>>>>>> ~[na:1.7.0_79]
>> >>>>>>>>>>>         at sun.nio.ch.SocketDispatcher.write(Unknown
Source)
>> >>>>>>>>>>> ~[na:1.7.0_79]
>> >>>>>>>>>>>         at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown
>> >>>>>>>>>>> Source) ~[na:1.7.0_79]
>> >>>>>>>>>>>         at sun.nio.ch.IOUtil.write(Unknown
Source)
>> ~[na:1.7.0_79]
>> >>>>>>>>>>>         at sun.nio.ch.SocketChannelImpl.write(Unknown
Source)
>> >>>>>>>>>>> ~[na:1.7.0_79]
>> >>>>>>>>>>>         at
>> >>>>>>>>>>>
>> org.apache.cassandra.io.util.DataOutputStreamAndChannel.write(DataOutputStreamAndChannel.java:48)
>> >>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13]
>> >>>>>>>>>>>         at
>> >>>>>>>>>>>
>> org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:44)
>> >>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13]
>> >>>>>>>>>>>         at
>> >>>>>>>>>>>
>> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:351)
>> >>>>>>>>>>> [apache-cassandra-2.1.13.jar:2.1.13]
>> >>>>>>>>>>>         at
>> >>>>>>>>>>>
>> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:323)
>> >>>>>>>>>>> [apache-cassandra-2.1.13.jar:2.1.13]
>> >>>>>>>>>>>         at java.lang.Thread.run(Unknown
Source) [na:1.7.0_79]
>> >>>>>>>>>>> INFO  [STREAM-IN-/192.168.1.140] 2016-05-24
22:44:58,625
>> >>>>>>>>>>> StreamResultFuture.java:180 - [Stream
>> #2c290460-20d4-11e6-930f-1b05ac77baf9]
>> >>>>>>>>>>> Session with /192.168.1.140 is complete
>> >>>>>>>>>>> WARN  [STREAM-IN-/192.168.1.140] 2016-05-24
22:44:58,627
>> >>>>>>>>>>> StreamResultFuture.java:207 - [Stream
>> #2c290460-20d4-11e6-930f-1b05ac77baf9]
>> >>>>>>>>>>> Stream failed
>> >>>>>>>>>>> ERROR [RMI TCP Connection(24)-127.0.0.1]
2016-05-24
>> 22:44:58,628
>> >>>>>>>>>>> StorageService.java:1075 - Error while
rebuilding node
>> >>>>>>>>>>> org.apache.cassandra.streaming.StreamException:
Stream failed
>> >>>>>>>>>>>         at
>> >>>>>>>>>>>
>> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
>> >>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13]
>> >>>>>>>>>>>         at
>> >>>>>>>>>>>
>> com.google.common.util.concurrent.Futures$4.run(Futures.java:1172)
>> >>>>>>>>>>> ~[guava-16.0.jar:na]
>> >>>>>>>>>>>         at
>> >>>>>>>>>>>
>> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
>> >>>>>>>>>>> ~[guava-16.0.jar:na]
>> >>>>>>>>>>>         at
>> >>>>>>>>>>>
>> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
>> >>>>>>>>>>> ~[guava-16.0.jar:na]
>> >>>>>>>>>>>         at
>> >>>>>>>>>>>
>> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
>> >>>>>>>>>>> ~[guava-16.0.jar:na]
>> >>>>>>>>>>>         at
>> >>>>>>>>>>>
>> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
>> >>>>>>>>>>> ~[guava-16.0.jar:na]
>> >>>>>>>>>>>         at
>> >>>>>>>>>>>
>> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:208)
>> >>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13]
>> >>>>>>>>>>>         at
>> >>>>>>>>>>>
>> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:184)
>> >>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13]
>> >>>>>>>>>>>         at
>> >>>>>>>>>>>
>> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:415)
>> >>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13]
>> >>>>>>>>>>>         at
>> >>>>>>>>>>>
>> org.apache.cassandra.streaming.StreamSession.sessionFailed(StreamSession.java:621)
>> >>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13]
>> >>>>>>>>>>>         at
>> >>>>>>>>>>>
>> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:475)
>> >>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13]
>> >>>>>>>>>>>         at
>> >>>>>>>>>>>
>> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:256)
>> >>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13]
>> >>>>>>>>>>>         at java.lang.Thread.run(Unknown
Source) ~[na:1.7.0_79]
>> >>>>>>>>>>> ERROR [STREAM-OUT-/192.168.1.140] 2016-05-24
22:44:58,629
>> >>>>>>>>>>> StreamSession.java:505 - [Stream
>> #2c290460-20d4-11e6-930f-1b05ac77baf9]
>> >>>>>>>>>>> Streaming error occurred
>> >>>>>>>>>>> java.io.IOException: Broken pipe
>> >>>>>>>>>>>         at sun.nio.ch.FileDispatcherImpl.write0(Native
Method)
>> >>>>>>>>>>> ~[na:1.7.0_79]
>> >>>>>>>>>>>         at sun.nio.ch.SocketDispatcher.write(Unknown
Source)
>> >>>>>>>>>>> ~[na:1.7.0_79]
>> >>>>>>>>>>>         at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown
>> >>>>>>>>>>> Source) ~[na:1.7.0_79]
>> >>>>>>>>>>>         at sun.nio.ch.IOUtil.write(Unknown
Source)
>> ~[na:1.7.0_79]
>> >>>>>>>>>>>         at sun.nio.ch.SocketChannelImpl.write(Unknown
Source)
>> >>>>>>>>>>> ~[na:1.7.0_79]
>> >>>>>>>>>>>         at
>> >>>>>>>>>>>
>> org.apache.cassandra.io.util.DataOutputStreamAndChannel.write(DataOutputStreamAndChannel.java:48)
>> >>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13]
>> >>>>>>>>>>>         at
>> >>>>>>>>>>>
>> org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:44)
>> >>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13]
>> >>>>>>>>>>>         at
>> >>>>>>>>>>>
>> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:351)
>> >>>>>>>>>>> [apache-cassandra-2.1.13.jar:2.1.13]
>> >>>>>>>>>>>         at
>> >>>>>>>>>>>
>> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:331)
>> >>>>>>>>>>> [apache-cassandra-2.1.13.jar:2.1.13]
>> >>>>>>>>>>>         at java.lang.Thread.run(Unknown
Source) [na:1.7.0_79]
>>
>>
>>
>> --
>> Eric Evans
>> john.eric.evans@gmail.com
>>
>
>

Mime
View raw message