Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 23488200AE2 for ; Fri, 27 May 2016 16:50:40 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 1C0D8160A12; Fri, 27 May 2016 14:50:40 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id BBE86160A10 for ; Fri, 27 May 2016 16:50:38 +0200 (CEST) Received: (qmail 74384 invoked by uid 500); 27 May 2016 14:50:37 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 74373 invoked by uid 99); 27 May 2016 14:50:37 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2016 14:50:37 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id D81BA1804FC for ; Fri, 27 May 2016 14:50:36 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.179 X-Spam-Level: * X-Spam-Status: No, score=1.179 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=datastax.com Received: from mx2-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id FYKLgKCouAZz for ; Fri, 27 May 2016 14:50:34 +0000 (UTC) Received: from mail-wm0-f49.google.com (mail-wm0-f49.google.com [74.125.82.49]) by mx2-lw-us.apache.org (ASF Mail Server at mx2-lw-us.apache.org) with ESMTPS id B65455F575 for ; Fri, 27 May 2016 14:50:33 +0000 (UTC) Received: by mail-wm0-f49.google.com with SMTP id a136so76547926wme.0 for ; Fri, 27 May 2016 07:50:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=datastax.com; s=google; h=mime-version:in-reply-to:references:date:message-id:subject:from:to; bh=JdxRunDU8re5vqFWk8qRQSy9VaqsiuT53gGbVt8tAZU=; b=bJ0C1tgRJ8J/ivGmxq5R6V2YIVVq8rDJjB7VtRQsYM8FNaMNflBTsHY07QiKkojJ2J ptKQbdohZQw0D3iW7+gizhwub2fSEymiQEYkjpsjG0jYTpOVaBzWuv9B5rTty0kKXJYA cHYnJhD3aKgd6VVvONjBVTMQLEPi+0vWkBNDU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to; bh=JdxRunDU8re5vqFWk8qRQSy9VaqsiuT53gGbVt8tAZU=; b=ZjLZs3+bnjId/h0TjGVQyJgWEWVrTQw/XGEM91z8TBYx2Pb6tr3G73MBWYWy06k4Ev 3LH+bsFqyq8gHLUs4dfRsq+l6TycE0tYJ4lOC2RggeZ2hW1LSv83W1zVCRy5ct+GAAaj Jbpf31FJGLurbrln92J/61QMjAeK9qOnZ/IV3RREL3s8JhtEVOwpJioY190RkL8gPLqs i+EJ4Mgc2AyvwUxN0TOPvUcIz1eTMQwHUDGLcZlOsUKDnV4Fh3nvvKeLUqX6hkRWUAbU vd/gD479x0RZOitwBCQsX8q1muaJLlAFeEgLaE+eWh0cWw7g5bnKOxJviDD5CeSFqNn6 Sv9Q== X-Gm-Message-State: ALyK8tJk2Hn1CVp2/ci+F6PaoUtf2l8GDEB6oAecNakkUGUmzw1NnkhoLGCzjj5i3lN9rs3cSZrA/YxDm5zyZ8cg MIME-Version: 1.0 X-Received: by 10.194.179.233 with SMTP id dj9mr15317788wjc.115.1464360627102; Fri, 27 May 2016 07:50:27 -0700 (PDT) Received: by 10.28.88.11 with HTTP; Fri, 27 May 2016 07:50:26 -0700 (PDT) Received: by 10.28.88.11 with HTTP; Fri, 27 May 2016 07:50:26 -0700 (PDT) In-Reply-To: References: Date: Fri, 27 May 2016 10:50:26 -0400 Message-ID: Subject: Re: Error while rebuilding a node: Stream failed From: Sebastian Estevez To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=047d7bae49720086780533d407cd archived-at: Fri, 27 May 2016 14:50:40 -0000 --047d7bae49720086780533d407cd Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Check ifconfig for dripped tpc messages. Let's rule out your network. all the best, Sebasti=C3=A1n On May 27, 2016 10:45 AM, "George Sigletos" wrote: > Hello, > > No there is no version mix. The first stack traces were indeed from > 2.1.13. Then I upgraded all nodes to 2.1.14. Still getting the same error= s > > > On Fri, May 27, 2016 at 4:39 PM, Eric Evans > wrote: > >> From the various stacktraces in this thread, it's obvious you are >> mixing versions 2.1.13 and 2.1.14. Topology changes like this aren't >> supported with mixed Cassandra versions. Sometimes it will work, >> sometimes it won't (and it will definitely not work in this instance). >> >> You should either upgrade your 2.1.13 nodes to 2.1.14 first, or add >> the new nodes using 2.1.13, and upgrade after. >> >> On Fri, May 27, 2016 at 8:41 AM, George Sigletos >> wrote: >> >> >>>> ERROR [STREAM-IN-/192.168.1.141] 2016-05-26 09:08:05,027 >> >>>> StreamSession.java:505 - [Stream >> #74c57bc0-231a-11e6-a698-1b05ac77baf9] >> >>>> Streaming error occurred >> >>>> java.lang.RuntimeException: Outgoing stream handler has been closed >> >>>> at >> >>>> >> org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionH= andler.java:138) >> >>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >> >>>> at >> >>>> >> org.apache.cassandra.streaming.StreamSession.receive(StreamSession.java:= 568) >> >>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >> >>>> at >> >>>> >> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSessi= on.java:457) >> >>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >> >>>> at >> >>>> >> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.= run(ConnectionHandler.java:263) >> >>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >> >>>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_79] >> >>>> >> >>>> And this is from the source node: >> >>>> >> >>>> ERROR [STREAM-OUT-/172.31.22.104] 2016-05-26 11:08:05,097 >> >>>> StreamSession.java:505 - [Stream >> #74c57bc0-231a-11e6-a698-1b05ac77baf9] >> >>>> Streaming error occurred >> >>>> java.io.IOException: Broken pipe >> >>>> at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) >> >>>> ~[na:1.7.0_79] >> >>>> at sun.nio.ch.FileChannelImpl.transferToDirectly(Unknown >> Source) >> >>>> ~[na:1.7.0_79] >> >>>> at sun.nio.ch.FileChannelImpl.transferTo(Unknown Source) >> >>>> ~[na:1.7.0_79] >> >>>> at >> >>>> >> org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(Com= pressedStreamWriter.java:84) >> >>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >> >>>> at >> >>>> >> org.apache.cassandra.streaming.messages.OutgoingFileMessage.serialize(Ou= tgoingFileMessage.java:88) >> >>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >> >>>> at >> >>>> >> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(= OutgoingFileMessage.java:49) >> >>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >> >>>> at >> >>>> >> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(= OutgoingFileMessage.java:41) >> >>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >> >>>> at >> >>>> >> org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMe= ssage.java:45) >> >>>> ~[apache-cassandra-2.1.14.jar:2.1.14] >> >>>> at >> >>>> >> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.= sendMessage(ConnectionHandler.java:358) >> >>>> [apache-cassandra-2.1.14.jar:2.1.14] >> >>>> at >> >>>> >> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.= run(ConnectionHandler.java:330) >> >>>> [apache-cassandra-2.1.14.jar:2.1.14] >> >> >> >>>>>>>>>>> ERROR [STREAM-IN-/192.168.1.140] 2016-05-24 22:44:57,704 >> >>>>>>>>>>> StreamSession.java:620 - [Stream >> #2c290460-20d4-11e6-930f-1b05ac77baf9] >> >>>>>>>>>>> Remote peer 192.168.1.140 failed stream session. >> >>>>>>>>>>> ERROR [STREAM-OUT-/192.168.1.140] 2016-05-24 22:44:57,705 >> >>>>>>>>>>> StreamSession.java:505 - [Stream >> #2c290460-20d4-11e6-930f-1b05ac77baf9] >> >>>>>>>>>>> Streaming error occurred >> >>>>>>>>>>> java.io.IOException: Connection timed out >> >>>>>>>>>>> at sun.nio.ch.FileDispatcherImpl.write0(Native Metho= d) >> >>>>>>>>>>> ~[na:1.7.0_79] >> >>>>>>>>>>> at sun.nio.ch.SocketDispatcher.write(Unknown Source) >> >>>>>>>>>>> ~[na:1.7.0_79] >> >>>>>>>>>>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown >> >>>>>>>>>>> Source) ~[na:1.7.0_79] >> >>>>>>>>>>> at sun.nio.ch.IOUtil.write(Unknown Source) >> ~[na:1.7.0_79] >> >>>>>>>>>>> at sun.nio.ch.SocketChannelImpl.write(Unknown Source= ) >> >>>>>>>>>>> ~[na:1.7.0_79] >> >>>>>>>>>>> at >> >>>>>>>>>>> >> org.apache.cassandra.io.util.DataOutputStreamAndChannel.write(DataOutput= StreamAndChannel.java:48) >> >>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13] >> >>>>>>>>>>> at >> >>>>>>>>>>> >> org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMe= ssage.java:44) >> >>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13] >> >>>>>>>>>>> at >> >>>>>>>>>>> >> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.= sendMessage(ConnectionHandler.java:351) >> >>>>>>>>>>> [apache-cassandra-2.1.13.jar:2.1.13] >> >>>>>>>>>>> at >> >>>>>>>>>>> >> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.= run(ConnectionHandler.java:323) >> >>>>>>>>>>> [apache-cassandra-2.1.13.jar:2.1.13] >> >>>>>>>>>>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_79= ] >> >>>>>>>>>>> INFO [STREAM-IN-/192.168.1.140] 2016-05-24 22:44:58,625 >> >>>>>>>>>>> StreamResultFuture.java:180 - [Stream >> #2c290460-20d4-11e6-930f-1b05ac77baf9] >> >>>>>>>>>>> Session with /192.168.1.140 is complete >> >>>>>>>>>>> WARN [STREAM-IN-/192.168.1.140] 2016-05-24 22:44:58,627 >> >>>>>>>>>>> StreamResultFuture.java:207 - [Stream >> #2c290460-20d4-11e6-930f-1b05ac77baf9] >> >>>>>>>>>>> Stream failed >> >>>>>>>>>>> ERROR [RMI TCP Connection(24)-127.0.0.1] 2016-05-24 >> 22:44:58,628 >> >>>>>>>>>>> StorageService.java:1075 - Error while rebuilding node >> >>>>>>>>>>> org.apache.cassandra.streaming.StreamException: Stream faile= d >> >>>>>>>>>>> at >> >>>>>>>>>>> >> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailu= re(StreamEventJMXNotifier.java:85) >> >>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13] >> >>>>>>>>>>> at >> >>>>>>>>>>> >> com.google.common.util.concurrent.Futures$4.run(Futures.java:1172) >> >>>>>>>>>>> ~[guava-16.0.jar:na] >> >>>>>>>>>>> at >> >>>>>>>>>>> >> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorServic= e.execute(MoreExecutors.java:297) >> >>>>>>>>>>> ~[guava-16.0.jar:na] >> >>>>>>>>>>> at >> >>>>>>>>>>> >> com.google.common.util.concurrent.ExecutionList.executeListener(Executio= nList.java:156) >> >>>>>>>>>>> ~[guava-16.0.jar:na] >> >>>>>>>>>>> at >> >>>>>>>>>>> >> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.ja= va:145) >> >>>>>>>>>>> ~[guava-16.0.jar:na] >> >>>>>>>>>>> at >> >>>>>>>>>>> >> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFu= ture.java:202) >> >>>>>>>>>>> ~[guava-16.0.jar:na] >> >>>>>>>>>>> at >> >>>>>>>>>>> >> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamRe= sultFuture.java:208) >> >>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13] >> >>>>>>>>>>> at >> >>>>>>>>>>> >> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(= StreamResultFuture.java:184) >> >>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13] >> >>>>>>>>>>> at >> >>>>>>>>>>> >> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.= java:415) >> >>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13] >> >>>>>>>>>>> at >> >>>>>>>>>>> >> org.apache.cassandra.streaming.StreamSession.sessionFailed(StreamSession= .java:621) >> >>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13] >> >>>>>>>>>>> at >> >>>>>>>>>>> >> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSessi= on.java:475) >> >>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13] >> >>>>>>>>>>> at >> >>>>>>>>>>> >> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.= run(ConnectionHandler.java:256) >> >>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13] >> >>>>>>>>>>> at java.lang.Thread.run(Unknown Source) ~[na:1.7.0_7= 9] >> >>>>>>>>>>> ERROR [STREAM-OUT-/192.168.1.140] 2016-05-24 22:44:58,629 >> >>>>>>>>>>> StreamSession.java:505 - [Stream >> #2c290460-20d4-11e6-930f-1b05ac77baf9] >> >>>>>>>>>>> Streaming error occurred >> >>>>>>>>>>> java.io.IOException: Broken pipe >> >>>>>>>>>>> at sun.nio.ch.FileDispatcherImpl.write0(Native Metho= d) >> >>>>>>>>>>> ~[na:1.7.0_79] >> >>>>>>>>>>> at sun.nio.ch.SocketDispatcher.write(Unknown Source) >> >>>>>>>>>>> ~[na:1.7.0_79] >> >>>>>>>>>>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown >> >>>>>>>>>>> Source) ~[na:1.7.0_79] >> >>>>>>>>>>> at sun.nio.ch.IOUtil.write(Unknown Source) >> ~[na:1.7.0_79] >> >>>>>>>>>>> at sun.nio.ch.SocketChannelImpl.write(Unknown Source= ) >> >>>>>>>>>>> ~[na:1.7.0_79] >> >>>>>>>>>>> at >> >>>>>>>>>>> >> org.apache.cassandra.io.util.DataOutputStreamAndChannel.write(DataOutput= StreamAndChannel.java:48) >> >>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13] >> >>>>>>>>>>> at >> >>>>>>>>>>> >> org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMe= ssage.java:44) >> >>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:2.1.13] >> >>>>>>>>>>> at >> >>>>>>>>>>> >> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.= sendMessage(ConnectionHandler.java:351) >> >>>>>>>>>>> [apache-cassandra-2.1.13.jar:2.1.13] >> >>>>>>>>>>> at >> >>>>>>>>>>> >> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.= run(ConnectionHandler.java:331) >> >>>>>>>>>>> [apache-cassandra-2.1.13.jar:2.1.13] >> >>>>>>>>>>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_79= ] >> >> >> >> -- >> Eric Evans >> john.eric.evans@gmail.com >> > > --047d7bae49720086780533d407cd Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

Check ifconfig for dripped tpc messages. Let's rule out = your network.

all the best,

Sebasti=C3=A1n

On May 27, 2016 10:45 AM, "George Sigletos&= quot; <sigletos@textkernel.nl<= /a>> wrote:
Hello,

No there is no version mix. The f= irst stack traces were indeed from 2.1.13. Then I upgraded all nodes to 2.1= .14. Still getting the same errors


On Fri, May 27, 2016 at 4:39 PM, Eric = Evans <john.eric.evans@gmail.com> wrote:
From the various stacktraces in this thread, it'= s obvious you are
mixing versions 2.1.13 and 2.1.14.=C2=A0 Topology changes like this aren= 9;t
supported with mixed Cassandra versions.=C2=A0 Sometimes it will work,
sometimes it won't (and it will definitely not work in this instance).<= br>
You should either upgrade your 2.1.13 nodes to 2.1.14 first, or add
the new nodes using 2.1.13, and upgrade after.

On Fri, May 27, 2016 at 8:41 AM, George Sigletos <sigletos@textkernel.nl> wrote:=

>>>> ERROR [STREAM-IN-/192.168.1.141] 2016-05-26 09:08:05,027 >>>> StreamSession.java:505 - [Stream #74c57bc0-231a-11e6-a698-= 1b05ac77baf9]
>>>> Streaming error occurred
>>>> java.lang.RuntimeException: Outgoing stream handler has be= en closed
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
>>>> org.apache.cassandra.streaming.ConnectionHandler.sendMessa= ge(ConnectionHandler.java:138)
>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
>>>> org.apache.cassandra.streaming.StreamSession.receive(Strea= mSession.java:568)
>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
>>>> org.apache.cassandra.streaming.StreamSession.messageReceiv= ed(StreamSession.java:457)
>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
>>>> org.apache.cassandra.streaming.ConnectionHandler$IncomingM= essageHandler.run(ConnectionHandler.java:263)
>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at java.lang.Thread.run(U= nknown Source) [na:1.7.0_79]
>>>>
>>>> And this is from the source node:
>>>>
>>>> ERROR [STREAM-OUT-/172.31.22.104] 2016-05-26 11:08:05,097 >>>> StreamSession.java:505 - [Stream #74c57bc0-231a-11e6-a698-= 1b05ac77baf9]
>>>> Streaming error occurred
>>>> java.io.IOException: Broken pipe
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at sun.nio.ch.FileChannel= Impl.transferTo0(Native Method)
>>>> ~[na:1.7.0_79]
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at sun.nio.ch.FileChannel= Impl.transferToDirectly(Unknown Source)
>>>> ~[na:1.7.0_79]
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at sun.nio.ch.FileChannel= Impl.transferTo(Unknown Source)
>>>> ~[na:1.7.0_79]
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
>>>> org.apache.cassandra.streaming.compress.CompressedStreamWr= iter.write(CompressedStreamWriter.java:84)
>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
>>>> org.apache.cassandra.streaming.messages.OutgoingFileMessag= e.serialize(OutgoingFileMessage.java:88)
>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
>>>> org.apache.cassandra.streaming.messages.OutgoingFileMessag= e$1.serialize(OutgoingFileMessage.java:49)
>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
>>>> org.apache.cassandra.streaming.messages.OutgoingFileMessag= e$1.serialize(OutgoingFileMessage.java:41)
>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
>>>> org.apache.cassandra.streaming.messages.StreamMessage.seri= alize(StreamMessage.java:45)
>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
>>>> org.apache.cassandra.streaming.ConnectionHandler$OutgoingM= essageHandler.sendMessage(ConnectionHandler.java:358)
>>>> [apache-cassandra-2.1.14.jar:2.1.14]
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
>>>> org.apache.cassandra.streaming.ConnectionHandler$OutgoingM= essageHandler.run(ConnectionHandler.java:330)
>>>> [apache-cassandra-2.1.14.jar:2.1.14]


>>>>>>>>>>> ERROR [S= TREAM-IN-/192.168.1.140] 2016-05-24 22:44:57,704
>>>>>>>>>>> StreamSession.java:620 - [Stre= am #2c290460-20d4-11e6-930f-1b05ac77baf9]
>>>>>>>>>>> Remote peer 192.168.1.140 fail= ed stream session.
>>>>>>>>>>> ERROR [STREAM-OUT-/192.168.1.140] 2016-05-24 22:44:57,705
>>>>>>>>>>> StreamSession.java:505 - [Stre= am #2c290460-20d4-11e6-930f-1b05ac77baf9]
>>>>>>>>>>> Streaming error occurred
>>>>>>>>>>> java.io.IOException: Connectio= n timed out
>>>>>>>>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>>>>>> ~[na:1.7.0_79]
>>>>>>>>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0at sun.nio.ch.SocketDispatcher.write(Unknown Source)
>>>>>>>>>>> ~[na:1.7.0_79]
>>>>>>>>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown
>>>>>>>>>>> Source) ~[na:1.7.0_79]
>>>>>>>>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0at sun.nio.ch.IOUtil.write(Unknown Source) ~[na:1.7.0_79]
>>>>>>>>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0at sun.nio.ch.SocketChannelImpl.write(Unknown Source)
>>>>>>>>>>> ~[na:1.7.0_79]
>>>>>>>>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0at
>>>>>>>>>>> org.apache.cassandra.io.util.D= ataOutputStreamAndChannel.write(DataOutputStreamAndChannel.java:48)
>>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:= 2.1.13]
>>>>>>>>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0at
>>>>>>>>>>> org.apache.cassandra.streaming= .messages.StreamMessage.serialize(StreamMessage.java:44)
>>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:= 2.1.13]
>>>>>>>>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0at
>>>>>>>>>>> org.apache.cassandra.streaming= .ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.jav= a:351)
>>>>>>>>>>> [apache-cassandra-2.1.13.jar:2= .1.13]
>>>>>>>>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0at
>>>>>>>>>>> org.apache.cassandra.streaming= .ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:323) >>>>>>>>>>> [apache-cassandra-2.1.13.jar:2= .1.13]
>>>>>>>>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0at java.lang.Thread.run(Unknown Source) [na:1.7.0_79]
>>>>>>>>>>> INFO=C2=A0 [STREAM-IN-/192.168.1.14= 0] 2016-05-24 22:44:58,625
>>>>>>>>>>> StreamResultFuture.java:180 - = [Stream #2c290460-20d4-11e6-930f-1b05ac77baf9]
>>>>>>>>>>> Session with /192.168.1.140 is = complete
>>>>>>>>>>> WARN=C2=A0 [STREAM-IN-/192.168.1.14= 0] 2016-05-24 22:44:58,627
>>>>>>>>>>> StreamResultFuture.java:207 - = [Stream #2c290460-20d4-11e6-930f-1b05ac77baf9]
>>>>>>>>>>> Stream failed
>>>>>>>>>>> ERROR [RMI TCP Connection(24)-= 127.0.0.1] 2016-05-24 22:44:58,628
>>>>>>>>>>> StorageService.java:1075 - Err= or while rebuilding node
>>>>>>>>>>> org.apache.cassandra.streaming= .StreamException: Stream failed
>>>>>>>>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0at
>>>>>>>>>>> org.apache.cassandra.streaming= .management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85= )
>>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:= 2.1.13]
>>>>>>>>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0at
>>>>>>>>>>> com.google.common.util.concurr= ent.Futures$4.run(Futures.java:1172)
>>>>>>>>>>> ~[guava-16.0.jar:na]
>>>>>>>>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0at
>>>>>>>>>>> com.google.common.util.concurr= ent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)=
>>>>>>>>>>> ~[guava-16.0.jar:na]
>>>>>>>>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0at
>>>>>>>>>>> com.google.common.util.concurr= ent.ExecutionList.executeListener(ExecutionList.java:156)
>>>>>>>>>>> ~[guava-16.0.jar:na]
>>>>>>>>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0at
>>>>>>>>>>> com.google.common.util.concurr= ent.ExecutionList.execute(ExecutionList.java:145)
>>>>>>>>>>> ~[guava-16.0.jar:na]
>>>>>>>>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0at
>>>>>>>>>>> com.google.common.util.concurr= ent.AbstractFuture.setException(AbstractFuture.java:202)
>>>>>>>>>>> ~[guava-16.0.jar:na]
>>>>>>>>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0at
>>>>>>>>>>> org.apache.cassandra.streaming= .StreamResultFuture.maybeComplete(StreamResultFuture.java:208)
>>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:= 2.1.13]
>>>>>>>>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0at
>>>>>>>>>>> org.apache.cassandra.streaming= .StreamResultFuture.handleSessionComplete(StreamResultFuture.java:184)
>>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:= 2.1.13]
>>>>>>>>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0at
>>>>>>>>>>> org.apache.cassandra.streaming= .StreamSession.closeSession(StreamSession.java:415)
>>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:= 2.1.13]
>>>>>>>>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0at
>>>>>>>>>>> org.apache.cassandra.streaming= .StreamSession.sessionFailed(StreamSession.java:621)
>>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:= 2.1.13]
>>>>>>>>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0at
>>>>>>>>>>> org.apache.cassandra.streaming= .StreamSession.messageReceived(StreamSession.java:475)
>>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:= 2.1.13]
>>>>>>>>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0at
>>>>>>>>>>> org.apache.cassandra.streaming= .ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:256) >>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:= 2.1.13]
>>>>>>>>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0at java.lang.Thread.run(Unknown Source) ~[na:1.7.0_79]
>>>>>>>>>>> ERROR [STREAM-OUT-/192.168.1.140] 2016-05-24 22:44:58,629
>>>>>>>>>>> StreamSession.java:505 - [Stre= am #2c290460-20d4-11e6-930f-1b05ac77baf9]
>>>>>>>>>>> Streaming error occurred
>>>>>>>>>>> java.io.IOException: Broken pi= pe
>>>>>>>>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>>>>>> ~[na:1.7.0_79]
>>>>>>>>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0at sun.nio.ch.SocketDispatcher.write(Unknown Source)
>>>>>>>>>>> ~[na:1.7.0_79]
>>>>>>>>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown
>>>>>>>>>>> Source) ~[na:1.7.0_79]
>>>>>>>>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0at sun.nio.ch.IOUtil.write(Unknown Source) ~[na:1.7.0_79]
>>>>>>>>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0at sun.nio.ch.SocketChannelImpl.write(Unknown Source)
>>>>>>>>>>> ~[na:1.7.0_79]
>>>>>>>>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0at
>>>>>>>>>>> org.apache.cassandra.io.util.D= ataOutputStreamAndChannel.write(DataOutputStreamAndChannel.java:48)
>>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:= 2.1.13]
>>>>>>>>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0at
>>>>>>>>>>> org.apache.cassandra.streaming= .messages.StreamMessage.serialize(StreamMessage.java:44)
>>>>>>>>>>> ~[apache-cassandra-2.1.13.jar:= 2.1.13]
>>>>>>>>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0at
>>>>>>>>>>> org.apache.cassandra.streaming= .ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.jav= a:351)
>>>>>>>>>>> [apache-cassandra-2.1.13.jar:2= .1.13]
>>>>>>>>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0at
>>>>>>>>>>> org.apache.cassandra.streaming= .ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:331) >>>>>>>>>>> [apache-cassandra-2.1.13.jar:2= .1.13]
>>>>>>>>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0at java.lang.Thread.run(Unknown Source) [na:1.7.0_79]



--
Eric Evans
john.eric.ev= ans@gmail.com

--047d7bae49720086780533d407cd--