cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "techpyaasa (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-10303) streaming for 'nodetool rebuild' fails after adding a datacenter
Date Wed, 28 Sep 2016 09:24:20 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-10303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15529032#comment-15529032
] 

techpyaasa commented on CASSANDRA-10303:
----------------------------------------

I'm facing similar exception during 'nodetool rebuild' when trying to add new data center(DC3)
in existing c*-2.0.17 cluster which has already 2 data centers DC1 , DC2.(Each DC has 3 groups
,with each group has 3 nodes , total 9 nodes per DC and approx 700GB data per node with RF-3
on all DCs)

{quote}
ERROR [STREAM-OUT-/xxx.xxx.198.191] 2016-09-27 00:28:10,327 StreamSession.java (line 461)
[Stream #30852870-8472-11e6-b043-3f260c696828] Streaming error occurred
java.io.IOException: Connection timed out
        at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
        at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
        at sun.nio.ch.IOUtil.write(IOUtil.java:65)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487)
        at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:44)
        at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:339)
        at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:311)
        at java.lang.Thread.run(Thread.java:745)
 INFO [STREAM-OUT-/xxx.xxx.198.191] 2016-09-27 00:28:10,347 StreamResultFuture.java (line
186) [Stream #30852870-8472-11e6-b043-3f260c696828] Session with /xxx.xxx.198.191 is complete
ERROR [STREAM-OUT-/xxx.xxx.198.191] 2016-09-27 00:28:10,347 StreamSession.java (line 461)
[Stream #30852870-8472-11e6-b043-3f260c696828] Streaming error occurred
java.io.IOException: Broken pipe
        at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
        at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
        at sun.nio.ch.IOUtil.write(IOUtil.java:65)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487)
        at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:44)
        at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:339)
        at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:319)
        at java.lang.Thread.run(Thread.java:745)
ERROR [STREAM-IN-/xxx.xxx.198.191] 2016-09-27 00:28:10,461 StreamSession.java (line 461) [Stream
#30852870-8472-11e6-b043-3f260c696828] Streaming error occurred
java.lang.RuntimeException: Outgoing stream handler has been closed
        at org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:126)
        at org.apache.cassandra.streaming.StreamSession.receive(StreamSession.java:524)
        at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:413)
        at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:245)
        at java.lang.Thread.run(Thread.java:745)
{quote}

"sysctl -w net.ipv4.tcp_keepalive_time=60 net.ipv4.tcp_keepalive_probes=3 net.ipv4.tcp_keepalive_intvl=10"

Does setting this would fix this issue ? And if so , is this enough to set this on new nodes
on which are going to run 'nodetool rebuild' or need to change this values on all existing
nodes from which data is going to get streamed?

Thanks in advance.





> streaming for 'nodetool rebuild' fails after adding a datacenter 
> -----------------------------------------------------------------
>
>                 Key: CASSANDRA-10303
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10303
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: jdk1.7
> cassandra 2.1.8
>            Reporter: zhaoyan
>
> we add another datacenter.
> use nodetool rebuild DC1
> stream from some node of old datacenter always hang up with these exception:
> {code}
> ERROR [Thread-1472] 2015-09-10 19:24:53,091 CassandraDaemon.java:223 - Exception in thread
Thread[Thread-1472,5,RMI Runtime]
> java.lang.RuntimeException: java.io.IOException: Connection timed out
>     at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.jar:na]
>     at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) ~[apache-cassandra-2.1.8.jar:2.1.8]
>     at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_60]
> Caused by: java.io.IOException: Connection timed out
>     at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.7.0_60]
>     at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[na:1.7.0_60]
>     at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[na:1.7.0_60]
>     at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[na:1.7.0_60]
>     at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) ~[na:1.7.0_60]
>     at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:59) ~[na:1.7.0_60]
>     at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109) ~[na:1.7.0_60]
>     at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) ~[na:1.7.0_60]
>     at org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:172)
~[apache-cassandra-2.1.8.jar:2.1.8]
>     at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.8.jar:2.1.8]
>     ... 1 common frames omitted
> {code}
> i must restart node to stop current rebuild, and rebuild agagin and again to success....



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message