cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "mlowicki (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-10992) Hanging streaming sessions
Date Tue, 12 Jan 2016 20:54:39 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-10992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15094919#comment-15094919
] 

mlowicki commented on CASSANDRA-10992:
--------------------------------------

Some IO errors I've found in logs:
{code}
ERROR [Thread-518762] 2016-01-12 14:36:11,130 CassandraDaemon.java:227 - Exception in thread
Thread[Thread-518762,5,main]
java.lang.RuntimeException: java.io.IOException: Connection timed out
        at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.jar:na]
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) ~[apache-cassandra-2.1.12.jar:2.1.12]
        at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_66]
Caused by: java.io.IOException: Connection timed out
        at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.8.0_66]
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[na:1.8.0_66]
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[na:1.8.0_66]
        at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[na:1.8.0_66]
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) ~[na:1.8.0_66]
        at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:59) ~[na:1.8.0_66]
        at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109) ~[na:1.8.0_66]
        at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) ~[na:1.8.0_66]
        at org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:178)
~[apache-cassandra-2.1.12.jar:2.1.12]
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.12.jar:2.1.12]
        ... 1 common frames omitted
{code}

{code}
ERROR [STREAM-IN-/10.210.58.133] 2016-01-12 15:01:39,450 StreamSession.java:505 - [Stream
#193dd5c0-b93b-11e5-a713-8fe7d1d062ea] Streaming error occurred
java.io.IOException: Connection timed out
        at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.8.0_66]
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[na:1.8.0_66]
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[na:1.8.0_66]
        at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[na:1.8.0_66]
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) ~[na:1.8.0_66]
        at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:51)
~[apache-cassandra-2.1.12.jar:2.1.12]
        at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:250)
~[apache-cassandra-2.1.12.jar:2.1.12]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66]
INFO  [STREAM-IN-/10.210.58.133] 2016-01-12 15:01:39,451 StreamResultFuture.java:180 - [Stream
#193dd5c0-b93b-11e5-a713-8fe7d1d062ea] Session with /10.210.58.133 is complete
WARN  [STREAM-IN-/10.210.58.133] 2016-01-12 15:01:39,451 StreamResultFuture.java:207 - [Stream
#193dd5c0-b93b-11e5-a713-8fe7d1d062ea] Stream failed
{code}

{code}
ERROR [Thread-404196] 2016-01-12 14:44:05,532 CassandraDaemon.java:227 - Exception in thread
Thread[Thread-404196,5,main]
java.lang.RuntimeException: java.nio.channels.AsynchronousCloseException
        at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.jar:na]
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) ~[apache-cassandra-2.1.12.jar:2.1.12]
        at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_66]
Caused by: java.nio.channels.AsynchronousCloseException: null
        at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:205)
~[na:1.8.0_66]
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:407) ~[na:1.8.0_66]
        at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:59) ~[na:1.8.0_66]
        at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109) ~[na:1.8.0_66]
        at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) ~[na:1.8.0_66]
        at org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:178)
~[apache-cassandra-2.1.12.jar:2.1.12]
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.12.jar:2.1.12]
        ... 1 common frames omitted
{code}

{code}
ERROR [STREAM-OUT-/10.210.3.224] 2016-01-12 14:44:12,114 StreamSession.java:505 - [Stream
#e7af3850-b93a-11e5-bebc-2f019a24a954] Streaming error occurred
java.io.IOException: Broken pipe
        at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) ~[na:1.8.0_66]
        at sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:427)
~[na:1.8.0_66]
        at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:492) ~[na:1.8.0_66]
        at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:607) ~[na:1.8.0_66]
        at org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:74)
~[apache-cassandra-2.1.12.jar:2.1.12]
        at org.apache.cassandra.streaming.messages.OutgoingFileMessage.serialize(OutgoingFileMessage.java:88)
~[apache-cassandra-2.1.12.jar:2.1.12]
        at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:49)
~[apache-cassandra-2.1.12.jar:2.1.12]
        at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:41)
~[apache-cassandra-2.1.12.jar:2.1.12]
        at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:45)
~[apache-cassandra-2.1.12.jar:2.1.12]
        at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:351)
[apache-cassandra-2.1.12.jar:2.1.12]
        at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:323)
[apache-cassandra-2.1.12.jar:2.1.12]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66]
INFO  [STREAM-OUT-/10.210.3.224] 2016-01-12 14:44:12,114 StreamResultFuture.java:180 - [Stream
#e7af3850-b93a-11e5-bebc-2f019a24a954] Session with /10.210.3.224 is complete
WARN  [STREAM-OUT-/10.210.3.224] 2016-01-12 14:44:12,114 StreamResultFuture.java:207 - [Stream
#e7af3850-b93a-11e5-bebc-2f019a24a954] Stream failed
{code}

{code}
ERROR [STREAM-OUT-/10.210.3.224] 2016-01-12 14:44:12,115 StreamSession.java:505 - [Stream
#e7af3850-b93a-11e5-bebc-2f019a24a954] Streaming error occurred
java.io.IOException: Broken pipe
        at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[na:1.8.0_66]
        at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[na:1.8.0_66]
        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[na:1.8.0_66]
        at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[na:1.8.0_66]
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) ~[na:1.8.0_66]
        at org.apache.cassandra.io.util.DataOutputStreamAndChannel.write(DataOutputStreamAndChannel.java:48)
~[apache-cassandra-2.1.12.jar:2.1.12]
        at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:44)
~[apache-cassandra-2.1.12.jar:2.1.12]
        at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:351)
[apache-cassandra-2.1.12.jar:2.1.12]
        at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:331)
[apache-cassandra-2.1.12.jar:2.1.12]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66]
ERROR [STREAM-OUT-/10.210.3.224] 2016-01-12 14:44:12,115 StreamSession.java:505 - [Stream
#e7af3850-b93a-11e5-bebc-2f019a24a954] Streaming error occurred
{code}



> Hanging streaming sessions
> --------------------------
>
>                 Key: CASSANDRA-10992
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10992
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: C* 2.1.12, Debian Wheezy
>            Reporter: mlowicki
>            Assignee: Paulo Motta
>             Fix For: 2.1.12
>
>         Attachments: apache-cassandra-2.1.12-SNAPSHOT.jar
>
>
> I've started recently running repair using [Cassandra Reaper|https://github.com/spotify/cassandra-reaper]
 (built-in {{nodetool repair}} doesn't work for me - CASSANDRA-9935). It behaves fine but
I've noticed hanging streaming sessions:
> {code}
> root@db1:~# date
> Sat Jan  9 16:43:00 UTC 2016
> root@db1:~# nt netstats -H | grep total
>         Receiving 5 files, 46.59 MB total. Already received 1 files, 11.32 MB total
>         Sending 7 files, 46.28 MB total. Already sent 7 files, 46.28 MB total
>         Receiving 6 files, 64.15 MB total. Already received 1 files, 12.14 MB total
>         Sending 5 files, 61.15 MB total. Already sent 5 files, 61.15 MB total
>         Receiving 4 files, 7.75 MB total. Already received 3 files, 7.58 MB total
>         Sending 4 files, 4.29 MB total. Already sent 4 files, 4.29 MB total
>         Receiving 12 files, 13.79 MB total. Already received 11 files, 7.66 MB total
>         Sending 5 files, 15.32 MB total. Already sent 5 files, 15.32 MB total
>         Receiving 8 files, 20.35 MB total. Already received 1 files, 13.63 MB total
>         Sending 38 files, 125.34 MB total. Already sent 38 files, 125.34 MB total
> root@db1:~# date
> Sat Jan  9 17:45:42 UTC 2016
> root@db1:~# nt netstats -H | grep total
>         Receiving 5 files, 46.59 MB total. Already received 1 files, 11.32 MB total
>         Sending 7 files, 46.28 MB total. Already sent 7 files, 46.28 MB total
>         Receiving 6 files, 64.15 MB total. Already received 1 files, 12.14 MB total
>         Sending 5 files, 61.15 MB total. Already sent 5 files, 61.15 MB total
>         Receiving 4 files, 7.75 MB total. Already received 3 files, 7.58 MB total
>         Sending 4 files, 4.29 MB total. Already sent 4 files, 4.29 MB total
>         Receiving 12 files, 13.79 MB total. Already received 11 files, 7.66 MB total
>         Sending 5 files, 15.32 MB total. Already sent 5 files, 15.32 MB total
>         Receiving 8 files, 20.35 MB total. Already received 1 files, 13.63 MB total
>         Sending 38 files, 125.34 MB total. Already sent 38 files, 125.34 MB total
> {code}
> Such sessions are left even when repair job is long time done (confirmed by checking
Reaper's and Cassandra's logs). {{streaming_socket_timeout_in_ms}} in cassandra.yaml is set
to default value (3600000).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message