cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Omri Iluz (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-10448) "Unknown type 0" Stream failure on Repair
Date Tue, 01 Dec 2015 06:34:11 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-10448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15033182#comment-15033182
] 

Omri Iluz commented on CASSANDRA-10448:
---------------------------------------

+1
any chance that this can be resolve in time for 2.2.4? this is blocking us from expanding
the cluster as well. happy to test patches that increase log level or dump more information.

> "Unknown type 0" Stream failure on Repair
> -----------------------------------------
>
>                 Key: CASSANDRA-10448
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10448
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Streaming and Messaging
>         Environment: Cassandra 2.2.2
> 5 Nodes in Google Compute Engine
> Java 1.8.0_60
>            Reporter: Omri Iluz
>            Assignee: Yuki Morishita
>             Fix For: 2.2.x
>
>         Attachments: casslogs.txt, receiversystem.log, sendersystem.log
>
>
> While running repair after upgrading to 2.2.2 I am getting many stream fail errors:
> {noformat}
> [2015-10-05 23:52:30,353] Repair session 4c181051-6bbb-11e5-acdb-d9a8bbd39330 for range
(59694553044959221,86389982480621619] failed with error [repair #4c181051-6bbb-11e5-acdb-d9a8bbd39330
on px/acti
> vities, (59694553044959221,86389982480621619]] Sync failed between /10.240.81.104 and
/10.240.134.221 (progress: 4%)
> {noformat}
> Logs from both sides of the stream:
> Sides 1 -
> {noformat}
> INFO  [STREAM-INIT-/10.240.81.104:52722] 2015-10-05 23:52:30,063 StreamResultFuture.java:111
- [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550 ID#0] Creating new streaming plan for Repair
> INFO  [STREAM-INIT-/10.240.81.104:52722] 2015-10-05 23:52:30,063 StreamResultFuture.java:118
- [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550, ID#0] Received streaming plan for Repair
> INFO  [STREAM-INIT-/10.240.81.104:52723] 2015-10-05 23:52:30,063 StreamResultFuture.java:118
- [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550, ID#0] Received streaming plan for Repair
> INFO  [STREAM-IN-/10.240.81.104] 2015-10-05 23:52:30,098 StreamResultFuture.java:168
- [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550 ID#0] Prepare completed. Receiving 13 files(517391317
bytes), sending 10 files(469491729 bytes)
> ERROR [STREAM-IN-/10.240.81.104] 2015-10-05 23:52:30,234 StreamSession.java:524 - [Stream
#239d8e60-6bbc-11e5-93ac-31bdef2dc550] Streaming error occurred
> java.lang.IllegalArgumentException: Unknown type 0
> 	at org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:96)
~[apache-cassandra-2.2.2.jar:2.2.2]
> 	at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:57)
~[apache-cassandra-2.2.2.jar:2.2.2]
> 	at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
~[apache-cassandra-2.2.2.jar:2.2.2]
> 	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
> INFO  [STREAM-IN-/10.240.81.104] 2015-10-05 23:52:30,302 StreamResultFuture.java:182
- [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550] Session with /10.240.81.104 is complete
> WARN  [STREAM-IN-/10.240.81.104] 2015-10-05 23:52:30,302 StreamResultFuture.java:209
- [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550] Stream failed
> {noformat}
> Side 2 -
> {noformat}
> INFO  [AntiEntropyStage:1] 2015-10-05 23:52:30,060 StreamResultFuture.java:86 - [Stream
#239d8e60-6bbc-11e5-93ac-31bdef2dc550] Executing streaming plan for Repair
> INFO  [StreamConnectionEstablisher:6] 2015-10-05 23:52:30,061 StreamSession.java:232
- [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550] Starting streaming to /10.240.134.221
> INFO  [StreamConnectionEstablisher:6] 2015-10-05 23:52:30,063 StreamCoordinator.java:213
- [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550, ID#0] Beginning stream session with /10.240.134.221
> INFO  [STREAM-IN-/10.240.134.221] 2015-10-05 23:52:30,098 StreamResultFuture.java:168
- [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550 ID#0] Prepare completed. Receiving 10 files(469491729
bytes), sending 13 files(517391317 bytes)
> INFO  [STREAM-IN-/10.240.134.221] 2015-10-05 23:52:30,349 StreamResultFuture.java:182
- [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550] Session with /10.240.134.221 is complete
> ERROR [STREAM-OUT-/10.240.134.221] 2015-10-05 23:52:30,349 StreamSession.java:524 - [Stream
#239d8e60-6bbc-11e5-93ac-31bdef2dc550] Streaming error occurred
> org.apache.cassandra.io.FSReadError: java.io.IOException: Broken pipe
> 	at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:144) ~[apache-cassandra-2.2.2.jar:2.2.2]
> 	at org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:79)
~[apache-cassandra-2.2.2.jar:2.2.2]
> 	at org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:76)
~[apache-cassandra-2.2.2.jar:2.2.2]
> 	at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.applyToChannel(BufferedDataOutputStreamPlus.java:293)
~[apache-cassandra-2.2.2.jar:2.2.2]
> 	at org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:75)
~[apache-cassandra-2.2.2.jar:2.2.2]
> 	at org.apache.cassandra.streaming.messages.OutgoingFileMessage.serialize(OutgoingFileMessage.java:96)
~[apache-cassandra-2.2.2.jar:2.2.2]
> 	at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:48)
~[apache-cassandra-2.2.2.jar:2.2.2]
> 	at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:40)
~[apache-cassandra-2.2.2.jar:2.2.2]
> 	at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:47)
~[apache-cassandra-2.2.2.jar:2.2.2]
> 	at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:363)
~[apache-cassandra-2.2.2.jar:2.2.2]
> 	at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:335)
~[apache-cassandra-2.2.2.jar:2.2.2]
> 	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
> Caused by: java.io.IOException: Broken pipe
> 	at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) ~[na:1.8.0_60]
> 	at sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:427) ~[na:1.8.0_60]
> 	at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:492) ~[na:1.8.0_60]
> 	at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:607) ~[na:1.8.0_60]
> 	at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:140) ~[apache-cassandra-2.2.2.jar:2.2.2]
> 	... 11 common frames omitted
> WARN  [STREAM-IN-/10.240.134.221] 2015-10-05 23:52:30,349 StreamResultFuture.java:209
- [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550] Stream failed
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message