cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paulo Motta (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-11345) Assertion Errors "Memory was freed" during streaming
Date Wed, 11 May 2016 18:16:13 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-11345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15280544#comment-15280544
] 

Paulo Motta commented on CASSANDRA-11345:
-----------------------------------------

[~vineus] It seems you are seeing the effects CASSANDRA-8343. The problem is that it takes
more than {{streaming_socket_timeout_in_ms}} (default=1h) to transfer this 50GB file, so the
sending node does not receive any message from the receiving node in the incoming socket and
times out the stream session. As a temporary workaround you should increase this property
to a larger value (10h or 20h) in sending nodes.

In any case we should probably keep this ticket open to prevent this memory was freed error
from happening when the stream session fails for some other reason.

> Assertion Errors "Memory was freed" during streaming
> ----------------------------------------------------
>
>                 Key: CASSANDRA-11345
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-11345
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Streaming and Messaging
>            Reporter: Jean-Francois Gosselin
>            Assignee: Paulo Motta
>
> We encountered the following AssertionError (twice on the same node) during a repair
:
> On node /172.16.63.41
> {noformat}
> INFO  [STREAM-IN-/10.174.216.160] 2016-03-09 02:38:13,900 StreamResultFuture.java:180
- [Stream #f6980580-e55f-11e5-8f08-ef9e099ce99e] Session with /10.174.216.160 is complete
                                                                                         
             
> WARN  [STREAM-IN-/10.174.216.160] 2016-03-09 02:38:13,900 StreamResultFuture.java:207
- [Stream #f6980580-e55f-11e5-8f08-ef9e099ce99e] Stream failed                           
> ERROR [STREAM-OUT-/10.174.216.160] 2016-03-09 02:38:13,906 StreamSession.java:505 - [Stream
#f6980580-e55f-11e5-8f08-ef9e099ce99e] Streaming error occurred                    
> java.lang.AssertionError: Memory was freed                                          
                                                                                         

>         at org.apache.cassandra.io.util.SafeMemory.checkBounds(SafeMemory.java:97) ~[apache-cassandra-2.1.13.jar:2.1.13]
                                                  
>         at org.apache.cassandra.io.util.Memory.getLong(Memory.java:249) ~[apache-cassandra-2.1.13.jar:2.1.13]
                                                             
>         at org.apache.cassandra.io.compress.CompressionMetadata.getTotalSizeForSections(CompressionMetadata.java:247)
~[apache-cassandra-2.1.13.jar:2.1.13]                
>         at org.apache.cassandra.streaming.messages.FileMessageHeader.size(FileMessageHeader.java:112)
~[apache-cassandra-2.1.13.jar:2.1.13]                                
>         at org.apache.cassandra.streaming.StreamSession.fileSent(StreamSession.java:546)
~[apache-cassandra-2.1.13.jar:2.1.13]                                             
>         at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:50)
~[apache-cassandra-2.1.13.jar:2.1.13]                      
>         at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:41)
~[apache-cassandra-2.1.13.jar:2.1.13]                      
>         at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:45)
~[apache-cassandra-2.1.13.jar:2.1.13]                                    
>         at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:351)
~[apache-cassandra-2.1.13.jar:2.1.13]           
>         at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:331)
~[apache-cassandra-2.1.13.jar:2.1.13]                   
>         at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]                      
                                                                                         

> {noformat}     
> On node /10.174.216.160
>  
> {noformat}       
> ERROR [STREAM-OUT-/172.16.63.41] 2016-03-09 02:38:14,140 StreamSession.java:505 - [Stream
#f6980580-e55f-11e5-8f08-ef9e099ce99e] Streaming error occurred                          
> java.io.IOException: Connection reset by peer                                       
                                                                                      
>         at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[na:1.7.0_65]       
                                                                                      
>         at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[na:1.7.0_65]
                                                                                     
>         at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[na:1.7.0_65]   
                                                                                      
>         at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[na:1.7.0_65]                   
                                                                                      
>         at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487) ~[na:1.7.0_65]
                                                                                  
>         at org.apache.cassandra.io.util.DataOutputStreamAndChannel.write(DataOutputStreamAndChannel.java:48)
~[apache-cassandra-2.1.13.jar:2.1.13]                     
>         at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:44)
~[apache-cassandra-2.1.13.jar:2.1.13]                                
>         at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:351)
[apache-cassandra-2.1.13.jar:2.1.13]        
>         at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:323)
[apache-cassandra-2.1.13.jar:2.1.13]                
>         at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]                      
                                                                                      
> INFO  [STREAM-IN-/172.16.63.41] 2016-03-09 02:38:14,142 StreamResultFuture.java:180 -
[Stream #f6980580-e55f-11e5-8f08-ef9e099ce99e] Session with /172.16.63.41 is complete
> WARN  [STREAM-IN-/172.16.63.41] 2016-03-09 02:38:14,142 StreamResultFuture.java:207 -
[Stream #f6980580-e55f-11e5-8f08-ef9e099ce99e] Stream failed                         
> ERROR [STREAM-OUT-/172.16.63.41] 2016-03-09 02:38:14,143 StreamSession.java:505 - [Stream
#f6980580-e55f-11e5-8f08-ef9e099ce99e] Streaming error occurred                  
> java.io.IOException: Broken pipe                                                    
                                                                                      
>         at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[na:1.7.0_65]       
                                                                                      
>         at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[na:1.7.0_65]
                                                                                     
>         at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[na:1.7.0_65]   
                                                                                      
>         at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[na:1.7.0_65]                   
                                                                                      
>         at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487) ~[na:1.7.0_65]
                                                                                  
>         at org.apache.cassandra.io.util.DataOutputStreamAndChannel.write(DataOutputStreamAndChannel.java:48)
~[apache-cassandra-2.1.13.jar:2.1.13]                     
>         at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:44)
~[apache-cassandra-2.1.13.jar:2.1.13]                                
>         at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:351)
[apache-cassandra-2.1.13.jar:2.1.13]        
>         at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:331)
[apache-cassandra-2.1.13.jar:2.1.13]                
>         at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]     
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message