cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alexey Burylov (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-10206) Incorrect handling of end-of stream leading to infinite loop in streaming session
Date Thu, 27 Aug 2015 16:12:46 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-10206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716958#comment-14716958
] 

Alexey Burylov commented on CASSANDRA-10206:
--------------------------------------------

{qoute}
Are you observing streaming hang when this happens?
{qoute}
Streaming session visible in JMX monitoring and node-tool unity:
For thread 17365 and 17633 of threaddump-1440686737505.tdump
{code}
nodetool **** netstats | grep -B1 "10.8.6.84"
Repair 834af7d0-4cb8-11e5-813b-273d25d97b13
    /10.8.6.84
        Receiving 3 files, 141990 bytes total. Already received 3 files, 141990 bytes total
            ***-tmp-ka-77703-Data.db 66507/66507 bytes(100%) received from idx:0/10.8.6.84
            ***-tmp-ka-77702-Data.db 55758/55758 bytes(100%) received from idx:0/10.8.6.84
            ***-tmp-ka-77701-Data.db 19725/19725 bytes(100%) received from idx:0/10.8.6.84
        Sending 3 files, 151858 bytes total. Already sent 3 files, 151858 bytes total
            ***-ka-59374-Data.db 20716/20716 bytes(100%) sent to idx:0/10.8.6.84
            ***-ka-75986-Data.db 75224/75224 bytes(100%) sent to idx:0/10.8.6.84
            ***-ka-53876-Data.db 55918/55918 bytes(100%) sent to idx:0/10.8.6.84
--
Repair 7b088ec1-4cb8-11e5-813b-273d25d97b13
    /10.8.6.84
        Receiving 3 files, 124841 bytes total. Already received 3 files, 124841 bytes total
            ***-tmp-ka-495105-Data.db 49758/49758 bytes(100%) received from idx:0/10.8.6.84
            ***-tmp-ka-495104-Data.db 24711/24711 bytes(100%) received from idx:0/10.8.6.84
            ***-tmp-ka-495106-Data.db 50372/50372 bytes(100%) received from idx:0/10.8.6.84
        Sending 4 files, 175697 bytes total. Already sent 4 files, 175697 bytes total
            ***-ka-489513-Data.db 49255/49255 bytes(100%) sent to idx:0/10.8.6.84
            ***-ka-483689-Data.db 74596/74596 bytes(100%) sent to idx:0/10.8.6.84
            ***-ka-494198-Data.db 25941/25941 bytes(100%) sent to idx:0/10.8.6.84
            ***-ka-493844-Data.db 25905/25905 bytes(100%) sent to idx:0/10.8.6.84
{code}


{qoute}
Are you observing streaming hang when this happens?
{qoute}
Yes it hang on 100%

{qoute}
What does other end of stream (thread dump) looks like?
{qoute}
I try to capture form live node. At  now, i can say what stream session continue even after
other site node rebooted.

> Incorrect handling of end-of stream leading to infinite loop in streaming session
> ---------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-10206
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10206
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: cassndra 2.1.8 12 nodes cluster in two DC, with enabled ssl encryption.
>            Reporter: Alexey Burylov
>             Fix For: 2.1.x
>
>         Attachments: cassandra-2.1-10206.txt, threaddump-1440601628223.tdump, threaddump-1440602926719.tdump,
threaddump-1440672821181.tdump, threaddump-1440686737505.tdump
>
>
> StreamMessage.deserialize (StreamMessage.java:63) function return null when receive end
of stream for ReadableByteChannel.read () function. IncomingMessageHandler.run (ConnectionHandler.java:257)
interpreting null - as unsupported message and ignore it. It resulting infinite loop when
IncomingMessageHandler try to read message from closed steram, and deserialize always return
null - to indicate close stream.
> It not happen on normal socket, because when normal socket closed it throw IOException.
But happen from time to time when iner-node ssl encryption enabled.
> I'm attach two tread dump from one node with two infinity lopped thread with id 614 and
583



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message