cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Burroughs (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-8188) don't block SocketThread for MessagingService
Date Thu, 20 Nov 2014 16:28:34 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-8188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Chris Burroughs updated CASSANDRA-8188:
---------------------------------------
    Attachment: handshake.stack.txt

Co-worker was faster with me and did get a stacktrace

> don't block SocketThread for MessagingService
> ---------------------------------------------
>
>                 Key: CASSANDRA-8188
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8188
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>            Reporter: yangwei
>            Assignee: yangwei
>             Fix For: 2.1.2
>
>         Attachments: 0001-don-t-block-SocketThread-for-MessagingService.patch, handshake.stack.txt
>
>
> We have two datacenters A and B.
> The node in A cannot handshake version with nodes in B, logs in A as follow:
> {noformat}	
> 	INFO [HANDSHAKE-/B] 2014-10-24 04:29:49,075 OutboundTcpConnection.java (line 395) Cannot
handshake version with B
>     TRACE [WRITE-/B] 2014-10-24 11:02:49,044 OutboundTcpConnection.java (line 368) unable
to connect to /B
> 		java.net.ConnectException: Connection refused
>         at sun.nio.ch.Net.connect0(Native Method)
>         at sun.nio.ch.Net.connect(Net.java:364)
>         at sun.nio.ch.Net.connect(Net.java:356)
>         at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:623)
>         at java.nio.channels.SocketChannel.open(SocketChannel.java:184)
>         at org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket(OutboundTcpConnectionPool.java:134)
>         at org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket(OutboundTcpConnectionPool.java:119)
>         at org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:299)
>         at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:150)
> {noformat}
>     
> The jstack output of nodes in B shows it blocks in inputStream.readInt resulting in SocketThread
not accept socket any more, logs as follow:
> {noformat}
> 	   java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
>         at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>         at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
>         at sun.nio.ch.IOUtil.read(IOUtil.java:197)
>         at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
>         - locked <0x00000007963747e8> (a java.lang.Object)
>         at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:203)
>         - locked <0x0000000796374848> (a java.lang.Object)
>         at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
>         - locked <0x00000007a5c7ca88> (a sun.nio.ch.SocketAdaptor$SocketInputStream)
>         at java.io.InputStream.read(InputStream.java:101)
>         at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:81)
>         - locked <0x00000007a5c7ca88> (a sun.nio.ch.SocketAdaptor$SocketInputStream)
>         at java.io.DataInputStream.readInt(DataInputStream.java:387)
>         at org.apache.cassandra.net.MessagingService$SocketThread.run(MessagingService.java:879)
> {noformat}
>        
> In nodes of B tcpdump shows retransmission of SYN,ACK during the tcp three-way handshake
phase because tcp implementation drops the last ack when the backlog queue is full.
> In nodes of B ss -tl shows "Recv-Q 51 Send-Q 50".
>         
> In nodes of B netstat -s shows “SYNs to LISTEN sockets dropped” and “times the
listen queue of a socket overflowed” are both increasing.
> This patch sets read timeout to 2 * OutboundTcpConnection.WAIT_FOR_VERSION_MAX_TIME for
the accepted socket.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message