cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ryan McGuire (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-6546) disablethrift results in unclosed file descriptors
Date Fri, 14 Mar 2014 17:24:46 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-6546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13935284#comment-13935284
] 

Ryan McGuire commented on CASSANDRA-6546:
-----------------------------------------

Tested the following:

Scenario 1:
 * rpc_max_threads: 16
 * rpc_server_type: sync
 * Create 16 pycassa thrift connections, try to create one more, it blocks due to rpc_max_threads
reached. Seems like correct behavior.

Scenario 2:
 * rpc_max_threads: 16
 * rpc_server_type: sync
 * Create 16 pycassa thrift connections, disable thrift, reenable thrift, try to create 16
more thrift connections, it works, and connectedThriftClients mbean counts up to ~32. This
can be looped as many times as you want, until maximum sockets allowed by ulimit is reached
(1024 on my machine). netstat shows all of those connections in ESTABLISHED state on my machine.

Scenario 3:
  * rpc_max_threads: 16
  * rpc_server_type: hsha
  * Create as many connections as you want, it never blocks until Liniux ulimit is reached.
With hsha enabled, rpc_max_threads does not cap the number of connections, though that may
be to due the asynchronous design. I haven't figured out how to check how many actual threads
are running.

Scenario 4:
  * rpc_max_threads: 16
  * rpc_server_type: hsha
  * Same scenario as 3 but introduce thrift disable/reenable between 16 connection starts.
Same effect, create as many connections as you want up to ulimit.

Scenario 2 seems like a definite bug, I'm not sure about 3 and 4.

[Here's the script I'm using to test|https://github.com/riptano/cassandra-dtest/blob/262b2918250aa53b8010804023a35f288d9e9073/thrift_hsha_test.py#L19]

> disablethrift results in unclosed file descriptors
> --------------------------------------------------
>
>                 Key: CASSANDRA-6546
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6546
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Jason Harvey
>            Assignee: Ryan McGuire
>            Priority: Minor
>
> Disabling thrift results in unclosed thrift sockets being left around.
> Steps to reproduce and observe:
> 1. Have a handful of clients connect via thrift.
> 2. Disable thrift.
> 3. Enable thrift, have the clients reconnect.
> 4. Observe netstat or lsof, and you'll find a lot of thrift sockets in CLOSE_WAIT state,
and they'll never go away.
>   * Also verifiable from org.apache.cassandra.metrics:type=Client,name=connectedThriftClients
MBean.
> What's extra fun about this is the leaked sockets still count towards your maximum RPC
thread count. As a result, toggling thrift enough times will result in an rpc_max_threads
number of CLOSED_WAIT sockets, with no new clients able to connect.
> This was reproduced with HSHA. I haven't tried it in sync yet.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message