cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mikhail Stepura (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-6546) disablethrift results in unclosed file descriptors
Date Wed, 30 Apr 2014 21:03:16 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-6546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986068#comment-13986068
] 

Mikhail Stepura edited comment on CASSANDRA-6546 at 4/30/14 9:01 PM:
---------------------------------------------------------------------

So, we have to cleanup slector keys in HSHA server before closing the selector itself.

There is no such issue with the SYNC server, and behavior observed by [~enigmacurry] in "Scenario
2" happens because his script didn't close the connections from *the client side*, and all
those ESTABLISHED connections are actually held by the client. As soon as the client(s) close
the connections (using {{pool.dispose()}} in dtest for ex.) those ESTABLISHED will be gone.
And that is correct.

In case of HSHA those ESTABLISHED would become CLOSE_WAIT, and attached patch cleans them
up.


was (Author: mishail):
So, we have to cleanup slector keys in HSHA server before closing the selector itself.

There is no such issue with the SYNC server, and behavior observed by [~enigmacurry] in "Scenario
2" happens because his script didn't close the connections from *the client side*, and all
those ESTABLISHED connections are actually held by the client. As soon as the client(s) close
the connection (using {{pool.dispose()}} in dtest for ex.}} those ESTABLISHED are gone. And
that is correct.

In case of HSHA those ESTABLISHED would become CLOSE_WAIT, and attached patch cleans them
up.

> disablethrift results in unclosed file descriptors
> --------------------------------------------------
>
>                 Key: CASSANDRA-6546
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6546
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Jason Harvey
>            Assignee: Mikhail Stepura
>            Priority: Minor
>             Fix For: 1.2.17, 2.0.8, 2.1 beta2
>
>         Attachments: 2014-04-30 12-22-17.png, 2014-04-30 12-22-58.png, CASSANDRA-6546-1.2.patch
>
>
> Disabling thrift results in unclosed thrift sockets being left around.
> Steps to reproduce and observe:
> 1. Have a handful of clients connect via thrift.
> 2. Disable thrift.
> 3. Enable thrift, have the clients reconnect.
> 4. Observe netstat or lsof, and you'll find a lot of thrift sockets in CLOSE_WAIT state,
and they'll never go away.
>   * Also verifiable from org.apache.cassandra.metrics:type=Client,name=connectedThriftClients
MBean.
> What's extra fun about this is the leaked sockets still count towards your maximum RPC
thread count. As a result, toggling thrift enough times will result in an rpc_max_threads
number of CLOSED_WAIT sockets, with no new clients able to connect.
> This was reproduced with HSHA. I haven't tried it in sync yet.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message