cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brandon Williams (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-4571) Strange permament socket descriptors increasing leads to "Too many open files"
Date Thu, 23 Aug 2012 18:22:42 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-4571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13440516#comment-13440516
] 

Brandon Williams commented on CASSANDRA-4571:
---------------------------------------------

I've seen this a few times, but never found a cause/resolution, so I'll go ahead and dump
what I know:

* All cases thus far seem to be upgrades, not new installations.

* 1.1 but less than 1.1.2 doesn't seem to exhibit

* Cassandra doesn't use unix sockets, at all

* This is fairly rare and only hits a handful of users

* some people have this happen on all nodes, some have it happen on only a portion

* going to such lengths as trying all kinds of different JVM versions and completely switching
OSes has not helped

One user wrote a simple app to track the lost FDs here: http://pastebin.com/faBkJueB and it
seemed to correlate with opening one sstable, and another user has corroborated that.  Both
report heavy reads on that CF.

No way to reproduce is yet known, I've failed in all my attempts.
                
> Strange permament socket descriptors increasing leads to "Too many open files"
> ------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-4571
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4571
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.1.2
>         Environment: CentOS 5.8 Linux 2.6.18-308.13.1.el5 #1 SMP Tue Aug 21 17:10:18
EDT 2012 x86_64 x86_64 x86_64 GNU/Linux. 
> java version "1.6.0_33"
> Java(TM) SE Runtime Environment (build 1.6.0_33-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
>            Reporter: Serg Shnerson
>            Priority: Critical
>
> On the two-node cluster there was found strange socket descriptors increasing. lsof -n
| grep java shows many rows like"
> java       8380 cassandra  113r     unix 0xffff8101a374a080            938348482 socket
> java       8380 cassandra  114r     unix 0xffff8101a374a080            938348482 socket
> java       8380 cassandra  115r     unix 0xffff8101a374a080            938348482 socket
> java       8380 cassandra  116r     unix 0xffff8101a374a080            938348482 socket
> java       8380 cassandra  117r     unix 0xffff8101a374a080            938348482 socket
> java       8380 cassandra  118r     unix 0xffff8101a374a080            938348482 socket
> java       8380 cassandra  119r     unix 0xffff8101a374a080            938348482 socket
> java       8380 cassandra  120r     unix 0xffff8101a374a080            938348482 socket
> " And number of this rows constantly increasing. After about 24 hours this situation
leads to error.
> We use PHPCassa client. Load is not so high (aroud ~50kb/s on write). 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message