cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "NIcholas Rushton (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-13133) Unclosed file descriptors when querying SnapshotsSize metric
Date Wed, 18 Jan 2017 23:43:26 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-13133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15828988#comment-15828988
] 

NIcholas Rushton commented on CASSANDRA-13133:
----------------------------------------------

only place I see directory file descriptors being explicitly opened is here: https://github.com/apache/cassandra/blob/cassandra-3.X/src/java/org/apache/cassandra/db/lifecycle/LogAwareFileLister.java#L83-L95

but, there may be more indirect ways they are being created.

> Unclosed file descriptors when querying SnapshotsSize metric
> ------------------------------------------------------------
>
>                 Key: CASSANDRA-13133
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13133
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: CentOS 7
>            Reporter: NIcholas Rushton
>              Labels: jmx, lhf, metrics, newbie
>             Fix For: 3.9
>
>
> Started to notice many open file descriptors (100k+) per node, growing at a rate of about
30 per minute in our cluster. After turning off our JMX exporting server(https://github.com/prometheus/jmx_exporter),
which gets queried every 30 seconds, the number of file descriptors remained static. 
> Digging a bit further I ran a jmx dump tool over all the cassandra metrics and tracked
the number of file descriptors after each query, boiling it down to a single metric causing
the number of file descriptors to increase:
> org.apache.cassandra.metrics:keyspace=tpsv1,name=SnapshotsSize,scope=events_by_engagement_id,type=Table
> running a query a few times against this metric shows the file descriptors increasing
after each query:
> {code}
> for _ in {0..3} 
> do 
>    java -jar jmx-dump-0.4.2-standalone.jar --port 7199 --dump org.apache.cassandra.metrics:keyspace=tpsv1,name=SnapshotsSize,scope=events_by_engagement_id,type=Table
> /dev/null; 
>    sudo lsof -p `pgrep -f CassandraDaemon` | fgrep "DIR" | awk '{a[$(NF)]+=1}END{for(k
in a){print k, a[k]}}' | grep "events_by" 
> done
> > /data/cassandra/data/tpsv1/events_by_engagement_id-01d8f450a54911e6917ec93f8a91ec71
33176
> > /data/cassandra/data/tpsv1/events_by_engagement_id-01d8f450a54911e6917ec93f8a91ec71
33177
> > /data/cassandra/data/tpsv1/events_by_engagement_id-01d8f450a54911e6917ec93f8a91ec71
33178
> > /data/cassandra/data/tpsv1/events_by_engagement_id-01d8f450a54911e6917ec93f8a91ec71
33179
> {code}
> it should be noted that the file descriptor is open on a directory, not an actual file



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message