cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andy Caldwell (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-9805) nodetool status causes garbage to be accrued
Date Wed, 27 Jan 2016 14:46:40 GMT


Andy Caldwell commented on CASSANDRA-9805:

Yes, we are, we're using the default number.  When we upgraded we migrated our deployments
over to vnodes, and all new deployments have used vnodes too.

The linked fix does sound related, once that patch lands we'll give the new build a go and
see if the issue has been resolved.

> nodetool status causes garbage to be accrued
> --------------------------------------------
>                 Key: CASSANDRA-9805
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Tools
>         Environment: Ubuntu 14.04 64-bit
> Cassandra 2.0.14
> Java 1.7.0 OpenJDK
>            Reporter: Andy Caldwell
> As part of monitoring our Cassandra clusters (generally 2-6 nodes) we were running `nodetool
status` regularly (~ every 5 minutes).  On Cassandra 1.2.12 this worked fine and had negligible
effect on the Cassandra database service.
> Having upgraded to Cassandra 2.0.14, we've found that, over time, the tenured memory
space slowly fills with `RMIConnectionImpl` objects (and some other associated objects) until
we start running into memory pressure and triggering proactive and then STW GC (which obviously
impact performance of the cluster).  It seems that these objects are kept around long enough
to get promoted to tenured from Eden and then don't get considered for collection (due to
internal reference cycles?).
> Very easy to reproduce, just call `nodetool status` in a loop and watch the memory usage
climb to capacity then drop to empty after STW.  No need to be accessing the DB keys at all.

This message was sent by Atlassian JIRA

View raw message