incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jedd Rashbrooke <jedd.rashbro...@imagini.net>
Subject Memory leak with Sun Java 1.6 ?
Date Fri, 10 Dec 2010 17:29:58 GMT
 Howdi,

 We're using Cassandra 0.6.6 - intending to wait until 0.7 before
 we do any more upgrades.

 We're running a cluster of 16 boxes of 7.1GB each, on Amazon EC2
 using Ubuntu 10.04 (LTS).

 Today we saw one box kick its little feet up, and after investigating
 the other machines, it looks like they're all approaching the same fate.

 Over the past month or so, it looks like memory has slowly
 been exhausted.  Both nodetool drain and jmap can't run, and
 produce this error:

     Error occurred during initialization of VM
     Could not reserve enough space for object heap

 We've got Xmx/Xms set to 4GB.

 top shows free memory around 50-80MB, file cache under
 10MB, and the java process at 12+GB virt and 7.1GB res.

 This feels like a Java problem, not a Cassandra one, but I'm
 open to suggestions.  To ensure I don't get bothered over
 the weekend we're doing a rolling restart of Cassandra on
 each of the boxes now.  The last time they were restarted
 was just over a month ago.  Now I'm wondering whether I
 should (until 0.7.1 is available) schedule in a slower rolling
 restart over several days, every few weeks.

 I've shared a Zabbix graph of system memory at:

     http://www.imagebam.com/image/3b4213110283969

 cheers,
 Jedd.

Mime
View raw message