incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mohit Anchlia <mohitanch...@gmail.com>
Subject Re: very confused by jmap dump of cassandra
Date Thu, 21 Feb 2013 17:33:49 GMT
Roughly how much data do you have per node?

Sent from my iPhone

On Feb 20, 2013, at 10:49 AM, "Hiller, Dean" <Dean.Hiller@nrel.gov> wrote:

> I took this jmap dump of cassandra(in production).  Before I restarted the whole production
cluster, I had some nodes running compaction and it looked like all memory had been consumed(kind
of like cassandra is not clearing out the caches or memtables fast enough).  I am trying to
still debug compaction causes slowness on the cluster since all cassandra.yaml files are pretty
much the defaults with size tiered compaction.
> 
> The weird thing is I dump and get a 5.4G heap.bin file and load that into ecipse who
tells me total is 142.8MB….what???? So low????, top was showing 1.9G at the time(and I took
this top snapshot later(2 hours after)… (how is eclipse profile telling me the jmap showed
142.8MB in use instead of 1.9G in use?)
> 
> Tasks: 398 total,   1 running, 397 sleeping,   0 stopped,   0 zombie
> Cpu(s):  2.8%us,  0.5%sy,  0.0%ni, 96.5%id,  0.1%wa,  0.0%hi,  0.1%si,  0.0%st
> Mem:  32854680k total, 31910708k used,   943972k free,    89776k buffers
> Swap: 33554424k total,    18288k used, 33536136k free, 23428596k cached
> 
>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 20909 cassandr  20   0 64.1g 9.2g 2.1g S 75.7 29.4 182:37.92 java
> 22455 cassandr  20   0 15288 1340  824 R  3.9  0.0   0:00.02 top
> 
> It almost seems like cassandra is not being good about memory management here as we slowly
get into a situation where compaction is run which takes out our memory(configured for 8G).
 I can easily go higher than 8G on these systems as I have 32gig each node, but there was
docs that said 8G is better for GC.  Has anyone else taken a jmap dump of cassandra?
> 
> Thanks,
> Dean
> 

Mime
View raw message