incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hiller, Dean" <Dean.Hil...@nrel.gov>
Subject very confused by jmap dump of cassandra
Date Wed, 20 Feb 2013 18:49:59 GMT
I took this jmap dump of cassandra(in production).  Before I restarted the whole production
cluster, I had some nodes running compaction and it looked like all memory had been consumed(kind
of like cassandra is not clearing out the caches or memtables fast enough).  I am trying to
still debug compaction causes slowness on the cluster since all cassandra.yaml files are pretty
much the defaults with size tiered compaction.

The weird thing is I dump and get a 5.4G heap.bin file and load that into ecipse who tells
me total is 142.8MB….what???? So low????, top was showing 1.9G at the time(and I took this
top snapshot later(2 hours after)… (how is eclipse profile telling me the jmap showed 142.8MB
in use instead of 1.9G in use?)

Tasks: 398 total,   1 running, 397 sleeping,   0 stopped,   0 zombie
Cpu(s):  2.8%us,  0.5%sy,  0.0%ni, 96.5%id,  0.1%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:  32854680k total, 31910708k used,   943972k free,    89776k buffers
Swap: 33554424k total,    18288k used, 33536136k free, 23428596k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
20909 cassandr  20   0 64.1g 9.2g 2.1g S 75.7 29.4 182:37.92 java
22455 cassandr  20   0 15288 1340  824 R  3.9  0.0   0:00.02 top

It almost seems like cassandra is not being good about memory management here as we slowly
get into a situation where compaction is run which takes out our memory(configured for 8G).
 I can easily go higher than 8G on these systems as I have 32gig each node, but there was
docs that said 8G is better for GC.  Has anyone else taken a jmap dump of cassandra?

Thanks,
Dean


Mime
View raw message