incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arindam Barua <aba...@247-inc.com>
Subject RE: java.lang.OutOfMemoryError: Java heap space
Date Tue, 29 Apr 2014 02:09:09 GMT

If you don’t have row caching turned on, you can try changing settings to get your memtables
to flush faster (read more at the datastax documention at [1]). Btw, if you are using a pre-1.2
version, your bloom filter might come into play as well.

All said and done, if your write request rate is too high for your ring to handle, you just
can’t flush fast enough, and you’ll need to move to physical machines and/or add nodes
to your ring.

-Arindam

[1] http://www.datastax.com/documentation/cassandra/2.0/cassandra/troubleshooting/trblshootDyingNodes_r.html


From: Gary Zhao [mailto:garyzhao@gmail.com]
Sent: Monday, April 28, 2014 5:34 PM
To: user@cassandra.apache.org
Subject: Re: java.lang.OutOfMemoryError: Java heap space

Yes, they are virtual machines, but we are using KVM. Is there any solutions for this issue
or we should use physical machines?

On Mon, Apr 28, 2014 at 10:38 AM, Prem Yadav <ipremyadav@gmail.com<mailto:ipremyadav@gmail.com>>
wrote:
Are the virtual machines? The last time I had this issues was because of VMWare "ballooning".
If not, what versions of Cassandra and Java are you using?

On Mon, Apr 28, 2014 at 6:30 PM, Gary Zhao <garyzhao@gmail.com<mailto:garyzhao@gmail.com>>
wrote:
BTW, the CPU usage on this node is pretty high, but data size is pretty small.

  PID USERNAME  THR PRI NICE  SIZE   RES   SHR STATE   TIME    CPU COMMAND
28674 cassandr   89  25    0 9451M 8970M  525M sleep  32.1H   329% java

UN  8.92 GB    256     35.4%  c2d9d02e-bdb3-47cb-af1b-eabc2eeb503b  rack1
UN  5.32 GB    256     33.1%  874a269f-96bc-4db6-9d15-7cac6771a2ea  rack1
DN  339.94 MB  256     31.5%  fba5b071-aee1-451f-ae42-49c3bbe70bbe  rack1

On Mon, Apr 28, 2014 at 10:25 AM, Gary Zhao <garyzhao@gmail.com<mailto:garyzhao@gmail.com>>
wrote:
Hello

I have a three nodes cluster. I noticed one node was always down. Restarting Cassandra fixes
it but it will go down again after a couple of days. I'm pretty new to Cassandra so I'm wondering
how I should troubleshoot it. Logs is as below.

 INFO [StorageServiceShutdownHook] 2014-04-28 13:21:05,091 ThriftServer.java (line 141) Stop
listening to thrift clients
ERROR [GossipStage:4] 2014-04-28 13:11:59,877 CassandraDaemon.java (line 196) Exception in
thread Thread[GossipStage:4,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [ACCEPT-/10.20.132.44<http://10.20.132.44>] 2014-04-28 13:06:10,261 CassandraDaemon.java
(line 196) Exception in thread Thread[ACCEPT-/10.20.132.44<http://10.20.132.44>,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [GossipStage:3] 2014-04-28 12:54:02,116 CassandraDaemon.java (line 196) Exception in
thread Thread[GossipStage:3,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [GossipStage:2] 2014-04-28 12:52:27,644 CassandraDaemon.java (line 196) Exception in
thread Thread[GossipStage:2,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [Thread-222] 2014-04-28 12:50:18,689 CassandraDaemon.java (line 196) Exception in thread
Thread[Thread-222,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [GossipTasks:1] 2014-04-28 12:47:12,879 CassandraDaemon.java (line 196) Exception in
thread Thread[GossipTasks:1,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [GossipTasks:1] 2014-04-28 13:24:59,113 CassandraDaemon.java (line 196) Exception in
thread Thread[GossipTasks:1,5,main]
java.lang.IllegalThreadStateException
at java.lang.Thread.start(Thread.java:704)
at org.apache.cassandra.service.CassandraDaemon$2.uncaughtException(CassandraDaemon.java:202)
at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.handleOrLog(DebuggableThreadPoolExecutor.java:220)
at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:79)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
 INFO [StorageServiceShutdownHook] 2014-04-28 13:25:35,105 Server.java (line 181) Stop listening
for CQL clients
 INFO [StorageServiceShutdownHook] 2014-04-28 13:25:35,105 Gossiper.java (line 1251) Announcing
shutdown
 INFO [StorageServiceShutdownHook] 2014-04-28 13:26:12,524 MessagingService.java (line 667)
Waiting for messaging service to quiesce

Thanks
Gary



Mime
View raw message