incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gary Zhao <garyz...@gmail.com>
Subject Re: java.lang.OutOfMemoryError: Java heap space
Date Tue, 29 Apr 2014 00:34:17 GMT
Yes, they are virtual machines, but we are using KVM. Is there any
solutions for this issue or we should use physical machines?


On Mon, Apr 28, 2014 at 10:38 AM, Prem Yadav <ipremyadav@gmail.com> wrote:

> Are the virtual machines? The last time I had this issues was because of
> VMWare "ballooning".
> If not, what versions of Cassandra and Java are you using?
>
>
> On Mon, Apr 28, 2014 at 6:30 PM, Gary Zhao <garyzhao@gmail.com> wrote:
>
>> BTW, the CPU usage on this node is pretty high, but data size is pretty
>> small.
>>
>>   PID USERNAME  THR PRI NICE  SIZE   RES   SHR STATE   TIME    CPU COMMAND
>> 28674 cassandr   89  25    0 9451M 8970M  525M sleep  32.1H   329% java
>>
>> UN  8.92 GB    256     35.4%  c2d9d02e-bdb3-47cb-af1b-eabc2eeb503b  rack1
>> UN  5.32 GB    256     33.1%  874a269f-96bc-4db6-9d15-7cac6771a2ea  rack1
>> DN  339.94 MB  256     31.5%  fba5b071-aee1-451f-ae42-49c3bbe70bbe  rack1
>>
>>
>> On Mon, Apr 28, 2014 at 10:25 AM, Gary Zhao <garyzhao@gmail.com> wrote:
>>
>>> Hello
>>>
>>> I have a three nodes cluster. I noticed one node was always down.
>>> Restarting Cassandra fixes it but it will go down again after a couple of
>>> days. I'm pretty new to Cassandra so I'm wondering how I should
>>> troubleshoot it. Logs is as below.
>>>
>>>  INFO [StorageServiceShutdownHook] 2014-04-28 13:21:05,091
>>> ThriftServer.java (line 141) Stop listening to thrift clients
>>> ERROR [GossipStage:4] 2014-04-28 13:11:59,877 CassandraDaemon.java (line
>>> 196) Exception in thread Thread[GossipStage:4,5,main]
>>> java.lang.OutOfMemoryError: Java heap space
>>> ERROR [ACCEPT-/10.20.132.44] 2014-04-28 13:06:10,261
>>> CassandraDaemon.java (line 196) Exception in thread Thread[ACCEPT-/
>>> 10.20.132.44,5,main]
>>> java.lang.OutOfMemoryError: Java heap space
>>> ERROR [GossipStage:3] 2014-04-28 12:54:02,116 CassandraDaemon.java (line
>>> 196) Exception in thread Thread[GossipStage:3,5,main]
>>> java.lang.OutOfMemoryError: Java heap space
>>> ERROR [GossipStage:2] 2014-04-28 12:52:27,644 CassandraDaemon.java (line
>>> 196) Exception in thread Thread[GossipStage:2,5,main]
>>> java.lang.OutOfMemoryError: Java heap space
>>> ERROR [Thread-222] 2014-04-28 12:50:18,689 CassandraDaemon.java (line
>>> 196) Exception in thread Thread[Thread-222,5,main]
>>> java.lang.OutOfMemoryError: Java heap space
>>> ERROR [GossipTasks:1] 2014-04-28 12:47:12,879 CassandraDaemon.java (line
>>> 196) Exception in thread Thread[GossipTasks:1,5,main]
>>> java.lang.OutOfMemoryError: Java heap space
>>> ERROR [GossipTasks:1] 2014-04-28 13:24:59,113 CassandraDaemon.java (line
>>> 196) Exception in thread Thread[GossipTasks:1,5,main]
>>> java.lang.IllegalThreadStateException
>>> at java.lang.Thread.start(Thread.java:704)
>>>  at
>>> org.apache.cassandra.service.CassandraDaemon$2.uncaughtException(CassandraDaemon.java:202)
>>> at
>>> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.handleOrLog(DebuggableThreadPoolExecutor.java:220)
>>>  at
>>> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:79)
>>> at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>>  at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> at java.lang.Thread.run(Thread.java:744)
>>>  INFO [StorageServiceShutdownHook] 2014-04-28 13:25:35,105 Server.java
>>> (line 181) Stop listening for CQL clients
>>>  INFO [StorageServiceShutdownHook] 2014-04-28 13:25:35,105 Gossiper.java
>>> (line 1251) Announcing shutdown
>>>  INFO [StorageServiceShutdownHook] 2014-04-28 13:26:12,524
>>> MessagingService.java (line 667) Waiting for messaging service to quiesce
>>>
>>> Thanks
>>> Gary
>>>
>>
>>
>

Mime
View raw message