cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Prem Yadav <ipremya...@gmail.com>
Subject Re: node keeps dying
Date Wed, 24 Sep 2014 21:32:17 GMT
BTW, thanks Michael.
I am surprised why I didn't search for Cassandra oom before.
I got some good links that discuss that. Will try to optimize and see how
it goes.


On Wed, Sep 24, 2014 at 10:27 PM, Prem Yadav <ipremyadav@gmail.com> wrote:

> Well its not the Linux OOM killer. The system is running with all default
> settings.
>
> Total memory 7GB- Cassandra gets assigned 2GB
> 2 core processors.
> Two rings with 3 nodes in each ring.
>
> On Wed, Sep 24, 2014 at 9:53 PM, Michael Shuler <michael@pbandjelly.org>
> wrote:
>
>> On 09/24/2014 11:32 AM, Prem Yadav wrote:
>>
>>> this is an issue that has happened a few times. We are using DSE 4.0
>>>
>>
>> I believe this is Apache Cassandra 2.0.5, which is better info for this
>> list.
>>
>>  One of the Cassandra nodes is detected as dead by the opscenter even
>>> though I can see the process is up.
>>>
>>> the logs show heap space error:
>>>
>>>   INFO [RMI TCP Connection(18270)-172.31.49.189] 2014-09-24 08:31:05,340
>>> StorageService.java (line 2538) Starting repair command #30766,
>>> repairing 1 ranges for keyspace <keyspace>
>>> ERROR [BatchlogTasks:1] 2014-09-24 08:48:54,780 CassandraDaemon.java
>>> (line 196) Exception in thread Thread[BatchlogTasks:1,5,main]
>>> java.lang.OutOfMemoryError: Java heap space
>>>          at java.util.ArrayList.<init>(Unknown Source)
>>>
>>
>> OOM.
>>
>> System environment and configuration modification details might be
>> helpful for others to give you advice. Searching for "cassandra oom" gave
>> me a few good links to read, and knowing some details about your nodes
>> might be really helpful. Additionally, CASSANDRA-7507 [0] suggests that an
>> OOM leaving the process running in an unclean state is not desired, and the
>> process should be killed.
>>
>> Several of the search links provide details on how to capture and dig
>> around a heap dump to aid in troubleshooting.
>>
>> [0] https://issues.apache.org/jira/browse/CASSANDRA-7507
>> --
>> Kind regards,
>> Michael
>>
>
>

Mime
View raw message