cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Prem Yadav <ipremya...@gmail.com>
Subject Re: node keeps dying
Date Wed, 24 Sep 2014 21:27:54 GMT
Well its not the Linux OOM killer. The system is running with all default
settings.

Total memory 7GB- Cassandra gets assigned 2GB
2 core processors.
Two rings with 3 nodes in each ring.

On Wed, Sep 24, 2014 at 9:53 PM, Michael Shuler <michael@pbandjelly.org>
wrote:

> On 09/24/2014 11:32 AM, Prem Yadav wrote:
>
>> this is an issue that has happened a few times. We are using DSE 4.0
>>
>
> I believe this is Apache Cassandra 2.0.5, which is better info for this
> list.
>
>  One of the Cassandra nodes is detected as dead by the opscenter even
>> though I can see the process is up.
>>
>> the logs show heap space error:
>>
>>   INFO [RMI TCP Connection(18270)-172.31.49.189] 2014-09-24 08:31:05,340
>> StorageService.java (line 2538) Starting repair command #30766,
>> repairing 1 ranges for keyspace <keyspace>
>> ERROR [BatchlogTasks:1] 2014-09-24 08:48:54,780 CassandraDaemon.java
>> (line 196) Exception in thread Thread[BatchlogTasks:1,5,main]
>> java.lang.OutOfMemoryError: Java heap space
>>          at java.util.ArrayList.<init>(Unknown Source)
>>
>
> OOM.
>
> System environment and configuration modification details might be helpful
> for others to give you advice. Searching for "cassandra oom" gave me a few
> good links to read, and knowing some details about your nodes might be
> really helpful. Additionally, CASSANDRA-7507 [0] suggests that an OOM
> leaving the process running in an unclean state is not desired, and the
> process should be killed.
>
> Several of the search links provide details on how to capture and dig
> around a heap dump to aid in troubleshooting.
>
> [0] https://issues.apache.org/jira/browse/CASSANDRA-7507
> --
> Kind regards,
> Michael
>

Mime
View raw message