cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vivek Mishra <mishra.v...@gmail.com>
Subject Re: node keeps dying
Date Thu, 25 Sep 2014 11:12:19 GMT
Increase heap size with Cassandra and try
On 25/09/2014 3:02 am, "Prem Yadav" <ipremyadav@gmail.com> wrote:
>
> BTW, thanks Michael.
> I am surprised why I didn't search for Cassandra oom before.
> I got some good links that discuss that. Will try to optimize and see how
it goes.
>
>
> On Wed, Sep 24, 2014 at 10:27 PM, Prem Yadav <ipremyadav@gmail.com> wrote:
>>
>> Well its not the Linux OOM killer. The system is running with all
default settings.
>>
>> Total memory 7GB- Cassandra gets assigned 2GB
>> 2 core processors.
>> Two rings with 3 nodes in each ring.
>>
>> On Wed, Sep 24, 2014 at 9:53 PM, Michael Shuler <michael@pbandjelly.org>
wrote:
>>>
>>> On 09/24/2014 11:32 AM, Prem Yadav wrote:
>>>>
>>>> this is an issue that has happened a few times. We are using DSE 4.0
>>>
>>>
>>> I believe this is Apache Cassandra 2.0.5, which is better info for this
list.
>>>
>>>> One of the Cassandra nodes is detected as dead by the opscenter even
>>>> though I can see the process is up.
>>>>
>>>> the logs show heap space error:
>>>>
>>>>   INFO [RMI TCP Connection(18270)-172.31.49.189] 2014-09-24
08:31:05,340
>>>> StorageService.java (line 2538) Starting repair command #30766,
>>>> repairing 1 ranges for keyspace <keyspace>
>>>> ERROR [BatchlogTasks:1] 2014-09-24 08:48:54,780 CassandraDaemon.java
>>>> (line 196) Exception in thread Thread[BatchlogTasks:1,5,main]
>>>> java.lang.OutOfMemoryError: Java heap space
>>>>          at java.util.ArrayList.<init>(Unknown Source)
>>>
>>>
>>> OOM.
>>>
>>> System environment and configuration modification details might be
helpful for others to give you advice. Searching for "cassandra oom" gave
me a few good links to read, and knowing some details about your nodes
might be really helpful. Additionally, CASSANDRA-7507 [0] suggests that an
OOM leaving the process running in an unclean state is not desired, and the
process should be killed.
>>>
>>> Several of the search links provide details on how to capture and dig
around a heap dump to aid in troubleshooting.
>>>
>>> [0] https://issues.apache.org/jira/browse/CASSANDRA-7507
>>> --
>>> Kind regards,
>>> Michael
>>
>>
>

Mime
View raw message