incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anurag Gujral <anurag.guj...@gmail.com>
Subject Re: Fatal error from a cassandra node
Date Thu, 31 Mar 2011 00:49:53 GMT
I am using version 0.7.3 I am not sure what happened I tried connecting to
the node and  when I could not connect I looked at error logs i saw the log
messages which I pasted below

On Wed, Mar 30, 2011 at 5:04 PM, Narendra Sharma
<narendra.sharma@gmail.com>wrote:

> OOM at startup with 16GB... seems like an issue. Which version are you
> using? Can you provide some details on "failed node"? What exactly happened?
> That might give some clue. Also, you might want to start with log level set
> to debug to findout what more on what exactly Cassandra is doing that is
> causing OOM.
>
> -Naren
>
>
> On Wed, Mar 30, 2011 at 4:45 PM, Anurag Gujral <anurag.gujral@gmail.com>wrote:
>
>>
>> I am using 16G of heap space how much more should i increase.
>> Please suggest
>>
>> Thanks
>> Anurag
>>
>> On Wed, Mar 30, 2011 at 11:43 AM, Narendra Sharma <
>> narendra.sharma@gmail.com> wrote:
>>
>>> http://wiki.apache.org/cassandra/MemtableThresholds#JVM_Heap_Size
>>>
>>>
>>>
>>> On Wed, Mar 30, 2011 at 11:41 AM, Peter Schuller <
>>> peter.schuller@infidyne.com> wrote:
>>>
>>>> >            I have 6 node cassandra cluster all are setup with same
>>>> > configuration    I am getting fatal exceptions in one of the nodes
>>>> > ERROR [Thread-604] 2011-03-29 20:19:13,218
>>>> AbstractCassandraDaemon.java
>>>> > (line 114) Fatal exception in thread Thread[Thread-604,5,main]
>>>> > java.lang.OutOfMemoryError: Java heap space
>>>> > ERROR [Thread-607] 2011-03-29 19:47:29,272
>>>> AbstractCassandraDaemon.java
>>>> > (line 114) Fatal exception in thread Thread[Thread-607,5,main]
>>>> > java.lang.OutOfMemoryError: Java heap space
>>>> > ERROR [Thread-605] 2011-03-29 19:38:09,081
>>>> AbstractCassandraDaemon.java
>>>> > (line 114) Fatal exception in thread Thread[Thread-605,5,main]
>>>> > java.lang.OutOfMemoryError: Java heap space
>>>> > ERROR [MutationStage:2] 2011-03-29 19:37:16,659
>>>> > DebuggableThreadPoolExecutor.java (line 103) Error in
>>>> ThreadPoolExecutor
>>>> > java.lang.OutOfMemoryError: Java heap space
>>>> > ERROR [GossipStage:1] 2011-03-29 20:27:29,898
>>>> AbstractCassandraDaemon.java
>>>> > (line 114) Fatal exception in thread Thread[GossipStage:1,5,main]
>>>> > java.lang.OutOfMemoryError: Java heap space
>>>> >
>>>> > All all the nodes have 32 G of ram.
>>>> >
>>>> > Everytime I try to restart the failed node I get the above errors.
>>>>
>>>> Unless something is outright wrong, it sounds like you need to
>>>> increase your JVM heap size in cassandra-env.sh. That you're getting
>>>> it on start-up sounds consistent with commit log reply filling the
>>>> heap in the form of memtables that are sized too big for your heap.
>>>>
>>>> There's a wiki page somewhere that describes the overall rule of thumb
>>>> for heap sizing, but I can't find it right now.
>>>>
>>>> --
>>>> / Peter Schuller
>>>>
>>>
>>>
>>>
>>> --
>>> Narendra Sharma
>>> Solution Architect
>>> *http://www.persistentsys.com*
>>> *http://narendrasharma.blogspot.com/*
>>>
>>>
>>>
>>
>
>
> --
> Narendra Sharma
> Solution Architect
> *http://www.persistentsys.com*
> *http://narendrasharma.blogspot.com/*
>
>
>

Mime
View raw message