Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 8720 invoked from network); 31 Mar 2011 01:18:19 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 31 Mar 2011 01:18:19 -0000 Received: (qmail 47443 invoked by uid 500); 31 Mar 2011 00:50:18 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 47418 invoked by uid 500); 31 Mar 2011 00:50:18 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 47410 invoked by uid 99); 31 Mar 2011 00:50:18 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 00:50:18 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of anurag.gujral@gmail.com designates 209.85.214.172 as permitted sender) Received: from [209.85.214.172] (HELO mail-iw0-f172.google.com) (209.85.214.172) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 00:50:14 +0000 Received: by iwn39 with SMTP id 39so2087307iwn.31 for ; Wed, 30 Mar 2011 17:49:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=2+pJHuPhoWXl5sOZfJkjLSBcuUQJ/ZwHFk4Wu09oVsc=; b=xdwZ4Z4AY7O+3TCqW+rZ7aSkCLMv67xEwf7Cv3HxMT3AE4lqDE0WZvpMs5kLWAs+bv l3MvqnZfo1s0kgAPAApJhn2H4qH5KREGaypP/wcNtddeVBm2naxfIKQDEiWGkJCOrDvP V4p9banqslz+EXjc+Zs8P8+xLM8c/8JpZqaZY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=gKmG5luA0dlGJNrazTBWs3vw1wKmtTlVeoRLo8f3mCSb/W0NkkYjdaHHPQUtfftxWX 60s/TC/iNEMamPC/BlnirnZbkhzKCuso9Ms9vYk4TxIWRjOKkqy5xl9171Pbzlh/Dm8y nn6CIYEJhsCPkRbN3iIIINU6ITugdDO6CEp8M= MIME-Version: 1.0 Received: by 10.231.171.205 with SMTP id i13mr1995356ibz.181.1301532593535; Wed, 30 Mar 2011 17:49:53 -0700 (PDT) Received: by 10.231.153.136 with HTTP; Wed, 30 Mar 2011 17:49:53 -0700 (PDT) In-Reply-To: References: Date: Wed, 30 Mar 2011 17:49:53 -0700 Message-ID: Subject: Re: Fatal error from a cassandra node From: Anurag Gujral To: user@cassandra.apache.org Cc: Narendra Sharma Content-Type: multipart/alternative; boundary=0016369f9b18e6bc91049fbcab98 --0016369f9b18e6bc91049fbcab98 Content-Type: text/plain; charset=ISO-8859-1 I am using version 0.7.3 I am not sure what happened I tried connecting to the node and when I could not connect I looked at error logs i saw the log messages which I pasted below On Wed, Mar 30, 2011 at 5:04 PM, Narendra Sharma wrote: > OOM at startup with 16GB... seems like an issue. Which version are you > using? Can you provide some details on "failed node"? What exactly happened? > That might give some clue. Also, you might want to start with log level set > to debug to findout what more on what exactly Cassandra is doing that is > causing OOM. > > -Naren > > > On Wed, Mar 30, 2011 at 4:45 PM, Anurag Gujral wrote: > >> >> I am using 16G of heap space how much more should i increase. >> Please suggest >> >> Thanks >> Anurag >> >> On Wed, Mar 30, 2011 at 11:43 AM, Narendra Sharma < >> narendra.sharma@gmail.com> wrote: >> >>> http://wiki.apache.org/cassandra/MemtableThresholds#JVM_Heap_Size >>> >>> >>> >>> On Wed, Mar 30, 2011 at 11:41 AM, Peter Schuller < >>> peter.schuller@infidyne.com> wrote: >>> >>>> > I have 6 node cassandra cluster all are setup with same >>>> > configuration I am getting fatal exceptions in one of the nodes >>>> > ERROR [Thread-604] 2011-03-29 20:19:13,218 >>>> AbstractCassandraDaemon.java >>>> > (line 114) Fatal exception in thread Thread[Thread-604,5,main] >>>> > java.lang.OutOfMemoryError: Java heap space >>>> > ERROR [Thread-607] 2011-03-29 19:47:29,272 >>>> AbstractCassandraDaemon.java >>>> > (line 114) Fatal exception in thread Thread[Thread-607,5,main] >>>> > java.lang.OutOfMemoryError: Java heap space >>>> > ERROR [Thread-605] 2011-03-29 19:38:09,081 >>>> AbstractCassandraDaemon.java >>>> > (line 114) Fatal exception in thread Thread[Thread-605,5,main] >>>> > java.lang.OutOfMemoryError: Java heap space >>>> > ERROR [MutationStage:2] 2011-03-29 19:37:16,659 >>>> > DebuggableThreadPoolExecutor.java (line 103) Error in >>>> ThreadPoolExecutor >>>> > java.lang.OutOfMemoryError: Java heap space >>>> > ERROR [GossipStage:1] 2011-03-29 20:27:29,898 >>>> AbstractCassandraDaemon.java >>>> > (line 114) Fatal exception in thread Thread[GossipStage:1,5,main] >>>> > java.lang.OutOfMemoryError: Java heap space >>>> > >>>> > All all the nodes have 32 G of ram. >>>> > >>>> > Everytime I try to restart the failed node I get the above errors. >>>> >>>> Unless something is outright wrong, it sounds like you need to >>>> increase your JVM heap size in cassandra-env.sh. That you're getting >>>> it on start-up sounds consistent with commit log reply filling the >>>> heap in the form of memtables that are sized too big for your heap. >>>> >>>> There's a wiki page somewhere that describes the overall rule of thumb >>>> for heap sizing, but I can't find it right now. >>>> >>>> -- >>>> / Peter Schuller >>>> >>> >>> >>> >>> -- >>> Narendra Sharma >>> Solution Architect >>> *http://www.persistentsys.com* >>> *http://narendrasharma.blogspot.com/* >>> >>> >>> >> > > > -- > Narendra Sharma > Solution Architect > *http://www.persistentsys.com* > *http://narendrasharma.blogspot.com/* > > > --0016369f9b18e6bc91049fbcab98 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable I am using version 0.7.3 I am not sure what happened I tried connecting to = the node and=A0 when I could not connect I looked at error logs i saw the l= og messages which I pasted below

On Wed, = Mar 30, 2011 at 5:04 PM, Narendra Sharma <narendra.sharma@gmail.com> w= rote:
OOM at startup wi= th 16GB... seems like an issue. Which version are you using? Can you provid= e some details on "failed node"? What exactly happened? That migh= t give some clue. Also, you might want to start with log level set to debug= to findout what more on what exactly Cassandra is doing that is causing OO= M.

-Naren


On Wed, Mar 30, 2011 at 4:45 PM, Anurag Gujral <= anurag.gujral@gmail.com> wrote:

I am using 16G of heap space how much more should i increase.
Please= suggest

Thanks
Anurag

On Wed, Mar 30, 2011 at 11:43 AM, N= arendra Sharma <narendra.sharma@gmail.com> wrote:
http://wiki.apache.org/cassandra/MemtableThresholds#JVM_Heap_Size


On W= ed, Mar 30, 2011 at 11:41 AM, Peter Schuller <peter.schuller@in= fidyne.com> wrote:
<= div>> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 I have 6 node cassandra cluster all= are setup with same
> configuration=A0 =A0 I am getting fatal exceptions in one of the nodes=
> ERROR [Thread-604] 2011-03-29 20:19:13,218 AbstractCassandraDaemon.jav= a
> (line 114) Fatal exception in thread Thread[Thread-604,5,main]
> java.lang.OutOfMemoryError: Java heap space
> ERROR [Thread-607] 2011-03-29 19:47:29,272 AbstractCassandraDaemon.jav= a
> (line 114) Fatal exception in thread Thread[Thread-607,5,main]
> java.lang.OutOfMemoryError: Java heap space
> ERROR [Thread-605] 2011-03-29 19:38:09,081 AbstractCassandraDaemon.jav= a
> (line 114) Fatal exception in thread Thread[Thread-605,5,main]
> java.lang.OutOfMemoryError: Java heap space
> ERROR [MutationStage:2] 2011-03-29 19:37:16,659
> DebuggableThreadPoolExecutor.java (line 103) Error in ThreadPoolExecut= or
> java.lang.OutOfMemoryError: Java heap space
> ERROR [GossipStage:1] 2011-03-29 20:27:29,898 AbstractCassandraDaemon.= java
> (line 114) Fatal exception in thread Thread[GossipStage:1,5,main]
> java.lang.OutOfMemoryError: Java heap space
>
> All all the nodes have 32 G of ram.
>
> Everytime I try to restart the failed node I get the above errors.

Unless something is outright wrong, it sounds like you need to<= br> increase your JVM heap size in cassandra-env.sh. That you're getting it on start-up sounds consistent with commit log reply filling the
heap in the form of memtables that are sized too big for your heap.

There's a wiki page somewhere that describes the overall rule of thumb<= br> for heap sizing, but I can't find it right now.

--
/ Peter Schuller



--
Narendra Sharma




--
Narendra Sharma

--0016369f9b18e6bc91049fbcab98--