Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4A858641A for ; Tue, 12 Jul 2011 22:22:42 +0000 (UTC) Received: (qmail 46324 invoked by uid 500); 12 Jul 2011 22:22:40 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 46250 invoked by uid 500); 12 Jul 2011 22:22:39 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 46238 invoked by uid 99); 12 Jul 2011 22:22:39 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Jul 2011 22:22:38 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of anurag.gujral@gmail.com designates 74.125.83.44 as permitted sender) Received: from [74.125.83.44] (HELO mail-gw0-f44.google.com) (74.125.83.44) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Jul 2011 22:22:31 +0000 Received: by gwb20 with SMTP id 20so2557068gwb.31 for ; Tue, 12 Jul 2011 15:22:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=nmst/hvYL2b1fI9FZzZulm/f8wPFN18GlCo0vJOfQII=; b=nNnllXW0icLDIut+w+QsL0mh/XHuBdQGLV2WmwC4jyMT0g8Ug0LzKz5qZth3jjmcrt NTGXMwe8WoWpQg1yfMYtByFNYa0F1KR6vXDreQh7N/JDoiQv6xMqsLAQAb0bcnmYpJ7e zDgRw8GbPkMfGeO3Q1RLpJ7Os/k7wQhuU+6WE= MIME-Version: 1.0 Received: by 10.236.79.134 with SMTP id i6mr700819yhe.75.1310509330683; Tue, 12 Jul 2011 15:22:10 -0700 (PDT) Received: by 10.236.95.13 with HTTP; Tue, 12 Jul 2011 15:22:10 -0700 (PDT) In-Reply-To: References: Date: Tue, 12 Jul 2011 15:22:10 -0700 Message-ID: Subject: Re: Out of memory error in cassandra From: Anurag Gujral To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=20cf3005104421580604a7e6bbb6 X-Virus-Checked: Checked by ClamAV on apache.org --20cf3005104421580604a7e6bbb6 Content-Type: text/plain; charset=ISO-8859-1 Hi Jonathan, Thanks for your mail. But no-one of the things mentioned in the link pertains to OOM error I we are seeing. thanks Anurag On Tue, Jul 12, 2011 at 10:42 AM, Jonathan Ellis wrote: > Have you seen > http://www.datastax.com/docs/0.8/troubleshooting/index#nodes-are-dying-with-oom-errors > ? > > On Mon, Jul 11, 2011 at 1:55 PM, Anurag Gujral > wrote: > > Hi All, > > I am getting following error from cassandra: > > ERROR [ReadStage:23] 2011-07-10 17:19:18,300 > > DebuggableThreadPoolExecutor.java (line 103) Error in ThreadPoolExecutor > > java.lang.OutOfMemoryError: Java heap space > > at > > > org.apache.cassandra.utils.BloomFilterSerializer.deserialize(BloomFilterSerializer.java:49) > > at > > > org.apache.cassandra.utils.BloomFilterSerializer.deserialize(BloomFilterSerializer.java:30) > > at > > > org.apache.cassandra.io.sstable.IndexHelper.defreezeBloomFilter(IndexHelper.java:117) > > at > > > org.apache.cassandra.io.sstable.IndexHelper.defreezeBloomFilter(IndexHelper.java:94) > > at > > > org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:107) > > at > > > org.apache.cassandra.db.columniterator.SSTableNamesIterator.(SSTableNamesIterator.java:72) > > at > > > org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:59) > > at > > > org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80) > > at > > > org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1311) > > at > > > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1203) > > at > > > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1131) > > at org.apache.cassandra.db.Table.getRow(Table.java:333) > > at > > > org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:60) > > at > > org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:69) > > at > > > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72) > > at > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > > at java.lang.Thread.run(Thread.java:636) > > INFO [ScheduledTasks:1] 2011-07-10 17:19:18,306 StatusLogger.java (line > 66) > > RequestResponseStage 0 0 > > ERROR [ReadStage:23] 2011-07-10 17:19:18,306 AbstractCassandraDaemon.java > > (line 114) Fatal exception in thread Thread[ReadStage:23,5,main] > > java.lang.OutOfMemoryError: Java heap space > > at > > > org.apache.cassandra.utils.BloomFilterSerializer.deserialize(BloomFilterSerializer.java:49) > > at > > > org.apache.cassandra.utils.BloomFilterSerializer.deserialize(BloomFilterSerializer.java:30) > > at > > > org.apache.cassandra.io.sstable.IndexHelper.defreezeBloomFilter(IndexHelper.java:117) > > at > > > org.apache.cassandra.io.sstable.IndexHelper.defreezeBloomFilter(IndexHelper.java:94) > > at > > > org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:107) > > at > > > org.apache.cassandra.db.columniterator.SSTableNamesIterator.(SSTableNamesIterator.java:72) > > at > > > org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:59) > > at > > > org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80) > > at > > > org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1311) > > at > > > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1203) > > at > > > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1131) > > > > > > Can someone please help debug this? The maximum heap size is 28G . > > > > I am not sure why cassandra is giving Out of memory error here. > > > > Thanks > > Anurag > > > > > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of DataStax, the source for professional Cassandra support > http://www.datastax.com > --20cf3005104421580604a7e6bbb6 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi Jonathan,
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 T= hanks for=A0 your mail. But no-one of the things mentioned in the link pert= ains to OOM error I we are seeing.
thanks
Anurag

On Tue, Jul 12, 2011 at 10:42 AM, Jonathan Ellis <jbellis@gmail.com> wrote:
Have you seen http://www.datastax.com/docs/0.8/tro= ubleshooting/index#nodes-are-dying-with-oom-errors
?

On Mon, Jul 11, 2011 at 1:55 PM, Anurag Gujral <anurag.gujral@gmail.com> wrote:
> Hi All,
> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 I am getting following error from cassa= ndra:
> ERROR [ReadStage:23] 2011-07-10 17:19:18,300
> DebuggableThreadPoolExecutor.java (line 103) Error in ThreadPoolExecut= or
> java.lang.OutOfMemoryError: Java heap space
> =A0=A0=A0=A0=A0=A0=A0 at
> org.apache.cassandra.utils.BloomFilterSerializer.deserialize(BloomFilt= erSerializer.java:49)
> =A0=A0=A0=A0=A0=A0=A0 at
> org.apache.cassandra.utils.BloomFilterSerializer.deserialize(BloomFilt= erSerializer.java:30)
> =A0=A0=A0=A0=A0=A0=A0 at
> org.apache.cassandra.io.sstable.IndexHelper.defreezeBloomFilter(IndexH= elper.java:117)
> =A0=A0=A0=A0=A0=A0=A0 at
> org.apache.cassandra.io.sstable.IndexHelper.defreezeBloomFilter(IndexH= elper.java:94)
> =A0=A0=A0=A0=A0=A0=A0 at
> org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTab= leNamesIterator.java:107)
> =A0=A0=A0=A0=A0=A0=A0 at
> org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init&g= t;(SSTableNamesIterator.java:72)
> =A0=A0=A0=A0=A0=A0=A0 at
> org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterat= or(NamesQueryFilter.java:59)
> =A0=A0=A0=A0=A0=A0=A0 at
> org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(Qu= eryFilter.java:80)
> =A0=A0=A0=A0=A0=A0=A0 at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFam= ilyStore.java:1311)
> =A0=A0=A0=A0=A0=A0=A0 at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamily= Store.java:1203)
> =A0=A0=A0=A0=A0=A0=A0 at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamily= Store.java:1131)
> =A0=A0=A0=A0=A0=A0=A0 at org.apache.cassandra.db.Table.getRow(Table.ja= va:333)
> =A0=A0=A0=A0=A0=A0=A0 at
> org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesRea= dCommand.java:60)
> =A0=A0=A0=A0=A0=A0=A0 at
> org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:69= )
> =A0=A0=A0=A0=A0=A0=A0 at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.j= ava:72)
> =A0=A0=A0=A0=A0=A0=A0 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j= ava:1110)
> =A0=A0=A0=A0=A0=A0=A0 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.= java:603)
> =A0=A0=A0=A0=A0=A0=A0 at java.lang.Thread.run(Thread.java:636)
> =A0INFO [ScheduledTasks:1] 2011-07-10 17:19:18,306 StatusLogger.java (= line 66)
> RequestResponseStage=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0= =A0=A0=A0=A0=A0 0
> ERROR [ReadStage:23] 2011-07-10 17:19:18,306 AbstractCassandraDaemon.j= ava
> (line 114) Fatal exception in thread Thread[ReadStage:23,5,main]
> java.lang.OutOfMemoryError: Java heap space
> =A0=A0=A0=A0=A0=A0=A0 at
> org.apache.cassandra.utils.BloomFilterSerializer.deserialize(BloomFilt= erSerializer.java:49)
> =A0=A0=A0=A0=A0=A0=A0 at
> org.apache.cassandra.utils.BloomFilterSerializer.deserialize(BloomFilt= erSerializer.java:30)
> =A0=A0=A0=A0=A0=A0=A0 at
> org.apache.cassandra.io.sstable.IndexHelper.defreezeBloomFilter(IndexH= elper.java:117)
> =A0=A0=A0=A0=A0=A0=A0 at
> org.apache.cassandra.io.sstable.IndexHelper.defreezeBloomFilter(IndexH= elper.java:94)
> =A0=A0=A0=A0=A0=A0=A0 at
> org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTab= leNamesIterator.java:107)
> =A0=A0=A0=A0=A0=A0=A0 at
> org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init&g= t;(SSTableNamesIterator.java:72)
> =A0=A0=A0=A0=A0=A0=A0 at
> org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterat= or(NamesQueryFilter.java:59)
> =A0=A0=A0=A0=A0=A0=A0 at
> org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(Qu= eryFilter.java:80)
> =A0=A0=A0=A0=A0=A0=A0 at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFam= ilyStore.java:1311)
> =A0=A0=A0=A0=A0=A0=A0 at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamily= Store.java:1203)
> =A0=A0=A0=A0=A0=A0=A0 at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamily= Store.java:1131)
>
>
> Can someone please help debug this? The maximum heap size is 28G .
>
> I am not sure why cassandra is giving Out of memory error here.
>
> Thanks
> Anurag
>



--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.c= om

--20cf3005104421580604a7e6bbb6--