Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2793A9026 for ; Fri, 22 Jun 2012 08:34:44 +0000 (UTC) Received: (qmail 18851 invoked by uid 500); 22 Jun 2012 08:34:42 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 18475 invoked by uid 500); 22 Jun 2012 08:34:41 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 18446 invoked by uid 99); 22 Jun 2012 08:34:40 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Jun 2012 08:34:40 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FSL_RCVD_USER,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of ravikumar.govindarajan@gmail.com designates 209.85.213.43 as permitted sender) Received: from [209.85.213.43] (HELO mail-yw0-f43.google.com) (209.85.213.43) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Jun 2012 08:34:33 +0000 Received: by yhkk6 with SMTP id k6so1583655yhk.30 for ; Fri, 22 Jun 2012 01:34:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=kHcmwol+B642wO2U16OyX+e/4xrxhqKMejoI4AmNvhY=; b=pu4V6/gCQebSVrYFT9o3+GgzCtEj0hn+1qoOEf7o9gwB09nfHcFqijDsreL4k54vNO pQHAfMUJeXPTRwJ4kWWJYOBRaLUJPBjuMt6d9YG+cOCNVO8AdHqF2rq/8pue9WFHcnEt shCmrpmhhN4KgKX3y8hjq+2rdRguQX2CoR3DzBxiU4RI6jo7KCbSPgmF4tJcaSgLqGWq FkX62X134NPXwUfb/FbCQiwhaliOIx240pkAV9jrLWITB1N2rwOGHPeow9d21X/RHFD3 eWubYP6dbF9cS+S2llDlIW57P4EbpzV6NutgjHjK2/D0WvONPhZOB462LCu7w7fJhBZC qyuA== MIME-Version: 1.0 Received: by 10.236.78.36 with SMTP id f24mr1488527yhe.20.1340354052253; Fri, 22 Jun 2012 01:34:12 -0700 (PDT) Received: by 10.147.38.12 with HTTP; Fri, 22 Jun 2012 01:34:12 -0700 (PDT) In-Reply-To: References: Date: Fri, 22 Jun 2012 14:04:12 +0530 Message-ID: Subject: Re: MurmurHash NPE during compaction From: Ravikumar Govindarajan To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=20cf300fab9128671404c30b7f62 --20cf300fab9128671404c30b7f62 Content-Type: text/plain; charset=ISO-8859-1 Thanks Aaron. Created a ticket https://issues.apache.org/jira/browse/CASSANDRA-4367 Funnny thing is, I don't see any of the SSTables that participated in the failed compaction. Will do an upgradesstables and find out if problem still persists On Mon, Jun 18, 2012 at 6:43 AM, aaron morton wrote: > Can you please create a ticket on > https://issues.apache.org/jira/browse/CASSANDRA > > Please include: > * CF definition including the bloom_filter_fp_chance > * If the data was upgraded from a previous version of cassandra. > * The names of the files that were being compacted. > > As a work around you can try using nodetool upgradetables to re-write the > files - this may also fail, but its could be worth trying. > > The next step would be to remove determine which files were causing the > issue (looking at the logs) and remove them from the data directory. Then > run repair to restore consistency. > > Cheers > > ----------------- > Aaron Morton > Freelance Developer > @aaronmorton > http://www.thelastpickle.com > > On 14/06/2012, at 11:38 PM, Ravikumar Govindarajan wrote: > > We received the following NPE during compaction of a large row. We are on > cassandra-1.0.7. Need some help here to find the root cause of the issue > > ERROR [CompactionExecutor:595] 2012-06-13 09:44:46,718 > AbstractCassandraDaemon.java (line 139) Fatal exception in thread > Thread[CompactionExecutor:595,1,main] > java.lang.NullPointerException > at > org.apache.cassandra.utils.MurmurHash.hash64(MurmurHash.java:102) > at > org.apache.cassandra.utils.BloomFilter.getHashBuckets(BloomFilter.java:103) > at > org.apache.cassandra.utils.BloomFilter.getHashBuckets(BloomFilter.java:92) > at org.apache.cassandra.utils.BloomFilter.add(BloomFilter.java:114) > at > org.apache.cassandra.db.ColumnIndexer.serialize(ColumnIndexer.java:96) > at > org.apache.cassandra.db.ColumnIndexer.serialize(ColumnIndexer.java:51) > at > org.apache.cassandra.db.compaction.PrecompactedRow.write(PrecompactedRow.java:135) > at > org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160) > at > org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:159) > at > org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:134) > at > org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114) > at > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) > at java.lang.Thread.run(Thread.java:619) > > Thanks and Regards, > Ravi > > > --20cf300fab9128671404c30b7f62 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Thanks Aaron.


Funnny thing is, I don't= see any of the SSTables that participated in the failed compaction.=A0

Will do an upgradesstables and find out if problem stil= l persists


On = Mon, Jun 18, 2012 at 6:43 AM, aaron morton <aaron@thelastpickle.com<= /a>> wrote:

Please include:
* CF definition including the= bloom_filter_fp_chance
* If the data was upgraded from a previou= s version of cassandra.=A0
* The names of the files that were bei= ng compacted.=A0

As a work around you can try using=A0nodetool upgradeta= bles to re-write the files - this may also fail, but its could be worth try= ing.=A0

The next step would be to remove determine= which files were causing the issue (looking at the logs) and remove them f= rom the data directory. Then run repair to restore consistency.=A0

Cheers

<= div style=3D"word-wrap:break-word">
-----------------
Aaron Morton
Freelance Deve= loper
@aaronmorton

On 14/06/2012, at 11:38 PM, Ravikumar Govindarajan wrote:
We received the following NPE during compac= tion of a large row. We are on cassandra-1.0.7. Need some help here to find= the root cause of the issue

=A0ERROR [CompactionExecutor:595] 2012-06-13 09:4= 4:46,718 AbstractCassandraDaemon.java (line 139) Fatal exception in thread = Thread[CompactionExecutor:595,1,main]
java.lang.NullPointerException
=A0 =A0 =A0 =A0 at org.apache= .cassandra.utils.MurmurHash.hash64(MurmurHash.java:102)
=A0 =A0 = =A0 =A0 at org.apache.cassandra.utils.BloomFilter.getHashBuckets(BloomFilte= r.java:103)
=A0 =A0 =A0 =A0 at org.apache.cassandra.utils.BloomFilter.getHashBucke= ts(BloomFilter.java:92)
=A0 =A0 =A0 =A0 at org.apache.cassandra.u= tils.BloomFilter.add(BloomFilter.java:114)
=A0 =A0 =A0 =A0 at org= .apache.cassandra.db.ColumnIndexer.serialize(ColumnIndexer.java:96)
=A0 =A0 =A0 =A0 at org.apache.cassandra.db.ColumnIndexer.serialize(Col= umnIndexer.java:51)
=A0 =A0 =A0 =A0 at org.apache.cassandra.db.co= mpaction.PrecompactedRow.write(PrecompactedRow.java:135)
=A0 =A0 = =A0 =A0 at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWrit= er.java:160)
=A0 =A0 =A0 =A0 at org.apache.cassandra.db.compaction.CompactionTask.e= xecute(CompactionTask.java:159)
=A0 =A0 =A0 =A0 at org.apache.cas= sandra.db.compaction.CompactionManager$1.call(CompactionManager.java:134)
=A0 =A0 =A0 =A0 at org.apache.cassandra.db.compaction.CompactionMa= nager$1.call(CompactionManager.java:114)
=A0 =A0 =A0 =A0 at java.util.concurrent.FutureTask$Sync.innerRun(Futur= eTask.java:303)
=A0 =A0 =A0 =A0 at java.util.concurrent.FutureTas= k.run(FutureTask.java:138)
=A0 =A0 =A0 =A0 at java.util.concurren= t.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
=A0 =A0 =A0 =A0 at java.util.concurrent.ThreadPoolExecutor$Worker.run(= ThreadPoolExecutor.java:907)
=A0 =A0 =A0 =A0 at java.lang.Thread.= run(Thread.java:619)

Thanks and Regards,
Ravi


--20cf300fab9128671404c30b7f62--