Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5533810466 for ; Thu, 25 Jul 2013 05:11:18 +0000 (UTC) Received: (qmail 45197 invoked by uid 500); 25 Jul 2013 05:11:15 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 45167 invoked by uid 500); 25 Jul 2013 05:11:14 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 45154 invoked by uid 99); 25 Jul 2013 05:11:12 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Jul 2013 05:11:12 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a81.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Jul 2013 05:11:04 +0000 Received: from homiemail-a81.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a81.g.dreamhost.com (Postfix) with ESMTP id 7FDDEA806E for ; Wed, 24 Jul 2013 22:10:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :content-type:message-id:mime-version:subject:date:references:to :in-reply-to; s=thelastpickle.com; bh=uJuZZJzFNPfLVbcR69ll9sucy1 U=; b=K88e+n62iILJ7uWDdUizmUmbJ7c8z3FKTr2KSEXduP1+V6ikLdUZvYUIir pWO8pQXxmu2+Abj3NQaUbSh3JQ2DaBtWs6EpHQOC64f/8YsEj61tfDzzZAm+Ij7d lksjL/nRfBA0AVZ6/QodaQciEtAuoK9l0J5TKMKdRQXi5RBJc= Received: from [172.16.1.7] (unknown [203.86.207.101]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a81.g.dreamhost.com (Postfix) with ESMTPSA id 99230A806C for ; Wed, 24 Jul 2013 22:10:42 -0700 (PDT) From: aaron morton Content-Type: multipart/alternative; boundary="Apple-Mail=_AE23185E-F786-4189-AE30-A56A15F2E1C2" Message-Id: <12ACDC5A-D55B-4D20-AA8B-E2ABD562D2C4@thelastpickle.com> Mime-Version: 1.0 (Mac OS X Mail 6.5 \(1508\)) Subject: Re: NPE in CompactionExecutor Date: Thu, 25 Jul 2013 17:10:40 +1200 References: To: user@cassandra.apache.org In-Reply-To: X-Mailer: Apple Mail (2.1508) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_AE23185E-F786-4189-AE30-A56A15F2E1C2 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 > There was no error stack, just that line in the log. It's odd that the stack is not there.=20 This is an unhanded exception when running compaction. It may be related = to the assertions.=20 If you can reproduce it please raise a ticket at = https://issues.apache.org/jira/browse/CASSANDRA Cheers ----------------- Aaron Morton Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 24/07/2013, at 3:50 AM, Paul Ingalls wrote: > I'm running the latest from the 1.2 branch as of a few days ago. I = needed one of the patches that will be in 1.2.7 >=20 > There was no error stack, just that line in the log. >=20 > I wiped the database (deleted all the files in the lib dir) and = restarted my data load, and am consistently running into the incorrect = row data size error, almost immediately=85 It seems to be specific to = compacting large rows. I have been unsuccessful in getting a large row = to compact=85 >=20 > Paul >=20 > On Jul 21, 2013, at 1:42 PM, aaron morton = wrote: >=20 >> What version are you running ?=20 >>=20 >>> ERROR [CompactionExecutor:38] 2013-07-19 17:01:34,494 = CassandraDaemon.java (line 192) Exception in thread = Thread[CompactionExecutor:38,1,main] >>> java.lang.NullPointerException >> What' the full error stack ?=20 >>=20 >>> Not sure if this is related or not, but I'm also getting a bunch of = AssertionErrors as well, even after running a scrub=85 >>>=20 >>> ERROR [CompactionExecutor:38] 2013-07-19 17:01:06,192 = CassandraDaemon.java (line 192) Exception in thread = Thread[CompactionExecutor:38,1,main] >>> java.lang.AssertionError: incorrect row data size 29502477 written = to = /mnt/datadrive/lib/cassandra/data/fanzo/tweets_by_team/fanzo-tweets_by_tea= m-tmp-ic-5262-Data.db; correct is 29725806 >> Double check that the scrub was successful.=20 >>=20 >> If it's not detecting / fixing the problem look for previous log = messages from that thread [CompactionExecutor:38] and see what sstables = it was compacting. Try remove those. But I would give scrub another = chance to get it sorted.=20 >>=20 >> Cheers >>=20 >> ----------------- >> Aaron Morton >> Cassandra Consultant >> New Zealand >>=20 >> @aaronmorton >> http://www.thelastpickle.com >>=20 >> On 20/07/2013, at 5:04 AM, Paul Ingalls = wrote: >>=20 >>> I'm seeing a number of NullPointerExceptions in the log of my = cluster. You can see the log line below. I'm thinking this is probably = bad. Any ideas? >>>=20 >>> ERROR [CompactionExecutor:38] 2013-07-19 17:01:34,494 = CassandraDaemon.java (line 192) Exception in thread = Thread[CompactionExecutor:38,1,main] >>> java.lang.NullPointerException >>>=20 >>> Not sure if this is related or not, but I'm also getting a bunch of = AssertionErrors as well, even after running a scrub=85 >>>=20 >>> ERROR [CompactionExecutor:38] 2013-07-19 17:01:06,192 = CassandraDaemon.java (line 192) Exception in thread = Thread[CompactionExecutor:38,1,main] >>> java.lang.AssertionError: incorrect row data size 29502477 written = to = /mnt/datadrive/lib/cassandra/data/fanzo/tweets_by_team/fanzo-tweets_by_tea= m-tmp-ic-5262-Data.db; correct is 29725806 >>> at = org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:16= 2) >>> at = org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.j= ava:162) >>> at = org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnab= le.java:48) >>> at = org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) >>> at = org.apache.cassandra.db.compaction.CompactionTask.executeInternal(Compacti= onTask.java:58) >>> at = org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(Abstract= CompactionTask.java:60) >>> at = org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionT= ask.run(CompactionManager.java:211) >>> at = java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>> at = java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:166) >>> at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1145) >>> at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :615) >>> at java.lang.Thread.run(Thread.java:724) >>>=20 >>>=20 >>=20 >=20 --Apple-Mail=_AE23185E-F786-4189-AE30-A56A15F2E1C2 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=windows-1252
There = was no error stack, just that line in the = log.
It's = odd that the stack is not there. 

This = is an unhanded exception when running compaction. It may be related to = the assertions. 
If you can reproduce it = please raise a ticket at https://issues.ap= ache.org/jira/browse/CASSANDRA


http://www.thelastpickle.com

On 24/07/2013, at 3:50 AM, Paul Ingalls <paulingalls@gmail.com> = wrote:

aaron@thelastpickle.com> = wrote:

What = version are you running ? 

ERROR [CompactionExecutor:38] 2013-07-19 17:01:34,494 = CassandraDaemon.java (line 192) Exception in thread = Thread[CompactionExecutor:38,1,main]
java.lang.NullPointerException
=
What' the full error stack = ? 

Not sure if = this is related or not, but I'm also getting a bunch of AssertionErrors = as well, even after running a scrub=85

ERROR = [CompactionExecutor:38] 2013-07-19 17:01:06,192 CassandraDaemon.java = (line 192) Exception in thread = Thread[CompactionExecutor:38,1,main]
java.lang.AssertionError: = incorrect row data size 29502477 written to = /mnt/datadrive/lib/cassandra/data/fanzo/tweets_by_team/fanzo-tweets_by_tea= m-tmp-ic-5262-Data.db; correct is 29725806
Double check that = the scrub was successful. 

If it's not = detecting / fixing the problem look for previous log messages from that = thread  [CompactionExecutor:38] and see what sstables it = was compacting. Try remove those. But I would give scrub another chance = to get it = sorted. 

Cheers

http://www.thelastpickle.com

On 20/07/2013, at 5:04 AM, Paul Ingalls <paulingalls@gmail.com> = wrote:

I'm seeing a number of NullPointerExceptions in the log of = my cluster.  You can see the log line below.  I'm thinking = this is probably bad.  Any ideas?

ERROR = [CompactionExecutor:38] 2013-07-19 17:01:34,494 CassandraDaemon.java = (line 192) Exception in thread = Thread[CompactionExecutor:38,1,main]
java.lang.NullPointerException
=
Not sure if this is related or not, but I'm also getting a bunch of = AssertionErrors as well, even after running a scrub=85

ERROR = [CompactionExecutor:38] 2013-07-19 17:01:06,192 CassandraDaemon.java = (line 192) Exception in thread = Thread[CompactionExecutor:38,1,main]
java.lang.AssertionError: = incorrect row data size 29502477 written to = /mnt/datadrive/lib/cassandra/data/fanzo/tweets_by_team/fanzo-tweets_by_tea= m-tmp-ic-5262-Data.db; correct is 29725806
at = org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:16= 2)
= at = org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.j= ava:162)
= at = org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnab= le.java:48)
= at = org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) = at = org.apache.cassandra.db.compaction.CompactionTask.executeInternal(Compacti= onTask.java:58)
at = org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(Abstract= CompactionTask.java:60)
at = org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionT= ask.run(CompactionManager.java:211)
at = java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) = at = java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at = java.util.concurrent.FutureTask.run(FutureTask.java:166)
at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1145)
= at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :615)
= at = java.lang.Thread.run(Thread.java:724)





= --Apple-Mail=_AE23185E-F786-4189-AE30-A56A15F2E1C2--