Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 853F19589 for ; Mon, 30 Apr 2012 04:05:33 +0000 (UTC) Received: (qmail 82443 invoked by uid 500); 30 Apr 2012 04:05:31 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 82421 invoked by uid 500); 30 Apr 2012 04:05:31 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 82408 invoked by uid 99); 30 Apr 2012 04:05:30 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Apr 2012 04:05:30 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a94.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Apr 2012 04:05:24 +0000 Received: from homiemail-a94.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a94.g.dreamhost.com (Postfix) with ESMTP id 09D2E38A05B for ; Sun, 29 Apr 2012 21:05:03 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; c=nofws; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; q=dns; s=thelastpickle.com; b=NwuLS1Fxmp 4M5xTuMQaTcbI+aBpso6UZVN7cZbv/x4BpLew/ogqVSdLMa96nBCYSi+qTIsSlLv +MErbc0Nc4DQPho9xXBjV+aTCJvStBoji5c0yNrXCVRsQSRz4GTR6r55zxzFCNj7 AR9sr3WzR/S0rodLUN3k1Vk+vo1uRib38= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; s=thelastpickle.com; bh=n1h6Edt+RZtDm3Hs 19135CNay10=; b=MchjqhBs+DcCkxtt9GCBJEEyVn6fX2yVdVKPjA9JpZfWDZyw ejFo7DTn5pSidHWe6FCKF3d13MVhVZgc6gR7ge5ddMuGGiVUIeO7ze892r0wrxxI zf9JR+P1P8OVbZ8ib6Bm9kdQSYVPDs6RVIULYrOXYU5+OrnZslvWuUiSVCk= Received: from [192.168.1.52] (unknown [203.109.195.29]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a94.g.dreamhost.com (Postfix) with ESMTPSA id 89D0638A00C for ; Sun, 29 Apr 2012 21:05:02 -0700 (PDT) From: aaron morton Mime-Version: 1.0 (Apple Message framework v1257) Content-Type: multipart/alternative; boundary="Apple-Mail=_7A721E93-C959-43CD-8E0B-6D5D253A25F4" Subject: Re: AssertionError: originally calculated column size ... Date: Mon, 30 Apr 2012 16:05:14 +1200 In-Reply-To: To: user@cassandra.apache.org References: Message-Id: <68AE65BC-D76B-45DD-916F-F1EAEA51BEF0@thelastpickle.com> X-Mailer: Apple Mail (2.1257) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_7A721E93-C959-43CD-8E0B-6D5D253A25F4 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Looks a bit like https://issues.apache.org/jira/browse/CASSANDRA-3579 = but that was fixed in 1.0.7 Is this still an issue ? Are you able to reproduce the fault ?=20 Cheers ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 27/04/2012, at 6:56 PM, Patrik Modesto wrote: > Hi, >=20 > I've 4 node cluster of Cassandra 1.0.9. There is a rfTest3 keyspace > with RF=3D3 and one CF with two secondary indexes. I'm importing data > into this CF using Hadoop Mapreduce job, each row has less than 10 > colkumns. =46rom JMX: > MaxRowSize: 1597 > MeanRowSize: 369 >=20 > And there are some tens of millions of rows. >=20 > It's write-heavy usage and there is a big pressure on each node, there > are quite some dropped mutations on each node. After ~12 hours of > inserting I see these assertion exceptiona on 3 out of four nodes: >=20 > ERROR 06:25:40,124 Fatal exception in thread = Thread[HintedHandoff:1,1,main] > java.lang.RuntimeException: java.util.concurrent.ExecutionException: > java.lang.AssertionError: originally calculated column size of > 629444349 but now it is 588008950 > at = org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInterna= l(HintedHandOffManager.java:388) > at = org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(Hinted= HandOffManager.java:256) > at = org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManag= er.java:84) > at = org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffMa= nager.java:437) > at = org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > at = java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.= java:886) > at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :908) > at java.lang.Thread.run(Thread.java:662) > Caused by: java.util.concurrent.ExecutionException: > java.lang.AssertionError: originally calculated column size of > 629444349 but now it is 588008950 > at = java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) > at java.util.concurrent.FutureTask.get(FutureTask.java:83) > at = org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInterna= l(HintedHandOffManager.java:384) > ... 7 more > Caused by: java.lang.AssertionError: originally calculated column size > of 629444349 but now it is 588008950 > at = org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompacte= dRow.java:124) > at = org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:16= 0) > at = org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.j= ava:161) > at = org.apache.cassandra.db.compaction.CompactionManager$7.call(CompactionMana= ger.java:380) > at = java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > ... 3 more >=20 >=20 > Few lines regarding Hints from the output.log: >=20 > INFO 06:21:26,202 Compacting large row > system/HintsColumnFamily:70000000000000000000000000000000 (1712834057 > bytes) incrementally > INFO 06:22:52,610 Compacting large row > system/HintsColumnFamily:10000000000000000000000000000000 (2616073981 > bytes) incrementally > INFO 06:22:59,111 flushing high-traffic column family > CFS(Keyspace=3D'system', ColumnFamily=3D'HintsColumnFamily') = (estimated > 305147360 bytes) > INFO 06:22:59,813 Enqueuing flush of > Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live > bytes, 7452 ops) > INFO 06:22:59,814 Writing > Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live > bytes, 7452 ops) --Apple-Mail=_7A721E93-C959-43CD-8E0B-6D5D253A25F4 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii Looks = a bit like https://issu= es.apache.org/jira/browse/CASSANDRA-3579 but that was fixed in = 1.0.7

Is this still an issue ? Are you able to = reproduce the fault = ? 

Cheers


http://www.thelastpickle.com

On 27/04/2012, at 6:56 PM, Patrik Modesto wrote:

Hi,

I've 4 node cluster of Cassandra 1.0.9. = There is a rfTest3 keyspace
with RF=3D3 and one CF with two secondary = indexes. I'm importing data
into this CF using Hadoop Mapreduce job, = each row has less than 10
colkumns. =46rom JMX:
MaxRowSize: =  1597
MeanRowSize: 369

And there are some tens of = millions of rows.

It's write-heavy usage and there is a big = pressure on each node, there
are quite some dropped mutations on each = node. After ~12 hours of
inserting I see these assertion exceptiona = on 3 out of four nodes:

ERROR 06:25:40,124 Fatal exception in = thread Thread[HintedHandoff:1,1,main]
java.lang.RuntimeException: = java.util.concurrent.ExecutionException:
java.lang.AssertionError: = originally calculated column size of
629444349 but now it is = 588008950
       at = org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInterna= l(HintedHandOffManager.java:388)
=        at = org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(Hinted= HandOffManager.java:256)
=        at = org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManag= er.java:84)
       at = org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffMa= nager.java:437)
       at = org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)        at = java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.= java:886)
       at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :908)
       at = java.lang.Thread.run(Thread.java:662)
Caused by: = java.util.concurrent.ExecutionException:
java.lang.AssertionError: = originally calculated column size of
629444349 but now it is = 588008950
       at = java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
=        at = java.util.concurrent.FutureTask.get(FutureTask.java:83)
=        at = org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInterna= l(HintedHandOffManager.java:384)
=        ... 7 more
Caused by: = java.lang.AssertionError: originally calculated column size
of = 629444349 but now it is 588008950
=        at = org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompacte= dRow.java:124)
       at = org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:16= 0)
       at = org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.j= ava:161)
       at = org.apache.cassandra.db.compaction.CompactionManager$7.call(CompactionMana= ger.java:380)
       at = java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
=        at = java.util.concurrent.FutureTask.run(FutureTask.java:138)
=        ... 3 more


Few = lines regarding Hints from the output.log:

INFO 06:21:26,202 = Compacting large = row
system/HintsColumnFamily:70000000000000000000000000000000 = (1712834057
bytes) incrementally
INFO 06:22:52,610 Compacting = large row
system/HintsColumnFamily:10000000000000000000000000000000 = (2616073981
bytes) incrementally
INFO 06:22:59,111 flushing = high-traffic column family
CFS(Keyspace=3D'system', = ColumnFamily=3D'HintsColumnFamily') (estimated
305147360 bytes)
= INFO 06:22:59,813 Enqueuing flush = of
Memtable-HintsColumnFamily@833933926(3814342/305147360 = serialized/live
bytes, 7452 ops)
INFO 06:22:59,814 = Writing
Memtable-HintsColumnFamily@833933926(3814342/305147360 = serialized/live
bytes, 7452 = ops)

= --Apple-Mail=_7A721E93-C959-43CD-8E0B-6D5D253A25F4--