Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 466CF10E73 for ; Tue, 30 Jul 2013 11:40:18 +0000 (UTC) Received: (qmail 55256 invoked by uid 500); 30 Jul 2013 11:40:15 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 55222 invoked by uid 500); 30 Jul 2013 11:40:15 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 55212 invoked by uid 99); 30 Jul 2013 11:40:15 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Jul 2013 11:40:15 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of pavel.kirienko.list@gmail.com designates 209.85.214.43 as permitted sender) Received: from [209.85.214.43] (HELO mail-bk0-f43.google.com) (209.85.214.43) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Jul 2013 11:40:10 +0000 Received: by mail-bk0-f43.google.com with SMTP id jm2so1949873bkc.16 for ; Tue, 30 Jul 2013 04:39:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=4ELwTA/heexEyR0ugjhXBB5KDGryFzneU40bgdJqWYQ=; b=cIySPvxM4A2CY6Ol7uLcZPOxqvX5VHeyjYrZBd8UDHCpRLPz/DsSs3czC2Kge2XRJh NrZ8hbZALiLx8LUgF4DN1F+7oYFU7GwLsHcO5HPlXr8K4/Y3wFieAn7a1kmeUcwbXq6j MFpevfj7Y18mKvcvg3ePhoJN5sy29/NIjGrVEdEtKKYFRDtARtLOxrj2vyzKSTgTQd8L N1gAH8yVmQmdusN+WXhpYk10CzI+lni692sIGXOKsVEmD90Q9I6SQlKklrxq0MNtp+w9 lZnJPAUYjAEM6Chfj8nTKpAhuCCGc3AMMypsPDlsDDedf7QkpuNF3ugi3qRuGeAz8ZNM blzA== MIME-Version: 1.0 X-Received: by 10.205.22.71 with SMTP id qv7mr919407bkb.20.1375184388892; Tue, 30 Jul 2013 04:39:48 -0700 (PDT) Received: by 10.204.185.70 with HTTP; Tue, 30 Jul 2013 04:39:48 -0700 (PDT) In-Reply-To: References: <80D90631-D25C-41BA-8765-38E0CD84A265@gmail.com> <32CB1D92-F3B7-41BF-B1B3-D4F07C488DFF@gmail.com> Date: Tue, 30 Jul 2013 15:39:48 +0400 Message-ID: Subject: Re: AssertionError: Incorrect row data size From: Pavel Kirienko To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=20cf30223bf5005c8a04e2b911d4 X-Virus-Checked: Checked by ClamAV on apache.org --20cf30223bf5005c8a04e2b911d4 Content-Type: text/plain; charset=ISO-8859-1 Also it is probably worth to mention: 1. I see no other errors in logs except that one; 2. Sometimes connected clients receive "Request did not complete within rpc_timeout.", even if they are accessing other tables. 3. Sometimes, some cells from another tables may read as NULL when they are in fact not empty. (This is really really bad, maybe I failed to configure something properly) I can provide more info if needed. Thanks in advance. On Tue, Jul 30, 2013 at 3:32 PM, Pavel Kirienko < pavel.kirienko.list@gmail.com> wrote: > Cassandra 1.2.8 still have this issue. > > Possible recipe to reproduce: create the table as described in the first > message of this thread; write 3000 rows of 10MB each at the rate about > 0.1..1 request per second. > Maybe this behavior is caused by incremental compaction of large rows... > > > On Mon, Jul 29, 2013 at 8:59 AM, Paul Ingalls wrote: > >> Great. Let me know what you find! >> >> Thanks! >> >> Paul >> >> Sent from my iPhone >> >> On Jul 27, 2013, at 2:47 AM, Pavel Kirienko < >> pavel.kirienko.list@gmail.com> wrote: >> >> Hi Paul, >> >> I checked out your issue, looks the same indeed. Probably this can be >> reproed simply by writing large rows (> 10MB) on high rates. >> I'm going to try 1.2.7 then will be back with results. >> >> >> On Sat, Jul 27, 2013 at 12:18 AM, Paul Ingalls wrote: >> >>> This is the same issue we have been seeing. Still no luck getting a >>> simple repro case for creating a JIRA issue. Do you have something simple >>> enough to drop in a JIRA report? >>> >>> Paul >>> >>> On Jul 26, 2013, at 8:06 AM, Pavel Kirienko < >>> pavel.kirienko.list@gmail.com> wrote: >>> >>> > Hi list, >>> > >>> > We run Cassandra 1.2 on three-node cluster. Each node has 16GB RAM, >>> single 200GB HDD with Ubuntu Server 12.04. >>> > >>> > There is an issue with one table that contains about 3000 rows, here >>> its describe-table: >>> > >>> > CREATE TABLE outputs ( >>> > appid text, >>> > staged boolean, >>> > field ascii, >>> > data blob, >>> > PRIMARY KEY (appid, staged, field) >>> > ) WITH >>> > bloom_filter_fp_chance=0.010000 AND >>> > caching='KEYS_ONLY' AND >>> > comment='' AND >>> > dclocal_read_repair_chance=0.000000 AND >>> > gc_grace_seconds=864000 AND >>> > read_repair_chance=0.100000 AND >>> > replicate_on_write='true' AND >>> > populate_io_cache_on_flush='false' AND >>> > compaction={'class': 'SizeTieredCompactionStrategy'} AND >>> > compression={'sstable_compression': 'SnappyCompressor'}; >>> > >>> > Column DATA contains blobs of size about 1..50MB, average size should >>> be something of 5MB. >>> > >>> > Sometimes this table expiriences huge write loads for few hours, at >>> such times I see suspicious things in logs: >>> > >>> > ERROR [CompactionExecutor:357] 2013-07-24 12:32:10,293 >>> CassandraDaemon.java (line 192) Exception in thread >>> Thread[CompactionExecutor:357,1,main] >>> > java.lang.AssertionError: incorrect row data size 172489604 written to >>> /var/lib/cassandra/data/woodpecker/outputs/woodpecker-outputs-tmp-ic-813-Data.db; >>> correct is 172489704 >>> > at >>> org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:162) >>> > at >>> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:162) >>> > at >>> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) >>> > at >>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) >>> > at >>> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58) >>> > at >>> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) >>> > at >>> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:211) >>> > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown >>> Source) >>> > at java.util.concurrent.FutureTask$Sync.innerRun(Unknown >>> Source) >>> > at java.util.concurrent.FutureTask.run(Unknown Source) >>> > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown >>> Source) >>> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown >>> Source) >>> > at java.lang.Thread.run(Unknown Source) >>> > >>> > What shall I do about this? >>> > >>> > Thanks in advance. >>> > Pavel. >>> > >>> >>> >> > --20cf30223bf5005c8a04e2b911d4 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Also it is probably worth to mention:
1. I see no other= errors in logs except that one;
2. Sometimes connected clients receive = "Request did not complete within rpc_timeout.", even if they are = accessing other tables.
3. Sometimes, some cells from another tables may read as NULL when they are= in fact not empty. (This is really really bad, maybe I failed to configure= something properly)

I can provide more info if need= ed.

Thanks in advance.


On Tue, Jul 30, 2013 at 3:32 PM,= Pavel Kirienko <pavel.kirienko.list@gmail.com> = wrote:
Cassandra 1.2.8 still have = this issue.

Possible recipe to reproduce: create the table as = described in the first message of this thread; write 3000 rows of 10MB each= at the rate about 0.1..1 request per second.
Maybe this behavior is caused by incremental compaction of large= rows...


On Mon, Jul 29, 2013 at 8:59= AM, Paul Ingalls <paulingalls@gmail.com> wrote:
Great. Let me know wh= at you find!

Thanks!

Paul=

Sent from my iPhone

On Jul 27, 2013, at 2:47 AM= , Pavel Kirienko <pavel.kirienko.list@gmail.com> wrote:

Hi Paul,

I checked out your issue, looks the same indeed. Probably this can b= e reproed simply by writing large rows (> 10MB) on high rates.
I'm going to try 1.2.7 then will be back with results.


On Sat,= Jul 27, 2013 at 12:18 AM, Paul Ingalls <paulingalls@gmail.com>= wrote:
This is the same issue we have been seeing. = =A0Still no luck getting a simple repro case for creating a JIRA issue. =A0= Do you have something simple enough to drop in a JIRA report?

Paul

On Jul 26, 2013, at 8:06 AM, Pavel Kirienko <pavel.kirienko.list@gmail.com&g= t; wrote:

> Hi list,
>
> We run Cassandra 1.2 on three-node cluster. Each node has 16GB RAM, si= ngle 200GB HDD with Ubuntu Server 12.04.
>
> There is an issue with one table that contains about 3000 rows, here i= ts describe-table:
>
> CREATE TABLE outputs (
> =A0 appid text,
> =A0 staged boolean,
> =A0 field ascii,
> =A0 data blob,
> =A0 PRIMARY KEY (appid, staged, field)
> ) WITH
> =A0 bloom_filter_fp_chance=3D0.010000 AND
> =A0 caching=3D'KEYS_ONLY' AND
> =A0 comment=3D'' AND
> =A0 dclocal_read_repair_chance=3D0.000000 AND
> =A0 gc_grace_seconds=3D864000 AND
> =A0 read_repair_chance=3D0.100000 AND
> =A0 replicate_on_write=3D'true' AND
> =A0 populate_io_cache_on_flush=3D'false' AND
> =A0 compaction=3D{'class': 'SizeTieredCompactionStrategy&#= 39;} AND
> =A0 compression=3D{'sstable_compression': 'SnappyCompresso= r'};
>
> Column DATA contains blobs of size about 1..50MB, average size should = be something of 5MB.
>
> Sometimes this table expiriences huge write loads for few hours, at su= ch times I see suspicious things in logs:
>
> ERROR [CompactionExecutor:357] 2013-07-24 12:32:10,293 CassandraDaemon= .java (line 192) Exception in thread Thread[CompactionExecutor:357,1,main]<= br> > java.lang.AssertionError: incorrect row data size 172489604 written to= /var/lib/cassandra/data/woodpecker/outputs/woodpecker-outputs-tmp-ic-813-D= ata.db; correct is 172489704
> =A0 =A0 =A0 =A0 at org.apache.cassandra.io.sstable.SSTableWriter.appen= d(SSTableWriter.java:162)
> =A0 =A0 =A0 =A0 at org.apache.cassandra.db.compaction.CompactionTask.r= unWith(CompactionTask.java:162)
> =A0 =A0 =A0 =A0 at org.apache.cassandra.io.util.DiskAwareRunnable.runM= ayThrow(DiskAwareRunnable.java:48)
> =A0 =A0 =A0 =A0 at org.apache.cassandra.utils.WrappedRunnable.run(Wrap= pedRunnable.java:28)
> =A0 =A0 =A0 =A0 at org.apache.cassandra.db.compaction.CompactionTask.e= xecuteInternal(CompactionTask.java:58)
> =A0 =A0 =A0 =A0 at org.apache.cassandra.db.compaction.AbstractCompacti= onTask.execute(AbstractCompactionTask.java:60)
> =A0 =A0 =A0 =A0 at org.apache.cassandra.db.compaction.CompactionManage= r$BackgroundCompactionTask.run(CompactionManager.java:211)
> =A0 =A0 =A0 =A0 at java.util.concurrent.Executors$RunnableAdapter.call= (Unknown Source)
> =A0 =A0 =A0 =A0 at java.util.concurrent.FutureTask$Sync.innerRun(Unkno= wn Source)
> =A0 =A0 =A0 =A0 at java.util.concurrent.FutureTask.run(Unknown Source)=
> =A0 =A0 =A0 =A0 at java.util.concurrent.ThreadPoolExecutor.runWorker(U= nknown Source)
> =A0 =A0 =A0 =A0 at java.util.concurrent.ThreadPoolExecutor$Worker.run(= Unknown Source)
> =A0 =A0 =A0 =A0 at java.lang.Thread.run(Unknown Source)
>
> What shall I do about this?
>
> Thanks in advance.
> Pavel.
>




--20cf30223bf5005c8a04e2b911d4--