incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pavel Kirienko <pavel.kirienko.l...@gmail.com>
Subject Re: AssertionError: Incorrect row data size
Date Tue, 30 Jul 2013 11:39:48 GMT
Also it is probably worth to mention:
1. I see no other errors in logs except that one;
2. Sometimes connected clients receive "Request did not complete within
rpc_timeout.", even if they are accessing other tables.
3. Sometimes, some cells from another tables may read as NULL when they are
in fact not empty. (This is really really bad, maybe I failed to configure
something properly)

I can provide more info if needed.

Thanks in advance.


On Tue, Jul 30, 2013 at 3:32 PM, Pavel Kirienko <
pavel.kirienko.list@gmail.com> wrote:

> Cassandra 1.2.8 still have this issue.
>
> Possible recipe to reproduce: create the table as described in the first
> message of this thread; write 3000 rows of 10MB each at the rate about
> 0.1..1 request per second.
> Maybe this behavior is caused by incremental compaction of large rows...
>
>
> On Mon, Jul 29, 2013 at 8:59 AM, Paul Ingalls <paulingalls@gmail.com>wrote:
>
>> Great. Let me know what you find!
>>
>> Thanks!
>>
>> Paul
>>
>> Sent from my iPhone
>>
>> On Jul 27, 2013, at 2:47 AM, Pavel Kirienko <
>> pavel.kirienko.list@gmail.com> wrote:
>>
>> Hi Paul,
>>
>> I checked out your issue, looks the same indeed. Probably this can be
>> reproed simply by writing large rows (> 10MB) on high rates.
>> I'm going to try 1.2.7 then will be back with results.
>>
>>
>> On Sat, Jul 27, 2013 at 12:18 AM, Paul Ingalls <paulingalls@gmail.com>wrote:
>>
>>> This is the same issue we have been seeing.  Still no luck getting a
>>> simple repro case for creating a JIRA issue.  Do you have something simple
>>> enough to drop in a JIRA report?
>>>
>>> Paul
>>>
>>> On Jul 26, 2013, at 8:06 AM, Pavel Kirienko <
>>> pavel.kirienko.list@gmail.com> wrote:
>>>
>>> > Hi list,
>>> >
>>> > We run Cassandra 1.2 on three-node cluster. Each node has 16GB RAM,
>>> single 200GB HDD with Ubuntu Server 12.04.
>>> >
>>> > There is an issue with one table that contains about 3000 rows, here
>>> its describe-table:
>>> >
>>> > CREATE TABLE outputs (
>>> >   appid text,
>>> >   staged boolean,
>>> >   field ascii,
>>> >   data blob,
>>> >   PRIMARY KEY (appid, staged, field)
>>> > ) WITH
>>> >   bloom_filter_fp_chance=0.010000 AND
>>> >   caching='KEYS_ONLY' AND
>>> >   comment='' AND
>>> >   dclocal_read_repair_chance=0.000000 AND
>>> >   gc_grace_seconds=864000 AND
>>> >   read_repair_chance=0.100000 AND
>>> >   replicate_on_write='true' AND
>>> >   populate_io_cache_on_flush='false' AND
>>> >   compaction={'class': 'SizeTieredCompactionStrategy'} AND
>>> >   compression={'sstable_compression': 'SnappyCompressor'};
>>> >
>>> > Column DATA contains blobs of size about 1..50MB, average size should
>>> be something of 5MB.
>>> >
>>> > Sometimes this table expiriences huge write loads for few hours, at
>>> such times I see suspicious things in logs:
>>> >
>>> > ERROR [CompactionExecutor:357] 2013-07-24 12:32:10,293
>>> CassandraDaemon.java (line 192) Exception in thread
>>> Thread[CompactionExecutor:357,1,main]
>>> > java.lang.AssertionError: incorrect row data size 172489604 written to
>>> /var/lib/cassandra/data/woodpecker/outputs/woodpecker-outputs-tmp-ic-813-Data.db;
>>> correct is 172489704
>>> >         at
>>> org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:162)
>>> >         at
>>> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:162)
>>> >         at
>>> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
>>> >         at
>>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>>> >         at
>>> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58)
>>> >         at
>>> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
>>> >         at
>>> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:211)
>>> >         at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
>>> Source)
>>> >         at java.util.concurrent.FutureTask$Sync.innerRun(Unknown
>>> Source)
>>> >         at java.util.concurrent.FutureTask.run(Unknown Source)
>>> >         at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
>>> Source)
>>> >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
>>> Source)
>>> >         at java.lang.Thread.run(Unknown Source)
>>> >
>>> > What shall I do about this?
>>> >
>>> > Thanks in advance.
>>> > Pavel.
>>> >
>>>
>>>
>>
>

Mime
View raw message