cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peter Haggerty (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-7543) Assertion error when compacting large row with map//list field or range tombstone
Date Tue, 09 Sep 2014 22:12:29 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-7543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127649#comment-14127649
] 

Peter Haggerty commented on CASSANDRA-7543:
-------------------------------------------

This is listed in the 2.0 CHANGES.txt as present in 2.0.10 but "Fix Versions" shows only 1.2.19.


> Assertion error when compacting large row with map//list field or range tombstone
> ---------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-7543
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-7543
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: linux
>            Reporter: Matt Byrd
>            Assignee: Yuki Morishita
>              Labels: compaction, map
>             Fix For: 1.2.19
>
>         Attachments: 0001-add-rangetombstone-test.patch, 0002-fix-rangetomebstone-not-included-in-LCR-size-calc.patch
>
>
> Hi,
> So in a couple of clusters we're hitting this problem when compacting large rows with
a schema which contains the map data-type.
> Here is an example of the error:
> {code}
> java.lang.AssertionError: incorrect row data size 87776427 written to /cassandra/X/Y/X-Y-tmp-ic-2381-Data.db;
correct is 87845952
> org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:162)
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:163)
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58)

> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)

> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:208)
> {code}
> I have a python script which reproduces the problem, by just writing lots of data to
a single partition key with a schema that contains the map data-type.
> I added some debug logging and found that the difference in bytes seen in the reproduction
(255) was due to the following pieces of data being written:
> {code}
> DEBUG [CompactionExecutor:3] 2014-07-13 00:38:42,891 ColumnIndex.java (line 168) DATASIZE
writeOpenedMarker columnIndex: org.apache.cassandra.db.ColumnIndex$Builder@6678a9d0 firstColumn:
[java.nio.HeapByteBuffer[pos=0 lim=34 cap=34], java.nio.HeapByteBuffer[pos=0 lim=34 cap=34]](deletedAt=1405237116014999,
localDeletion=1405237116) startPosition: 262476 endPosition: 262561 diff: 85 
> DEBUG [CompactionExecutor:3] 2014-07-13 00:38:43,007 ColumnIndex.java (line 168) DATASIZE
writeOpenedMarker columnIndex: org.apache.cassandra.db.ColumnIndex$Builder@6678a9d0 firstColumn:
org.apache.cassandra.db.Column@3e5b5939 startPosition: 328157 endPosition: 328242 diff: 85

> DEBUG [CompactionExecutor:3] 2014-07-13 00:38:44,159 ColumnIndex.java (line 168) DATASIZE
writeOpenedMarker columnIndex: org.apache.cassandra.db.ColumnIndex$Builder@6678a9d0 firstColumn:
org.apache.cassandra.db.Column@fc3299b startPosition: 984105 endPosition: 984190 diff: 85
> {code}
> So looking at the code you can see that there are extra range tombstones written on the
column index border (in ColumnIndex where tombstoneTracker.writeOpenedMarker is called) which
aren't accounted for in LazilyCompactedRow.columnSerializedSize.
> This is where the difference comes from in the assertion error, so the solution is just
to account for this data.
> I have a patch which does just this, by keeping track of the extra data written out via
tombstoneTracker.writeOpenedMarker in ColumnIndex and adding it back to the dataSize in LazilyCompactedRow.java,
where it serialises out the row size.
> After applying the patch the reproduction stops producing the AssertionError.
> I know this is not a problem in 2.0 + because of singe pass compaction, however there
are lots of 1.2 clusters out there still which might run into this.
> Please let me know if you've any questions.
> Thanks,
> Matt



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message