cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pavel Pletnev (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-7775) Cassandra attempts to flush an empty memtable into disk and fails
Date Tue, 09 Sep 2014 13:26:29 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-7775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14126967#comment-14126967
] 

Pavel Pletnev edited comment on CASSANDRA-7775 at 9/9/14 1:26 PM:
------------------------------------------------------------------

After trying to flush corrupted memtable MemtablePostFlusher thread stuck and start to pill
up.
This is my trace:

 INFO [FlushWriter:4] 2014-09-09 14:55:03,405 Memtable.java (line 355) Writing Memtable-ucc_ttl16318@1568409896(230/2300
serialized/live bytes, 2 ops)
 INFO [OptionalTasks:1] 2014-09-09 14:55:03,410 ColumnFamilyStore.java (line 794) Enqueuing
flush of Memtable-sstable_activity@1141428663(38070/1495207 serialized/live bytes, 2800 ops)
ERROR [FlushWriter:4] 2014-09-09 14:55:03,610 CassandraDaemon.java (line 199) Exception in
thread Thread[FlushWriter:4,5,main]
java.lang.RuntimeException: Cannot get comparator 1 in org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type).
This might due to a mismatch between the schema and the data read
        at org.apache.cassandra.db.marshal.CompositeType.getComparator(CompositeType.java:133)
        at org.apache.cassandra.db.marshal.AbstractCompositeType.split(AbstractCompositeType.java:100)
        at org.apache.cassandra.io.sstable.ColumnNameHelper.minComponents(ColumnNameHelper.java:103)
        at org.apache.cassandra.db.ColumnFamily.getColumnStats(ColumnFamily.java:443)
        at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:194)
        at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:389)
        at org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:342)
        at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IndexOutOfBoundsException: index (1) must be less than size (1)
        at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:306)
        at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:285)
        at com.google.common.collect.SingletonImmutableList.get(SingletonImmutableList.java:45)
        at org.apache.cassandra.db.marshal.CompositeType.getComparator(CompositeType.java:124)
        ... 11 more

Version: 2.0.10


was (Author: merentafo):
After trying to flush corrupted memtable MemtablePostFlusher thread stuck and start to pill
up.
This is my trace:

 INFO [FlushWriter:4] 2014-09-09 14:55:03,405 Memtable.java (line 355) Writing Memtable-ucc_ttl16318@1568409896(230/2300
serialized/live bytes, 2 ops)
 INFO [OptionalTasks:1] 2014-09-09 14:55:03,410 ColumnFamilyStore.java (line 794) Enqueuing
flush of Memtable-sstable_activity@1141428663(38070/1495207 serialized/live bytes, 2800 ops)
ERROR [FlushWriter:4] 2014-09-09 14:55:03,610 CassandraDaemon.java (line 199) Exception in
thread Thread[FlushWriter:4,5,main]
java.lang.RuntimeException: Cannot get comparator 1 in org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type).
This might due to a mismatch between the schema and the data read
        at org.apache.cassandra.db.marshal.CompositeType.getComparator(CompositeType.java:133)
        at org.apache.cassandra.db.marshal.AbstractCompositeType.split(AbstractCompositeType.java:100)
        at org.apache.cassandra.io.sstable.ColumnNameHelper.minComponents(ColumnNameHelper.java:103)
        at org.apache.cassandra.db.ColumnFamily.getColumnStats(ColumnFamily.java:443)
        at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:194)
        at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:389)
        at org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:342)
        at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IndexOutOfBoundsException: index (1) must be less than size (1)
        at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:306)
        at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:285)
        at com.google.common.collect.SingletonImmutableList.get(SingletonImmutableList.java:45)
        at org.apache.cassandra.db.marshal.CompositeType.getComparator(CompositeType.java:124)
        ... 11 more

> Cassandra attempts to flush an empty memtable into disk and fails
> -----------------------------------------------------------------
>
>                 Key: CASSANDRA-7775
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-7775
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: $ nodetool version
> ReleaseVersion: 2.0.6
> $ java -version
> java version "1.7.0_51"
> Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
> Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)
>            Reporter: Omri Bahumi
>
> I'm not sure what triggers this flush, but when it happens the following appears in our
logs:
> {code}
>  INFO [OptionalTasks:1] 2014-08-15 02:24:20,115 ColumnFamilyStore.java (line 785) Enqueuing
flush of Memtable-app_recs_best_in_expr_prefix2@1219170646(0/0 serialized/live bytes, 0 ops)
>  INFO [FlushWriter:34] 2014-08-15 02:24:20,116 Memtable.java (line 331) Writing Memtable-app_recs_best_in_expr_prefix2@1219170646(0/0
serialized/live bytes, 0 ops)
> ERROR [FlushWriter:34] 2014-08-15 02:24:20,127 CassandraDaemon.java (line 196) Exception
in thread Thread[FlushWriter:34,5,main]
> java.lang.RuntimeException: Cannot get comparator 1 in org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type).
This might due to a mismatch between the schema and the data read
>         at org.apache.cassandra.db.marshal.CompositeType.getComparator(CompositeType.java:133)
>         at org.apache.cassandra.db.marshal.CompositeType.getComparator(CompositeType.java:140)
>         at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:96)
>         at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:35)
>         at org.apache.cassandra.db.RangeTombstone$Tracker$1.compare(RangeTombstone.java:125)
>         at org.apache.cassandra.db.RangeTombstone$Tracker$1.compare(RangeTombstone.java:122)
>         at java.util.TreeMap.compare(TreeMap.java:1188)
>         at java.util.TreeMap$NavigableSubMap.<init>(TreeMap.java:1264)
>         at java.util.TreeMap$AscendingSubMap.<init>(TreeMap.java:1699)
>         at java.util.TreeMap.tailMap(TreeMap.java:905)
>         at java.util.TreeSet.tailSet(TreeSet.java:350)
>         at java.util.TreeSet.tailSet(TreeSet.java:383)
>         at org.apache.cassandra.db.RangeTombstone$Tracker.update(RangeTombstone.java:203)
>         at org.apache.cassandra.db.ColumnIndex$Builder.add(ColumnIndex.java:192)
>         at org.apache.cassandra.db.ColumnIndex$Builder.build(ColumnIndex.java:138)
>         at org.apache.cassandra.io.sstable.SSTableWriter.rawAppend(SSTableWriter.java:202)
>         at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:187)
>         at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:365)
>         at org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:318)
>         at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:744)
> Caused by: java.lang.IndexOutOfBoundsException: index (1) must be less than size (1)
>         at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:306)
>         at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:285)
>         at com.google.common.collect.SingletonImmutableList.get(SingletonImmutableList.java:45)
>         at org.apache.cassandra.db.marshal.CompositeType.getComparator(CompositeType.java:124)
>         ... 23 more
> {code}
> After this happens, the MemtablePostFlusher thread pool starts piling up.
> When trying to restart the cluster, a similar exception occurs when trying to replay
the commit log.
> Our way of recovering from this is to delete all commit logs in the faulty node, start
it and issue a repair.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message