cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yuki Morishita (JIRA)" <j...@apache.org>
Subject [jira] [Reopened] (CASSANDRA-3306) Error in LeveledCompactionStrategy
Date Tue, 23 Oct 2012 22:44:12 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-3306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Yuki Morishita reopened CASSANDRA-3306:
---------------------------------------

      Assignee: Yuki Morishita

This error actually happens on 1.1. And I can easily reproduce with unit test(Test code attached).

{code}
    [junit] ERROR 17:34:46,696 Fatal exception in thread Thread[CompactionExecutor:3,1,main]
    [junit] java.lang.AssertionError: Expecting new size of 2, got 1 while replacing [SSTableReader(path='build/test/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-hf-1-Data.db'),
SSTableReader(path='build/test/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-hf-5-Data.db'),
SSTableReader(path='build/test/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-hf-4-Data.db'),
SSTableReader(path='build/test/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-hf-2-Data.db')]
by [SSTableReader(path='build/test/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-hf-6-Data.db')]
in View(pending_count=0, sstables=[SSTableReader(path='build/test/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-hf-1-Data.db'),
SSTableReader(path='build/test/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-hf-2-Data.db'),
SSTableReader(path='build/test/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-hf-4-Data.db'),
SSTableReader(path='build/test/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-hf-4-Data.db'),
SSTableReader(path='build/test/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-hf-5-Data.db')],
compacting=[SSTableReader(path='build/test/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-hf-1-Data.db'),
SSTableReader(path='build/test/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-hf-5-Data.db'),
SSTableReader(path='build/test/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-hf-4-Data.db'),
SSTableReader(path='build/test/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-hf-2-Data.db')])
    [junit] 	at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:651)
    [junit] 	at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:616)
    [junit] 	at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:320)
    [junit] 	at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:253)
    [junit] 	at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:994)
    [junit] 	at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:200)
    [junit] 	at org.apache.cassandra.db.compaction.CompactionManager$1.runMayThrow(CompactionManager.java:154)
    [junit] 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
    [junit] 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    [junit] 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    [junit] 	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    [junit] 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    [junit] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    [junit] 	at java.lang.Thread.run(Thread.java:680)
{code}

The cause is actually in streaming. StreamInSession can add duplicate reference to SSTable
to DataTracker when it is left even after stream session finishes. This typically happens
when source node is marked as dead by FailureDetector during streaming session(GC storm is
the one I saw) and keep sending file in same session after the node comes back.
                
> Error in LeveledCompactionStrategy
> ----------------------------------
>
>                 Key: CASSANDRA-3306
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3306
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 1.0.0
>            Reporter: Radim Kolar
>            Assignee: Yuki Morishita
>
> during stress testing, i always get this error making leveledcompaction strategy unusable.
Should be easy to reproduce - just write fast.
> ERROR [CompactionExecutor:6] 2011-10-04 15:48:52,179 AbstractCassandraDaemon.java (line
133) Fatal exception in thread Thread[CompactionExecutor:6,5,main]
> java.lang.AssertionError
> 	at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:580)
> 	at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546)
> 	at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268)
> 	at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232)
> 	at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960)
> 	at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199)
> 	at org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:47)
> 	at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131)
> 	at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 	at java.lang.Thread.run(Thread.java:662)
> and this is in json data for table:
> {
>   "generations" : [ {
>     "generation" : 0,
>     "members" : [ 459, 460, 461, 462, 463, 464, 465, 466, 467, 468, 469, 470, 471, 472,
473, 474, 475, 476, 477, 478, 479, 480, 481, 482, 483, 484 ]
>   }, {
>     "generation" : 1,
>     "members" : [ ]
>   }, {
>     "generation" : 2,
>     "members" : [ ]
>   }, {
>     "generation" : 3,
>     "members" : [ ]
>   }, {
>     "generation" : 4,
>     "members" : [ ]
>   }, {
>     "generation" : 5,
>     "members" : [ ]
>   }, {
>     "generation" : 6,
>     "members" : [ ]
>   }, {
>     "generation" : 7,
>     "members" : [ ]
>   } ]
> }

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message