cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rudolf van der Leeden <rudolf.vanderlee...@scoreloop.com>
Subject Re: 2 nodes throwing exceptions trying to compact after upgrade to 1.1.2 from 1.1.0
Date Tue, 17 Jul 2012 04:22:19 GMT
Stay with 1.1.2 and create your CF with  compaction_strategy_class='SizeTieredCompactionStrategy'



On 16.07.2012, at 22:17, Bryce Godfrey wrote:

> Thanks, is there a way around this for now or should I fall back to 1.1.0?
>  
>  
> From: Rudolf van der Leeden [mailto:rudolf.vanderleeden@scoreloop.com] 
> Sent: Monday, July 16, 2012 12:55 PM
> To: user@cassandra.apache.org
> Cc: Rudolf van der Leeden
> Subject: Re: 2 nodes throwing exceptions trying to compact after upgrade to 1.1.2 from
1.1.0
>  
> See      https://issues.apache.org/jira/browse/CASSANDRA-4411
> The bug is related to LCS (leveled compaction) and has been fixed.
>  
>  
> On 16.07.2012, at 20:32, Bryce Godfrey wrote:
> 
> 
> This may not be directly related to the upgrade to 1.1.2, but I was running on 1.1.0
for a while with no issues, and I did the upgrade to 1.1.2 a few days ago.
>  
> 2 of my nodes started throwing lots of promote exceptions, and then a lot of the beforeAppend
exceptions from then on every few minutes.  This is on the high update CF that’s using leveled
compaction and compression.  The other 3 nodes are not experiencing this.  I can send entire
log files if desired.
> These 2 nodes now have much higher load #’s then the other 3, and I’m assuming that’s
because they are failing with the compaction errors?
>  
> $
> INFO [CompactionExecutor:1783] 2012-07-13 07:35:23,268 CompactionTask.java (line 109)
Compacting [SSTableReader(path='/opt/cassandra/data/MonitoringData/Properties/MonitoringData-Properties-hd-392322-Data$
> ERROR [CompactionExecutor:1783] 2012-07-13 07:35:29,696 AbstractCassandraDaemon.java
(line 134) Exception in thread Thread[CompactionExecutor:1783,1,main]
> java.lang.AssertionError
>         at org.apache.cassandra.db.compaction.LeveledManifest.promote(LeveledManifest.java:214)
>         at org.apache.cassandra.db.compaction.LeveledCompactionStrategy.handleNotification(LeveledCompactionStrategy.java:158)
>         at org.apache.cassandra.db.DataTracker.notifySSTablesChanged(DataTracker.java:531)
>         at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:254)
>         at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:978)
>         at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:200)
>         at org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:50)
>         at org.apache.cassandra.db.compaction.CompactionManager$1.runMayThrow(CompactionManager.java:150)
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
>         at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
>         at java.util.concurrent.FutureTask.run(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
>  
> INFO [CompactionExecutor:3310] 2012-07-16 11:14:02,481 CompactionTask.java (line 109)
Compacting [SSTableReader(path='/opt/cassandra/data/MonitoringData/Properties/MonitoringData-Properties-hd-369173-Data$
> ERROR [CompactionExecutor:3310] 2012-07-16 11:14:04,031 AbstractCassandraDaemon.java
(line 134) Exception in thread Thread[CompactionExecutor:3310,1,main]
> java.lang.RuntimeException: Last written key DecoratedKey(150919285004100953907590722809541628889,
5b30363334353237652d383966382d653031312d623131632d3030313535643031373530325d5b436f6d70757465725b4d5350422d$
>         at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:134)
>         at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:153)
>         at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:159)
>         at org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:50)
>         at org.apache.cassandra.db.compaction.CompactionManager$1.runMayThrow(CompactionManager.java:150)
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
>         at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
>         at java.util.concurrent.FutureTask.run(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
>  


Mime
View raw message