cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brandon Williams (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-12764) Compaction performance issues with many sstables, during transaction commit phase
Date Mon, 10 Oct 2016 19:51:20 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-12764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15563295#comment-15563295
] 

Brandon Williams commented on CASSANDRA-12764:
----------------------------------------------

bq. I ended up going for a solution along the lines of "move all data files out, then in batches
of 500 sstables move them back in, run 'nodetool refresh', wait for compaction, then move
the next batch".

FWIW, that was my suggestion, and I recall was about 25 times faster.

> Compaction performance issues with many sstables, during transaction commit phase
> ---------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-12764
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-12764
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Compaction
>            Reporter: Tom van der Woerdt
>              Labels: lcs
>
> An issue with a script flooded my cluster with sstables. There is now a table with 100k
sstables, all on the order of KBytes, and it's taking a long time (ETA 20 days) to compact,
even though the table is only ~30GB.
> Stack trace :
> {noformat}
> "CompactionExecutor:308" #7541 daemon prio=1 os_prio=4 tid=0x00007fa22af35400 nid=0x41eb
runnable [0x00007fdbea48d000]
>    java.lang.Thread.State: RUNNABLE
> 	at java.util.TimSort.countRunAndMakeAscending(TimSort.java:360)
> 	at java.util.TimSort.sort(TimSort.java:220)
> 	at java.util.Arrays.sort(Arrays.java:1438)
> 	at com.google.common.collect.Ordering.sortedCopy(Ordering.java:817)
> 	at org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:209)
> 	at org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:211)
> 	at org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:211)
> 	at org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:211)
> 	at org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:211)
> 	at org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:211)
> 	at org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:211)
> 	at org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:211)
> 	at org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:210)
> 	at org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:210)
> 	at org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:210)
> 	at org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:210)
> 	at org.apache.cassandra.utils.IntervalTree.<init>(IntervalTree.java:50)
> 	at org.apache.cassandra.db.lifecycle.SSTableIntervalTree.<init>(SSTableIntervalTree.java:40)
> 	at org.apache.cassandra.db.lifecycle.SSTableIntervalTree.build(SSTableIntervalTree.java:50)
> 	at org.apache.cassandra.db.lifecycle.View$4.apply(View.java:288)
> 	at org.apache.cassandra.db.lifecycle.View$4.apply(View.java:283)
> 	at com.google.common.base.Functions$FunctionComposition.apply(Functions.java:216)
> 	at org.apache.cassandra.db.lifecycle.Tracker.apply(Tracker.java:128)
> 	at org.apache.cassandra.db.lifecycle.Tracker.apply(Tracker.java:101)
> 	at org.apache.cassandra.db.lifecycle.LifecycleTransaction.checkpoint(LifecycleTransaction.java:307)
> 	at org.apache.cassandra.db.lifecycle.LifecycleTransaction.checkpoint(LifecycleTransaction.java:288)
> 	at org.apache.cassandra.io.sstable.SSTableRewriter.doPrepare(SSTableRewriter.java:368)
> 	at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> 	at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.doPrepare(CompactionAwareWriter.java:84)
> 	at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> 	at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184)
> 	at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.finish(CompactionAwareWriter.java:94)
> 	at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:194)
> 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> 	at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78)
> 	at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> 	at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:263)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> {noformat}
> IntervalTree shows in a lot of stack traces I've taken on several nodes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message