cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Benedict (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-12764) Compaction performance issues with many sstables, during transaction commit phase
Date Mon, 10 Oct 2016 14:02:21 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-12764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15562395#comment-15562395
] 

Benedict commented on CASSANDRA-12764:
--------------------------------------

bq. Sadly, that would've taken 20 days

Well, that sounds like a more pressing concern.  These tickets shouldn't have materially affected
the ETA of a major compaction.  It's possible it entered a GC death spiral, as probably >6Gb
of heap would have been needed for the reader buffers alone (though it's been a while since
I've checked our behaviours here; it's possible they're lazily initialised and with tiny sstables
they might not all have been needed to be materialised at once).  

Anyway, it sounds potentially worth filing a ticket to investigate that, as it should be the
get-out-of-jail free card for the problems you had.

Perhaps it's worth filing a ticket to explicitly make major compaction robust to literally
arbitrary numbers of files also.  i.e. by performing multiple rounds of "major compaction"
if the file count is too high to do at once.

> Compaction performance issues with many sstables, during transaction commit phase
> ---------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-12764
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-12764
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Compaction
>            Reporter: Tom van der Woerdt
>
> An issue with a script flooded my cluster with sstables. There is now a table with 100k
sstables, all on the order of KBytes, and it's taking a long time (ETA 20 days) to compact,
even though the table is only ~30GB.
> Stack trace :
> {noformat}
> "CompactionExecutor:308" #7541 daemon prio=1 os_prio=4 tid=0x00007fa22af35400 nid=0x41eb
runnable [0x00007fdbea48d000]
>    java.lang.Thread.State: RUNNABLE
> 	at java.util.TimSort.countRunAndMakeAscending(TimSort.java:360)
> 	at java.util.TimSort.sort(TimSort.java:220)
> 	at java.util.Arrays.sort(Arrays.java:1438)
> 	at com.google.common.collect.Ordering.sortedCopy(Ordering.java:817)
> 	at org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:209)
> 	at org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:211)
> 	at org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:211)
> 	at org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:211)
> 	at org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:211)
> 	at org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:211)
> 	at org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:211)
> 	at org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:211)
> 	at org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:210)
> 	at org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:210)
> 	at org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:210)
> 	at org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:210)
> 	at org.apache.cassandra.utils.IntervalTree.<init>(IntervalTree.java:50)
> 	at org.apache.cassandra.db.lifecycle.SSTableIntervalTree.<init>(SSTableIntervalTree.java:40)
> 	at org.apache.cassandra.db.lifecycle.SSTableIntervalTree.build(SSTableIntervalTree.java:50)
> 	at org.apache.cassandra.db.lifecycle.View$4.apply(View.java:288)
> 	at org.apache.cassandra.db.lifecycle.View$4.apply(View.java:283)
> 	at com.google.common.base.Functions$FunctionComposition.apply(Functions.java:216)
> 	at org.apache.cassandra.db.lifecycle.Tracker.apply(Tracker.java:128)
> 	at org.apache.cassandra.db.lifecycle.Tracker.apply(Tracker.java:101)
> 	at org.apache.cassandra.db.lifecycle.LifecycleTransaction.checkpoint(LifecycleTransaction.java:307)
> 	at org.apache.cassandra.db.lifecycle.LifecycleTransaction.checkpoint(LifecycleTransaction.java:288)
> 	at org.apache.cassandra.io.sstable.SSTableRewriter.doPrepare(SSTableRewriter.java:368)
> 	at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> 	at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.doPrepare(CompactionAwareWriter.java:84)
> 	at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> 	at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184)
> 	at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.finish(CompactionAwareWriter.java:94)
> 	at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:194)
> 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> 	at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78)
> 	at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> 	at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:263)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> {noformat}
> IntervalTree shows in a lot of stack traces I've taken on several nodes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message