cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeremy Hanna (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-12763) Compaction performance issues when a table has a lot of sstables
Date Mon, 10 Oct 2016 16:20:20 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-12763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15562717#comment-15562717
] 

Jeremy Hanna commented on CASSANDRA-12763:
------------------------------------------

As an aside, you may want to determine why you have a lot of small sstables in the first place.
 One common cause is when people change the number of flush writers but don't change the memtable
cleanup threshold.  So it flushes much more frequently, resulting in lots of small sstables.
 Granted we want to improve the handling of a lot of small sstables, but flushing larger sstables
will help avoid the problem.

> Compaction performance issues when a table has a lot of sstables
> ----------------------------------------------------------------
>
>                 Key: CASSANDRA-12763
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-12763
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Compaction
>            Reporter: Tom van der Woerdt
>
> An issue with a script flooded my cluster with sstables. There is now a table with 100k
sstables, all on the order of KBytes, and it's taking a long time (ETA 20 days) to compact,
even though the table is only ~30GB.
> Stack trace :
> {noformat}
> "CompactionExecutor:269" #7536 daemon prio=1 os_prio=4 tid=0x00007f4acd40fc00 nid=0x14f8
runnable [0x00007f4798436000]
>    java.lang.Thread.State: RUNNABLE
> 	at java.io.UnixFileSystem.list(Native Method)
> 	at java.io.File.list(File.java:1122)
> 	at java.io.File.listFiles(File.java:1248)
> 	at org.apache.cassandra.db.lifecycle.LogRecord.getExistingFiles(LogRecord.java:268)
> 	at org.apache.cassandra.db.lifecycle.LogRecord.make(LogRecord.java:150)
> 	at org.apache.cassandra.db.lifecycle.LogFile.makeRecord(LogFile.java:293)
> 	at org.apache.cassandra.db.lifecycle.LogFile.add(LogFile.java:283)
> 	at org.apache.cassandra.db.lifecycle.LogTransaction.obsoleted(LogTransaction.java:158)
> 	at org.apache.cassandra.db.lifecycle.Helpers.prepareForObsoletion(Helpers.java:134)
> 	at org.apache.cassandra.db.lifecycle.LifecycleTransaction.doPrepare(LifecycleTransaction.java:193)
> 	at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> 	at org.apache.cassandra.io.sstable.SSTableRewriter.doPrepare(SSTableRewriter.java:376)
> 	at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> 	at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.doPrepare(CompactionAwareWriter.java:84)
> 	at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> 	at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184)
> 	at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.finish(CompactionAwareWriter.java:94)
> 	at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:194)
> 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> 	at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78)
> 	at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> 	at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:263)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> {noformat}
> listFiles is being called over and over, apparently scaling with the number of files
in the compaction.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message