cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Harvey (JIRA)" <>
Subject [jira] [Updated] (CASSANDRA-4492) Large HintsColumnFamily compactions hang
Date Sun, 05 Aug 2012 04:59:02 GMT


Jason Harvey updated CASSANDRA-4492:

    Attachment:     (was: jstack)
> Large HintsColumnFamily compactions hang
> ----------------------------------------
>                 Key: CASSANDRA-4492
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 1.0.11
>            Reporter: Jason Harvey
>            Priority: Minor
> Running into an issue on a 6 node ring running 1.0.11 where whenever a somewhat large
set of hints build up (seen as low as 400MB), compaction on the hints CF hangs indefinitely.
Nothing of note in the logs. In some cases, the compaction hangs before a tmp sstable is even
> I've wiped out every hints sstable I have and restarted several times. The issue always
comes back rather quickly and predictably after wiping the sstables. Compaction always seems
to succeed if the hints CFs are rather small.
> Hints are enabled, and my hint window is the default of 1hr. I do have some copies of
HintsColumnFamily sstables that do replicate this issue. However, the hints may contain confidential
data. If they'd be helpful in troubleshooting this issue, let me know and I can see about
sending them directly.
> This ring was upgraded from 1.0.7. I didn't keep any hints from the upgrade.
> Here is the output I see from compactionstats where a compaction has hung. The 'bytes
compacted' column never changes.
> {code}
> pending tasks: 1
>           compaction type        keyspace   column family bytes compacted     bytes total
>                Compaction          systemHintsColumnFamily          268082       464784758
> {code}
> The hung thread stack is as follows: (full jstack attached, as well)
> {code}
> "CompactionExecutor:37" daemon prio=10 tid=0x00000000063df800 nid=0x49d9 waiting on condition
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x000000050f2e0e58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at java.util.concurrent.locks.LockSupport.park(
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
>         at java.util.concurrent.LinkedBlockingQueue.take(
>         at org.apache.cassandra.db.compaction.ParallelCompactionIterable$Deserializer.computeNext(
>         at org.apache.cassandra.db.compaction.ParallelCompactionIterable$Deserializer.computeNext(
>         at
>         at
>         at org.apache.cassandra.utils.MergeIterator$Candidate.advance(
>         at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(
>         at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(
>         at
>         at
>         at org.apache.cassandra.db.compaction.ParallelCompactionIterable$Unwrapper.computeNext(
>         at org.apache.cassandra.db.compaction.ParallelCompactionIterable$Unwrapper.computeNext(
>         at
>         at
>         at$7.computeNext(
>         at
>         at
>         at org.apache.cassandra.db.compaction.CompactionTask.execute(
>         at org.apache.cassandra.db.compaction.CompactionManager$
>         at java.util.concurrent.FutureTask$Sync.innerRun(
>         at
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(
>         at java.util.concurrent.ThreadPoolExecutor$
>         at
> {code}

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


View raw message