cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tom van der Woerdt (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (CASSANDRA-13469) Compaction stops; new sstables aren't recognized
Date Sun, 23 Apr 2017 13:48:04 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-13469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Tom van der Woerdt resolved CASSANDRA-13469.
--------------------------------------------
    Resolution: Invalid

... never mind. This is most likely a configuration issue on my side.

> Compaction stops; new sstables aren't recognized
> ------------------------------------------------
>
>                 Key: CASSANDRA-13469
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13469
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Tom van der Woerdt
>
> cfstats shows me this, so I know I'm in trouble :
> {code}
>                 SSTable count: 3419
>                 SSTables in each level: [2, 20/10, 154/100, 1034/1000, 0, 0, 0, 0, 0]
> {code}
> Disk space on these nodes (about five of them) doubled overnight, due to the high write
volume not getting compacted. `nodetool compactionstats' shows me zero pending compactions.
> After a restart things are "better" :
> {noformat}
>                 SSTable count: 3432
>                 SSTables in each level: [1280/4, 21/10, 262/100, 1851/1000, 15, 0, 0,
0, 0]
> {noformat}
> compactionstats also shows some compactions again.
> jstack for one compaction thread (there are 24 of them, all looking the exact same) :
> {noformat}
> "CompactionExecutor:73017" #203194 daemon prio=1 os_prio=4 tid=0x00007edd1e597c00 nid=0xf7e4
waiting on condition [0x00007efd16782000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>     at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for  <0x00000002c1859520> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>     at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>     at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>     at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>     at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>     at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
>     at org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$3/335359181.run(Unknown
Source)
>     at java.lang.Thread.run(Thread.java:745)
> {noformat}
> Sadly I'm pretty clueless about what happened here. :(
> Oh, and tpstats :
> {noformat}
> Pool Name                    Active   Pending      Completed   Blocked  All time blocked
> MutationStage                     0         0     1098466484         0              
  0
> ViewMutationStage                 0         0              0         0              
  0
> ReadStage                         2         0      759723051         0              
  0
> RequestResponseStage              0         0     1773795694         0              
  0
> ReadRepairStage                   0         0       37372223         0              
  0
> CounterMutationStage              0         0              0         0              
  0
> MiscStage                         0         0              0         0              
  0
> CompactionExecutor                0         0         336794         0              
  0
> MemtableReclaimMemory             0         0          75324         0              
  0
> PendingRangeCalculator            0         0             68         0              
  0
> GossipStage                       0         0        1363346         0              
  0
> SecondaryIndexManagement          0         0              0         0              
  0
> HintsDispatcher                   1         1             38         0              
  0
> MigrationStage                    0         0         274693         0              
  0
> MemtablePostFlush                 0         0         248358         0              
  0
> ValidationExecutor                0         0           7882         0              
  0
> Sampler                           0         0              0         0              
  0
> MemtableFlushWriter               0         0          75324         0              
  0
> InternalResponseStage             0         0         259871         0              
  0
> AntiEntropyStage                  0         0          20469         0              
  0
> CacheCleanupExecutor              0         0              0         0              
  0
> Native-Transport-Requests         0         0     1489435995         0            188985
> Message type           Dropped
> READ                         2
> RANGE_SLICE                  0
> _TRACE                       0
> HINT                         0
> MUTATION                    20
> COUNTER_MUTATION             0
> BATCH_STORE                  0
> BATCH_REMOVE                 0
> REQUEST_RESPONSE             0
> PAGED_RANGE                  0
> READ_REPAIR                  4
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message