cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tom van der Woerdt (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CASSANDRA-13469) Compaction stops; new sstables aren't recognized
Date Sun, 23 Apr 2017 13:27:04 GMT
Tom van der Woerdt created CASSANDRA-13469:
----------------------------------------------

             Summary: Compaction stops; new sstables aren't recognized
                 Key: CASSANDRA-13469
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13469
             Project: Cassandra
          Issue Type: Bug
            Reporter: Tom van der Woerdt


cfstats shows me this, so I know I'm in trouble :

{code}
                SSTable count: 3419
                SSTables in each level: [2, 20/10, 154/100, 1034/1000, 0, 0, 0, 0, 0]
{code}

Disk space on these nodes (about five of them) doubled overnight, due to the high write volume
not getting compacted. `nodetool compactionstats' shows me zero pending compactions.

After a restart things are "better" :

{noformat}
                SSTable count: 3432
                SSTables in each level: [1280/4, 21/10, 262/100, 1851/1000, 15, 0, 0, 0, 0]
{noformat}

compactionstats also shows some compactions again.

jstack for one compaction thread (there are 24 of them, all looking the exact same) :

{noformat}
"CompactionExecutor:73017" #203194 daemon prio=1 os_prio=4 tid=0x00007edd1e597c00 nid=0xf7e4
waiting on condition [0x00007efd16782000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000002c1859520> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
    at org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$3/335359181.run(Unknown
Source)
    at java.lang.Thread.run(Thread.java:745)
{noformat}

Sadly I'm pretty clueless about what happened here. :(

Oh, and tpstats :

{noformat}
Pool Name                    Active   Pending      Completed   Blocked  All time blocked
MutationStage                     0         0     1098466484         0                 0
ViewMutationStage                 0         0              0         0                 0
ReadStage                         2         0      759723051         0                 0
RequestResponseStage              0         0     1773795694         0                 0
ReadRepairStage                   0         0       37372223         0                 0
CounterMutationStage              0         0              0         0                 0
MiscStage                         0         0              0         0                 0
CompactionExecutor                0         0         336794         0                 0
MemtableReclaimMemory             0         0          75324         0                 0
PendingRangeCalculator            0         0             68         0                 0
GossipStage                       0         0        1363346         0                 0
SecondaryIndexManagement          0         0              0         0                 0
HintsDispatcher                   1         1             38         0                 0
MigrationStage                    0         0         274693         0                 0
MemtablePostFlush                 0         0         248358         0                 0
ValidationExecutor                0         0           7882         0                 0
Sampler                           0         0              0         0                 0
MemtableFlushWriter               0         0          75324         0                 0
InternalResponseStage             0         0         259871         0                 0
AntiEntropyStage                  0         0          20469         0                 0
CacheCleanupExecutor              0         0              0         0                 0
Native-Transport-Requests         0         0     1489435995         0            188985

Message type           Dropped
READ                         2
RANGE_SLICE                  0
_TRACE                       0
HINT                         0
MUTATION                    20
COUNTER_MUTATION             0
BATCH_STORE                  0
BATCH_REMOVE                 0
REQUEST_RESPONSE             0
PAGED_RANGE                  0
READ_REPAIR                  4
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message