cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Fabien Rousseau (Created) (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CASSANDRA-3344) Compaction throttling can be too slow
Date Tue, 11 Oct 2011 10:31:11 GMT
Compaction throttling can be too slow
-------------------------------------

                 Key: CASSANDRA-3344
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3344
             Project: Cassandra
          Issue Type: Bug
    Affects Versions: 0.8.6
            Reporter: Fabien Rousseau
            Priority: Minor


Compaction throttling needs to know how many active compactions are running (to divide bandwith
for each active compaction).

The way active compaction is counted can be broken because it counts the number of active
threads in the executor BUT the thread starts by acquiring a lock.
If the lock can't be acquired immediately : the thread is seen as "active" but does not participate
in IO operations.
The case can happen when major compaction are triggered (major compaction acquire a write
lock, while minor compactions acquire a read lock).

Having compaction througput to 16Mb/s, we observed is the following  (two times) :
 - only 1 active compaction (a long one for a few hours) starting at 16Mb/s, then after some
time running at 2Mb/s, thus taking a very long time to complete
 - many pending compactions

Using JMX and monitoring the stack trace of the compaction threads showed that :
 - 1 thread was effectively compacting
 - 1 thread was waiting to acquire the write lock (due to a major compaction)
 - 6 threads were waiting to acquire the read lock (probably due to the thread above trying
to acquire the write lock)

Attached is a proposed patch (very simple, not yet tested) which counts only active compactions.



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message