cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dan Kinder (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CASSANDRA-13923) Flushers blocked due to many SSTables
Date Mon, 02 Oct 2017 21:02:00 GMT
Dan Kinder created CASSANDRA-13923:
--------------------------------------

             Summary: Flushers blocked due to many SSTables
                 Key: CASSANDRA-13923
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13923
             Project: Cassandra
          Issue Type: Bug
          Components: Compaction, Local Write-Read Paths
         Environment: Cassandra 3.11.0
Centos 6 (downgraded JNA)
64GB RAM
12-disk JBOD
            Reporter: Dan Kinder
         Attachments: cassandra-jstack-readstage.txt, cassandra-jstack.txt

This started on the mailing list and I'm not 100% sure of the root cause, feel free to re-title
if needed.

I just upgraded Cassandra from 2.2.6 to 3.11.0. Within a few hours of serving traffic, thread
pools begin to back up and grow pending tasks indefinitely. This happens to multiple different
stages (Read, Mutation) and consistently builds pending tasks for MemtablePostFlush and MemtableFlushWriter.

Using jstack shows that there is blocking going on when trying to call getCompactionCandidates,
which seems to happen on flush. We have fairly large nodes that have ~15,000 SSTables per
node, all LCS.

I seems like this can cause reads to get blocked because they try to acquire a read lock when
calling shouldDefragment.

And writes, of course, block once we can't allocate anymore memtables, because flushes are
backed up.

We did not have this problem in 2.2.6, so it seems like there is some regression causing it
to be incredibly slow trying to do calls like getCompactionCandidates that list out the SSTables.

In our case this causes nodes to build up pending tasks and simply stop responding to requests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Mime
View raw message