Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Date: Mon, 2 Oct 2017 21:02:00 +0000 (UTC)
From: "Dan Kinder (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: <JIRA.13106542.1506978068000.254583.1506978120079@Atlassian.JIRA>
In-Reply-To: <JIRA.13106542.1506978068000@Atlassian.JIRA>
References: <JIRA.13106542.1506978068000@Atlassian.JIRA> <JIRA.13106542.1506978068841@jira-lw-us.apache.org>
Subject: [jira] [Created] (CASSANDRA-13923) Flushers blocked due to many
 SSTables
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Mon, 02 Oct 2017 21:02:04 -0000

Dan Kinder created CASSANDRA-13923:
--------------------------------------

             Summary: Flushers blocked due to many SSTables
                 Key: CASSANDRA-13923
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13923
             Project: Cassandra
          Issue Type: Bug
          Components: Compaction, Local Write-Read Paths
         Environment: Cassandra 3.11.0
Centos 6 (downgraded JNA)
64GB RAM
12-disk JBOD
            Reporter: Dan Kinder
         Attachments: cassandra-jstack-readstage.txt, cassandra-jstack.txt

This started on the mailing list and I'm not 100% sure of the root cause, feel free to re-title if needed.

I just upgraded Cassandra from 2.2.6 to 3.11.0. Within a few hours of serving traffic, thread pools begin to back up and grow pending tasks indefinitely. This happens to multiple different stages (Read, Mutation) and consistently builds pending tasks for MemtablePostFlush and MemtableFlushWriter.

Using jstack shows that there is blocking going on when trying to call getCompactionCandidates, which seems to happen on flush. We have fairly large nodes that have ~15,000 SSTables per node, all LCS.

I seems like this can cause reads to get blocked because they try to acquire a read lock when calling shouldDefragment.

And writes, of course, block once we can't allocate anymore memtables, because flushes are backed up.

We did not have this problem in 2.2.6, so it seems like there is some regression causing it to be incredibly slow trying to do calls like getCompactionCandidates that list out the SSTables.

In our case this causes nodes to build up pending tasks and simply stop responding to requests.


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org