cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dan Kinder (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-13923) Flushers blocked due to many SSTables
Date Wed, 04 Oct 2017 17:50:00 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-13923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16191699#comment-16191699
] 

Dan Kinder commented on CASSANDRA-13923:
----------------------------------------

So far the nodes seem to be performing better, they are doing a lot of compactions. So, that
patch seemed to do the trick.

Sadly I was not able to get a jstack with -l.

> Flushers blocked due to many SSTables
> -------------------------------------
>
>                 Key: CASSANDRA-13923
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13923
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Compaction, Local Write-Read Paths
>         Environment: Cassandra 3.11.0
> Centos 6 (downgraded JNA)
> 64GB RAM
> 12-disk JBOD
>            Reporter: Dan Kinder
>            Assignee: Marcus Eriksson
>         Attachments: cassandra-jstack-readstage.txt, cassandra-jstack.txt
>
>
> This started on the mailing list and I'm not 100% sure of the root cause, feel free to
re-title if needed.
> I just upgraded Cassandra from 2.2.6 to 3.11.0. Within a few hours of serving traffic,
thread pools begin to back up and grow pending tasks indefinitely. This happens to multiple
different stages (Read, Mutation) and consistently builds pending tasks for MemtablePostFlush
and MemtableFlushWriter.
> Using jstack shows that there is blocking going on when trying to call getCompactionCandidates,
which seems to happen on flush. We have fairly large nodes that have ~15,000 SSTables per
node, all LCS.
> I seems like this can cause reads to get blocked because they try to acquire a read lock
when calling shouldDefragment.
> And writes, of course, block once we can't allocate anymore memtables, because flushes
are backed up.
> We did not have this problem in 2.2.6, so it seems like there is some regression causing
it to be incredibly slow trying to do calls like getCompactionCandidates that list out the
SSTables.
> In our case this causes nodes to build up pending tasks and simply stop responding to
requests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Mime
View raw message