cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcus Eriksson (JIRA)" <>
Subject [jira] [Assigned] (CASSANDRA-13923) Flushers blocked due to many SSTables
Date Tue, 03 Oct 2017 07:29:00 GMT


Marcus Eriksson reassigned CASSANDRA-13923:

    Assignee: Marcus Eriksson

> Flushers blocked due to many SSTables
> -------------------------------------
>                 Key: CASSANDRA-13923
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Compaction, Local Write-Read Paths
>         Environment: Cassandra 3.11.0
> Centos 6 (downgraded JNA)
> 64GB RAM
> 12-disk JBOD
>            Reporter: Dan Kinder
>            Assignee: Marcus Eriksson
>         Attachments: cassandra-jstack-readstage.txt, cassandra-jstack.txt
> This started on the mailing list and I'm not 100% sure of the root cause, feel free to
re-title if needed.
> I just upgraded Cassandra from 2.2.6 to 3.11.0. Within a few hours of serving traffic,
thread pools begin to back up and grow pending tasks indefinitely. This happens to multiple
different stages (Read, Mutation) and consistently builds pending tasks for MemtablePostFlush
and MemtableFlushWriter.
> Using jstack shows that there is blocking going on when trying to call getCompactionCandidates,
which seems to happen on flush. We have fairly large nodes that have ~15,000 SSTables per
node, all LCS.
> I seems like this can cause reads to get blocked because they try to acquire a read lock
when calling shouldDefragment.
> And writes, of course, block once we can't allocate anymore memtables, because flushes
are backed up.
> We did not have this problem in 2.2.6, so it seems like there is some regression causing
it to be incredibly slow trying to do calls like getCompactionCandidates that list out the
> In our case this causes nodes to build up pending tasks and simply stop responding to

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message