cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Philip Thompson (JIRA)" <>
Subject [jira] [Updated] (CASSANDRA-9914) Millions of fake pending compaction tasks + high CPU
Date Tue, 28 Jul 2015 20:01:05 GMT


Philip Thompson updated CASSANDRA-9914:
         Assignee: Marcus Eriksson
    Fix Version/s: 2.1.x

> Millions of fake pending compaction tasks + high CPU
> ----------------------------------------------------
>                 Key: CASSANDRA-9914
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: CentOS
>            Reporter: Robbie Strickland
>            Assignee: Marcus Eriksson
>             Fix For: 2.1.x
>         Attachments: cass_high_cpu.png, high_pending_compactions.txt
> We have a 3-node test cluster (initially running 2.1.8) with *zero traffic* and about
10GB of data on each node.  It's showing millions of pending compaction tasks (but no actual
work in progress), and the CPUs are pegged on all three nodes.  The task count goes down rapidly,
but then jumps back up again seconds later.  All tables are set to STCS.  The issue persists
after restart, but takes a few minutes before it becomes a problem.  SSTable counts are below
10 for every table.  We're also seeing 20s Old Gen GC pauses about every 2-3 mins.
> This started happening after bulk loading some old data.  We started seeing very long
GC pauses (sometimes 30 min or more) that would bring down the nodes.  We then truncated this
table, which resulted in the current behavior.  We attempted to roll back our cluster to 2.1.7
patched with CASSANDRA-9637, but we observed the same behavior.

This message was sent by Atlassian JIRA

View raw message