cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dan Kinder (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-13943) Infinite compaction of L0 SSTables in JBOD
Date Thu, 12 Oct 2017 00:50:00 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-13943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16201286#comment-16201286
] 

Dan Kinder commented on CASSANDRA-13943:
----------------------------------------

I've rolled out https://github.com/krummas/cassandra/commits/marcuse/13215 now per the last
comment on https://issues.apache.org/jira/browse/CASSANDRA-13215 and am currently running
relocatesstables. So far it is not doing the infinite compaction thing.

However it's a bit painful, it seems like nodetool compactionstats and nodetool cfstats no
longer work, they hang indefinitely for me.

> Infinite compaction of L0 SSTables in JBOD
> ------------------------------------------
>
>                 Key: CASSANDRA-13943
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13943
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Compaction
>         Environment: Cassandra 3.11.0 / Centos 6
>            Reporter: Dan Kinder
>            Assignee: Marcus Eriksson
>         Attachments: debug.log, debug.log-with-commit-d8f3f2780
>
>
> I recently upgraded from 2.2.6 to 3.11.0.
> I am seeing Cassandra loop infinitely compacting the same data over and over. Attaching
logs.
> It is compacting two tables, one on /srv/disk10, the other on /srv/disk1. It does create
new SSTables but immediately recompacts again. Note that I am not inserting anything at the
moment, there is no flushing happening on this table (Memtable switch count has not changed).
> My theory is that it somehow thinks those should be compaction candidates. But they shouldn't
be, they are on different disks and I ran nodetool relocatesstables as well as nodetool compact.
So, it tries to compact them together, but the compaction results in the exact same 2 SSTables
on the 2 disks, because the keys are split by data disk.
> This is pretty serious, because all our nodes right now are consuming CPU doing this
for multiple tables, it seems.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Mime
View raw message