cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcus Eriksson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-13943) Infinite compaction of L0 SSTables in JBOD
Date Thu, 12 Oct 2017 18:49:00 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-13943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16202454#comment-16202454
] 

Marcus Eriksson commented on CASSANDRA-13943:
---------------------------------------------

I think you could lower the number of {{concurrent_compactors}} (maybe 10 or so) and you probably
need to set {{compaction_throughput_mb_per_sec}} to something conservative, I would start
with 10 and then increase if things look good.

> Infinite compaction of L0 SSTables in JBOD
> ------------------------------------------
>
>                 Key: CASSANDRA-13943
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13943
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Compaction
>         Environment: Cassandra 3.11.0 / Centos 6
>            Reporter: Dan Kinder
>            Assignee: Marcus Eriksson
>         Attachments: cassandra-jstack-2017-10-12-infinite-sstable-adding.txt, cassandra-jstack-2017-10-12.txt,
cassandra.yaml, debug.log, debug.log-with-commit-d8f3f2780, debug.log.1.zip, debug.log.zip,
jvm.options
>
>
> I recently upgraded from 2.2.6 to 3.11.0.
> I am seeing Cassandra loop infinitely compacting the same data over and over. Attaching
logs.
> It is compacting two tables, one on /srv/disk10, the other on /srv/disk1. It does create
new SSTables but immediately recompacts again. Note that I am not inserting anything at the
moment, there is no flushing happening on this table (Memtable switch count has not changed).
> My theory is that it somehow thinks those should be compaction candidates. But they shouldn't
be, they are on different disks and I ran nodetool relocatesstables as well as nodetool compact.
So, it tries to compact them together, but the compaction results in the exact same 2 SSTables
on the 2 disks, because the keys are split by data disk.
> This is pretty serious, because all our nodes right now are consuming CPU doing this
for multiple tables, it seems.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Mime
View raw message