Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Date: Fri, 2 May 2014 15:26:14 +0000 (UTC)
From: "Benedict (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: <JIRA.12711937.1399033413809.220126.1399044374753@arcas>
In-Reply-To: <JIRA.12711937.1399033413809@arcas>
References: <JIRA.12711937.1399033413809@arcas>
Subject: [jira] [Commented] (CASSANDRA-7139) Default concurrent_compactors
 is probably too high
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/CASSANDRA-7139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13987792#comment-13987792 ] 

Benedict commented on CASSANDRA-7139:
-------------------------------------

How about: 1 per disk, with a cap of 8, say? Boxes with 12+ (even 24+) disks aren't totally uncommon and you could see the same problem there as well. 

This should all be less of a problem with CASSANDRA-6696 as we'll be able to actually schedule on a per-disk basis and have no risk of referring to files on other disks, so we just want a sensible number to avoid breaking anyone who hasn't tuned their nodes between now and then.

> Default concurrent_compactors is probably too high
> --------------------------------------------------
>
>                 Key: CASSANDRA-7139
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-7139
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>            Reporter: Benedict
>            Priority: Minor
>             Fix For: 2.0.8, 2.1 rc1
>
>
> The default number of concurrent compactors is probably too high for modern hardware with spinning disks for storage: A modern blade can easily have 24+ Cores, which would result in a default of 24 concurrent compactions. This not only increases random IO, it also keeps around a lot of obsoleted files for an unnecessarily long time, as each compaction keeps references to any possibly overlapping files that it isn't itself compacting - but these can have been obsoleted part way through by compactions that finished earlier. If you factor in the default compaction throughput rate of 16Mb/s, anything but a single default concurrent_compactor makes very little sense, as a single thread should always be able to handle 16Mb/s, will cause less interference with other processes, and permits obsoleted files to be immediately removed.
> See [http://imgur.com/HDqhxFp] for a graph demonstrating the result of making this change on a box with 24-cores and 8Tb of storage (first spike is default settings)


--
This message was sent by Atlassian JIRA
(v6.2#6252)