Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Date: Tue, 13 Sep 2016 13:40:21 +0000 (UTC)
From: "Sergio Bossa (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: <JIRA.12827721.1430931031000.559042.1473774021067@Atlassian.JIRA>
In-Reply-To: <JIRA.12827721.1430931031000@Atlassian.JIRA>
References: <JIRA.12827721.1430931031000@Atlassian.JIRA> <JIRA.12827721.1430931031849@arcas>
Subject: [jira] [Commented] (CASSANDRA-9318) Bound the number of in-flight
 requests at the coordinator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Tue, 13 Sep 2016 13:40:23 -0000


    [ https://issues.apache.org/jira/browse/CASSANDRA-9318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487229#comment-15487229 ] 

Sergio Bossa commented on CASSANDRA-9318:
-----------------------------------------

Testing should be now completed and all the main point have been verified:
1) No performance regressions with back-pressure disabled.
2) No performance regressions with back-pressure enabled and normally behaving cluster.
3) Reduced dropped mutations, typically by an order of magnitude, for continuous heavy write workloads, and full recovery via hints: tested with a 4 nodes RF=3 cluster, ByteMan-based rate limiting applied in several scenarios to one or more nodes, and "slow" back-pressure.

[~eduard.tudenhoefner], [~rhatch], do you have anything to add?

I've also fixed a bug along the way and rebased to the latest trunk: [~Stefania], do you mind to have a final (hopefully!) review?

Updated patch and test runs:
| [trunk patch|https://github.com/apache/cassandra/compare/trunk...sbtourist:CASSANDRA-9318-trunk?expand=1] | [testall|https://cassci.datastax.com/view/Dev/view/sbtourist/job/sbtourist-CASSANDRA-9318-trunk-testall/] | [dtest|https://cassci.datastax.com/view/Dev/view/sbtourist/job/sbtourist-CASSANDRA-9318-trunk-dtest/] | [dtest (back-pressure enabled)|https://cassci.datastax.com/view/Dev/view/sbtourist/job/sbtourist-CASSANDRA-9318-bp-true-trunk-dtest/] |

> Bound the number of in-flight requests at the coordinator
> ---------------------------------------------------------
>
>                 Key: CASSANDRA-9318
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-9318
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Local Write-Read Paths, Streaming and Messaging
>            Reporter: Ariel Weisberg
>            Assignee: Sergio Bossa
>         Attachments: 9318-3.0-nits-trailing-spaces.patch, backpressure.png, limit.btm, no_backpressure.png
>
>
> It's possible to somewhat bound the amount of load accepted into the cluster by bounding the number of in-flight requests and request bytes.
> An implementation might do something like track the number of outstanding bytes and requests and if it reaches a high watermark disable read on client connections until it goes back below some low watermark.
> Need to make sure that disabling read on the client connection won't introduce other issues.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)