Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Date: Thu, 23 Jul 2015 14:32:06 +0000 (UTC)
From: "Jonathan Ellis (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: <JIRA.12741464.1410785101000.271657.1437661926217@Atlassian.JIRA>
In-Reply-To: <JIRA.12741464.1410785101000@Atlassian.JIRA>
References: <JIRA.12741464.1410785101000@Atlassian.JIRA>
 <JIRA.12741464.1410785101718@arcas>
Subject: [jira] [Commented] (CASSANDRA-7937) Apply backpressure gently when
 overloaded with writes
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/CASSANDRA-7937?page=3Dcom.atlas=
sian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D=
14638908#comment-14638908 ]=20

Jonathan Ellis commented on CASSANDRA-7937:
-------------------------------------------

I think we can do as well with a simpler approach by using MessagingService=
 queues as a proxy for target's load.  (If the target is overwhelmed it wil=
l read slower from the socket and our queue will not drain; if it is not mo=
re-than-usually-overwhelmed but clients are sending us so many requests for=
 that target that we still can't drain it fast enough, then we should also =
pause accepting extra requests.)

See CASSANDRA-9318 and in particular my summary [here|https://issues.apache=
.org/jira/browse/CASSANDRA-9318?focusedCommentId=3D14604649&page=3Dcom.atla=
ssian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14604649].

(NB feel free to reassign to Jacek if he has free cycles.)

> Apply backpressure gently when overloaded with writes
> -----------------------------------------------------
>
>                 Key: CASSANDRA-7937
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-7937
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>         Environment: Cassandra 2.0
>            Reporter: Piotr Ko=C5=82aczkowski
>            Assignee: Jacek Lewandowski
>              Labels: performance
>
> When writing huge amounts of data into C* cluster from analytic tools lik=
e Hadoop or Apache Spark, we can see that often C* can't keep up with the l=
oad. This is because analytic tools typically write data "as fast as they c=
an" in parallel, from many nodes and they are not artificially rate-limited=
, so C* is the bottleneck here. Also, increasing the number of nodes doesn'=
t really help, because in a collocated setup this also increases number of =
Hadoop/Spark nodes (writers) and although possible write performance is hig=
her, the problem still remains.
> We observe the following behavior:
> 1. data is ingested at an extreme fast pace into memtables and flush queu=
e fills up
> 2. the available memory limit for memtables is reached and writes are no =
longer accepted
> 3. the application gets hit by "write timeout", and retries repeatedly, i=
n vain=20
> 4. after several failed attempts to write, the job gets aborted=20
> Desired behaviour:
> 1. data is ingested at an extreme fast pace into memtables and flush queu=
e fills up
> 2. after exceeding some memtable "fill threshold", C* applies adaptive ra=
te limiting to writes - the more the buffers are filled-up, the less writes=
/s are accepted, however writes still occur within the write timeout.
> 3. thanks to slowed down data ingestion, now flush can finish before all =
the memory gets used
> Of course the details how rate limiting could be done are up for a discus=
sion.
> It may be also worth considering putting such logic into the driver, not =
C* core, but then C* needs to expose at least the following information to =
the driver, so we could calculate the desired maximum data rate:
> 1. current amount of memory available for writes before they would comple=
tely block
> 2. total amount of data queued to be flushed and flush progress (amount o=
f data to flush remaining for the memtable currently being flushed)
> 3. average flush write speed


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)