cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Ellis (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-9318) Bound the number of in-flight requests at the coordinator
Date Mon, 11 Jan 2016 14:27:40 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-9318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15091963#comment-15091963
] 

Jonathan Ellis edited comment on CASSANDRA-9318 at 1/11/16 2:26 PM:
--------------------------------------------------------------------

Since most of that discussion is implementation details, I'll quote the relevant part:

bq. With consistency level less then ALL mutation processing can move to background (meaning
client was answered, but there is still work to do on behalf of the request). If background
request rate completion is lower than incoming request rate background request will accumulate
and eventually will exhaust all memory resources. This patch's aim is to prevent this situation
by monitoring how much memory all current background request take and when some threshold
is passed stop moving request to background (by not replying to a client until either memory
consumptions moves below the threshold or request is fully completed).

bq. There are two main point where each background mutation consumes memory: holding frozen
mutation until operation is complete in order to hint 
if it does not) and on rpc queue to each replica where it sits until it's sent out on the
wire. The patch accounts for both of those separately and limits the former to be 10% of total
memory and the later to be 6M. Why 6M? The best answer I can give is why not :) But on a more
serious note the number should be small enough so that all the data can be sent out in a reasonable
amount of time and one shard is not capable to achieve even close to a full bandwidth, so
empirical evidence shows 6M to be a good number. 


was (Author: jbellis):
Since most of that discussion is implementation details, I'll quote the relevant part:

bq. With consistency level less then ALL mutation processing can move to
background (meaning client was answered, but there is still work to
do on behalf of the request). If background request rate completion
is lower than incoming request rate background request will accumulate
and eventually will exhaust all memory resources. This patch's aim is
to prevent this situation by monitoring how much memory all current
background request take and when some threshold is passed stop moving
request to background (by not replying to a client until either memory
consumptions moves below the threshold or request is fully completed).

bq. There are two main point where each background mutation consumes memory:
holding frozen mutation until operation is complete in order to hint it
if it does not) and on rpc queue to each replica where it sits until it's
sent out on the wire. The patch accounts for both of those separately
and limits the former to be 10% of total memory and the later to be 6M.
Why 6M? The best answer I can give is why not :) But on a more serious
note the number should be small enough so that all the data can be
sent out in a reasonable amount of time and one shard is not capable to
achieve even close to a full bandwidth, so empirical evidence shows 6M
to be a good number. 

> Bound the number of in-flight requests at the coordinator
> ---------------------------------------------------------
>
>                 Key: CASSANDRA-9318
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-9318
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Local Write-Read Paths, Streaming and Messaging
>            Reporter: Ariel Weisberg
>            Assignee: Ariel Weisberg
>             Fix For: 2.1.x, 2.2.x
>
>
> It's possible to somewhat bound the amount of load accepted into the cluster by bounding
the number of in-flight requests and request bytes.
> An implementation might do something like track the number of outstanding bytes and requests
and if it reaches a high watermark disable read on client connections until it goes back below
some low watermark.
> Need to make sure that disabling read on the client connection won't introduce other
issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message