cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sergio Bossa (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-9318) Bound the number of in-flight requests at the coordinator
Date Mon, 18 Jul 2016 11:33:21 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-9318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15382114#comment-15382114
] 

Sergio Bossa commented on CASSANDRA-9318:
-----------------------------------------

I've pushed a new update based on the most recent discussions, more specifically:
* I've implemented (and locally tested) rate limiting based on either the fastest or slowest
replica; rate limiting on the average rate required computing such rate for all replicas in
the whole cluster each RPC timeout interval (2 secs by default), which I deemed to be not
worth it.
* I've removed the low ratio configuration: if we aim to keep the cluster as "responsive"
as possible, throwing exceptions or prematurely hinting doesn't make sense.
* {{SP.sendToHintedEndpoints}} is largely unchanged: given we only rate limit by a single
(fastest/slowest) replica, sorting the endpoints by back-pressure doesn't make sense either.

Answering to [~Stefania]'s latest comments:

bq. The proposal doesn't address: "The API should provide for reporting load to clients so
they can do real load balancing across coordinators and not just round-robin."

This is misleading: the implemented back-pressure mechanism is cross-replica, so load balancing
to different coordinators would be useless.

bq. we can add metrics or another mechanism later on in follow up tickets, as well as consider
implementing the memory-threshold strategy.

Metrics can be added in this ticket once we settle on the core algorithm, but then yes any
reporting mechanism to clients should be probably dealt with separately as it would probably
involve changes to the native protocol (and I'm not sure what's the usual procedure in such
case).

Regarding the memory-threshold strategy, I would like to stress once again the fact it's a
coordinator-only back-pressure mechanism which wouldn't directly benefit replicas; also, [~aweisberg]
showed in his tests that such strategy isn't even needed given the local hints back-pressure/overflowing
mechanisms already implemented. So my vote goes against it, but if you/others really want
it, I'm perfectly fine implementing it, but it will have to be in this ticket, as it requires
some more changes to the back-pressure API.

> Bound the number of in-flight requests at the coordinator
> ---------------------------------------------------------
>
>                 Key: CASSANDRA-9318
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-9318
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Local Write-Read Paths, Streaming and Messaging
>            Reporter: Ariel Weisberg
>            Assignee: Sergio Bossa
>         Attachments: 9318-3.0-nits-trailing-spaces.patch, backpressure.png, limit.btm,
no_backpressure.png
>
>
> It's possible to somewhat bound the amount of load accepted into the cluster by bounding
the number of in-flight requests and request bytes.
> An implementation might do something like track the number of outstanding bytes and requests
and if it reaches a high watermark disable read on client connections until it goes back below
some low watermark.
> Need to make sure that disabling read on the client connection won't introduce other
issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message