cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ariel Weisberg (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-9318) Bound the number of in-flight requests at the coordinator
Date Thu, 14 Jul 2016 15:00:24 GMT


Ariel Weisberg commented on CASSANDRA-9318:

I incorrectly thought request threads block until all responses are received. They only block
until enough responses to satisfy the consistency level are received. So running out of request
threads is not always going to be an issue because you have enough slow nodes involved in
the request.

So rate limiting will work in that it can artificially increase your CL (not really but still)
to reduce throughput and avoid timeouts. This will also have the effect of preventing the
coordinator from using more memory because request threads can't meet their CL and move on
to process new requests. Instead they will block in a rate limiter. 

My measurements showed that you can provision enough memory to let the timeout kick in so
I am not sure that is a useful behavior. Sure it eliminates timeouts, but if that is the end
goal maybe we need a consistency level that is something like CL.ALL but tolerates unavailable
nodes. That would have the same effect without rate limiting. It's still a partial solution
because you can't write at full speed to ranges that don't contain slow nodes.

> Bound the number of in-flight requests at the coordinator
> ---------------------------------------------------------
>                 Key: CASSANDRA-9318
>                 URL:
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Local Write-Read Paths, Streaming and Messaging
>            Reporter: Ariel Weisberg
>            Assignee: Sergio Bossa
>         Attachments: 9318-3.0-nits-trailing-spaces.patch, backpressure.png, limit.btm,
> It's possible to somewhat bound the amount of load accepted into the cluster by bounding
the number of in-flight requests and request bytes.
> An implementation might do something like track the number of outstanding bytes and requests
and if it reaches a high watermark disable read on client connections until it goes back below
some low watermark.
> Need to make sure that disabling read on the client connection won't introduce other

This message was sent by Atlassian JIRA

View raw message