cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Benedict (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-9318) Bound the number of in-flight requests at the coordinator
Date Fri, 08 May 2015 19:29:02 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-9318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14535339#comment-14535339
] 

Benedict commented on CASSANDRA-9318:
-------------------------------------

I'm filling the blanks for CASSANDRA-5039, since I didn't write the ticket, but that is the
only sane implementation: for one, we no longer have a BlockingQueue in most places that matter,
but more importantly a bound that blocked task submission could lead to problems with various
other systems seizing up, and e.g. write saturation stopping us serving reads, etc.

bq.  At CL=1 and the way I hear people think about availability in C* I think what you want
is to get better at failing to hinting before the coordinator or processing node overloads.
Under overload conditions CL=1 is basically synonymous with writing hints right?

Right.

bq. Maybe this is a congestion control problem? If we piggybacked information in responses
on congestion issues maybe we could make better decisions about new requests such as rejecting
a %age or going straight to hints before resources have been committed across the cluster?

Yes, I completely agree (in broad strokes, anyway), but that's much more complex than we seem
to be discussing here, and I'm pretty sure not 2.1 material. Ideally we would be sending information
back to coordinators informing them that we have been load shedding, so that they can invalidate
their handlers for us in preference to those for anyone else. Another related ticket is CASSANDRA-3852,
which also IMO depends on a system like this. 

> Bound the number of in-flight requests at the coordinator
> ---------------------------------------------------------
>
>                 Key: CASSANDRA-9318
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-9318
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Ariel Weisberg
>            Assignee: Ariel Weisberg
>             Fix For: 2.1.x
>
>
> It's possible to somewhat bound the amount of load accepted into the cluster by bounding
the number of in-flight requests and request bytes.
> An implementation might do something like track the number of outstanding bytes and requests
and if it reaches a high watermark disable read on client connections until it goes back below
some low watermark.
> Need to make sure that disabling read on the client connection won't introduce other
issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message