cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ariel Weisberg (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-9318) Bound the number of in-flight requests at the coordinator
Date Mon, 14 Dec 2015 23:13:46 GMT


Ariel Weisberg commented on CASSANDRA-9318:

Further review shows that with an 8 gigabyte heap backpressure isn't getting a chance to kick
in. Several memtables end up being queued at once for flushing but the peak memory utilization
only a few hundred megabytes. I am back to not having a workload that demonstrate that backpressure
has value.

I tried artificially constraining disk throughput to 32 megabytes/sec and ended up hitting
OOM in hints. Hints appear to allocate ByteBuffers indefinitely so they can't provide backpressure
based on available disk capacity/throughput anymore. Hints are submitted directly from the
contributing thread so there is no such thing as in-flight hint anymore. I'll clean up the
existing hint overload protection so that it kicks in based on number of pending buffers and
see where that gets me.

I am guessing that in practice hint overload is going to be a lot harder to hit now that it
is based on flat files. Clearly sequential IO performance outstrips the ability to process
even large incoming mutations.

> Bound the number of in-flight requests at the coordinator
> ---------------------------------------------------------
>                 Key: CASSANDRA-9318
>                 URL:
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Local Write-Read Paths, Streaming and Messaging
>            Reporter: Ariel Weisberg
>            Assignee: Ariel Weisberg
>             Fix For: 2.1.x, 2.2.x
> It's possible to somewhat bound the amount of load accepted into the cluster by bounding
the number of in-flight requests and request bytes.
> An implementation might do something like track the number of outstanding bytes and requests
and if it reaches a high watermark disable read on client connections until it goes back below
some low watermark.
> Need to make sure that disabling read on the client connection won't introduce other

This message was sent by Atlassian JIRA

View raw message