cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rick Branson (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-7594) Disruptor Thrift server worker thread pool not adjustable
Date Wed, 23 Jul 2014 17:07:39 GMT


Rick Branson commented on CASSANDRA-7594:

The ideal situation would be if rpc_min_threads and rpc_max_threads still worked, at least
roughly. The problem with implementing this is that the TDisruptorServer only creates fixed-size
thread pools specified by numWorkersPerSelector. The variable-size thread pools are useful
because they allow working around temporary congestion problems by using a large number of
executor threads, but fall back to a smaller thread pool when unneeded (i.e. 128 -> 2048).
An example of this is when a two replicas for the same range are simultaneously slow or unresponsive
using QUORUM on RF=3. The RPC thread pool is *rapidly* consumed, which can cause fallout to
other, unrelated requests.

> Disruptor Thrift server worker thread pool not adjustable
> ---------------------------------------------------------
>                 Key: CASSANDRA-7594
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Rick Branson
>            Assignee: Pavel Yaskevich
> For the THsHaDisruptorServer, there may not be enough threads to run blocking StorageProxy
methods. The current number of worker threads is hardcoded at 2 per selector, so 2 * numAvailableProcessors(),
or 64 threads on a 16-core hyperthreaded machine. StorageProxy methods block these threads,
so this puts an upper bound on the throughput if hsha is enabled. If operations take 10ms
on average, the node can only handle a maximum of 6,400 operations per second. This is a regression
from hsha on 1.2.x, where the thread pool was tunable using rpc_min_threads and rpc_max_threads.

This message was sent by Atlassian JIRA

View raw message