cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sylvain Lebresne (JIRA)" <>
Subject [jira] [Updated] (CASSANDRA-5926) The native protocol server can deadlock
Date Fri, 23 Aug 2013 11:59:51 GMT


Sylvain Lebresne updated CASSANDRA-5926:

    Attachment: 5926.txt

Attaching patch that switches to MemoryAwareThreadPoolExecutor. One interesting detail is
that MATPE always forces corePoolSize == maxPoolSize, so doing that pretty much deprecates
the native_transport_min_threads in the yaml. I'd say this is probably fine in practice though.

> The native protocol server can deadlock
> ---------------------------------------
>                 Key: CASSANDRA-5926
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Sylvain Lebresne
>            Assignee: Sylvain Lebresne
>             Fix For: 1.2.9
>         Attachments: 5926.txt
> Until CASSANDRA-5239 (i.e. since StorageProxy is blocking), the native protocol server
needs to use a thread per request being processed. For that, it currently use a DebuggableThreadPoolExecutor,
but with a limited queue. The rational being that we don't want to OOM if a client overwhelm
the server. Rather, we prefer blocking (which DTPE gives us) on the submission of new request
by the netty worker threads when all threads are busy.
> However, as it happens, when netty sends back a response to a query, there is cases where
some events (technically, InterestChanged and WriteComplete events) are send up the pipeline.
And those event are submitted on the request executor as other requests. Long story short,
a request thread can end blocking on the submission to its own executor, hence deadlocking.
> The simplest solution is probably to reuse MemoryAwareThreadPoolExecutor from netty rather
that our own DTPE as it also allow to block task submission when all threads are busy but
knows not to block it's own internal events.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message