cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mike Malone (JIRA)" <j...@apache.org>
Subject [jira] Issue Comment Edited: (CASSANDRA-1358) Message deserializer pool will never grow beyond a single thread.
Date Wed, 04 Aug 2010 20:28:18 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-1358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895405#action_12895405
] 

Mike Malone edited comment on CASSANDRA-1358 at 8/4/10 4:26 PM:
----------------------------------------------------------------

So are the message deserializer threads blocking on a response from (or pushing a task onto
a queue for) a different thread pool then? The problem we saw / are seeing is that the deserializer
pool backs up and it causes the server to flap because (apparently) it's unable to process
gossip requests / responses in a timely manner. Since all inter-node RPC goes through the
deserializer, this queue ends up causing all sorts of crazy havoc when it gets backed up.

Even supposing the MDP _isn't_ the bottleneck, the current code seems to be a mistake. If
the intent is to have a single threaded executor, the second argument should be 1. If the
intent is to have a multiple-threaded executor, the first argument should not be 1.

      was (Author: mmalone):
    So are the message deserializer threads blocking on a response from (or pushing a task
onto a queue for) a different thread pool then? The problem we saw / are seeing is that the
deserializer pool backs up and it causes the server to flap because (apparently) it's unable
to process get gossip requests / responses of in a timely manner. Since all inter-node RPC
goes through the deserializer, this queue ends up causing all sorts of crazy havoc when it
gets backed up.

Even supposing the MDP _isn't_ the bottleneck, the current code seems to be a mistake. If
the intent is to have a single threaded executor, the second argument should be 1. If the
intent is to have a multiple-threaded executor, the first argument should not be 1.
  
> Message deserializer pool will never grow beyond a single thread.
> -----------------------------------------------------------------
>
>                 Key: CASSANDRA-1358
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1358
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 0.6.3
>         Environment: All.
>            Reporter: Mike Malone
>            Priority: Minor
>
> The message deserialization process can become a bottleneck that prevents efficient resource
utilization because the executor that manages the deserialization process will never grow
beyond a single thread. The message deserializer executor is instantiated in the MessagingService
constructor as a JMXEnableThreadPoolExecutor, which extends java.util.concurrent.ThreadPoolExecutor.
The thread pool is instantiated with a corePoolSize of 1 and a maximumPoolSize of Runtime.getRuntime().availableProcessors().
But, according to the ThreadPoolExecutor documentation "using an unbounded queue (for example
a LinkedBlockingQueue without a predefined capacity) will cause new tasks to be queued in
cases where all corePoolSize threads are busy. Thus, no more than corePoolSize threads will
ever be created. (And the value of the maximumPoolSize therefore doesn't have any effect.)"
> The message deserializer pool uses a LinkedBlockingQueue, so there will never be more
than one deserialization thread. This issue became a problem in our production cluster when
the MESSAGE-DESERIALIZER-POOL began to back up on a node that was only lightly loaded. We
increased the core pool size to 4 and the situation improved, but the deserializer pool was
still backing up while the machine was not fully utilized (less than 100% CPU utilization).
This leads me to think that the deserializer thread is blocking on some sort of I/O, which
seems like it shouldn't happen.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message