hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nicolas Liochon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-11590) use a specific ThreadPoolExecutor
Date Thu, 15 Oct 2015 09:18:05 GMT

    [ https://issues.apache.org/jira/browse/HBASE-11590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14958560#comment-14958560

Nicolas Liochon commented on HBASE-11590:

The issue is that the ThreadPoolExecutor  leaked all over the place, often for monitoring
All lot of code depends on ThreadPoolExecutor  rather than the ExecutorService...

For example, see 
 * This class will coalesce increments from a thift server if
 * hbase.regionserver.thrift.coalesceIncrement is set to true. Turning this
 * config to true will cause the thrift server to queue increments into an
 * instance of this class. The thread pool associated with this class will drain
 * the coalesced increments as the thread is able. This can cause data loss if the
 * thrift server dies or is shut down before everything in the queue is drained.
public class IncrementCoalescer implements IncrementCoalescerMBean {
// snip
  // MBean get/set methods
  public int getQueueSize() {
    return pool.getQueue().size();
  public int getMaxQueueSize() {
    return this.maxQueueSize;
  public void setMaxQueueSize(int newSize) {
    this.maxQueueSize = newSize;

  public long getPoolCompletedTaskCount() {
    return pool.getCompletedTaskCount();
  public long getPoolTaskCount() {
    return pool.getTaskCount();
  public int getPoolLargestPoolSize() {
    return pool.getLargestPoolSize();
  public int getCorePoolSize() {
    return pool.getCorePoolSize();
  public void setCorePoolSize(int newCoreSize) {
  public int getMaxPoolSize() {
    return pool.getMaximumPoolSize();
  public void setMaxPoolSize(int newMaxSize) {

I'm going to limit this patch to the easy/client stuff...

> use a specific ThreadPoolExecutor
> ---------------------------------
>                 Key: HBASE-11590
>                 URL: https://issues.apache.org/jira/browse/HBASE-11590
>             Project: HBase
>          Issue Type: Bug
>          Components: Client, Performance
>    Affects Versions: 1.0.0, 2.0.0
>            Reporter: Nicolas Liochon
>            Assignee: Nicolas Liochon
>            Priority: Minor
>             Fix For: 2.0.0
>         Attachments: ExecutorServiceTest.java, LifoThreadPoolExecutorSQP.java, UnitQueueP.java,
UnitQueuePU.java, tp.patch
> The JDK TPE creates all the threads in the pool. As a consequence, we create (by default)
256 threads even if we just need a few.
> The attached TPE create threads only if we have something in the queue.
> On a PE test with replica on, it improved the 99 latency percentile by 5%. 
> Warning: there are likely some race conditions, but I'm posting it here because there
is may be an implementation available somewhere we can use, or a good reason not to do that.
So feedback welcome as usual. 

This message was sent by Atlassian JIRA

View raw message