accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ACCUMULO-834) thrift server responds slowly to large numbers of requests
Date Thu, 15 Nov 2012 22:35:13 GMT

    [ https://issues.apache.org/jira/browse/ACCUMULO-834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13498408#comment-13498408
] 

Hudson commented on ACCUMULO-834:
---------------------------------

Integrated in Accumulo-Trunk #553 (See [https://builds.apache.org/job/Accumulo-Trunk/553/])
    ACCUMULO-834 adjust threadpool size quickly: storm events (like root tablet moves) cause
lots of connections on large clusters (Revision 1410025)

     Result = SUCCESS
ecn : 
Files : 
* /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/util/TServerUtils.java

                
> thrift server responds slowly to large numbers of requests
> ----------------------------------------------------------
>
>                 Key: ACCUMULO-834
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-834
>             Project: Accumulo
>          Issue Type: Bug
>          Components: tserver
>    Affects Versions: 1.4.1
>         Environment: large scale cluster
>            Reporter: Eric Newton
>            Assignee: Keith Turner
>            Priority: Minor
>             Fix For: 1.5.0
>
>
> Testing on a large cluster, I found that the time to gather data from tablet servers
got very long, resulting in "blocky" graphs in the monitor.  After instrumenting the master,
I found that each request to a tablet server was taking longer, and there were many servers.
 In the tserver logs, I saw lots of adjustments to the number of threads in the tablet server.
 At a large scale, say, hundreds of servers, and hundreds of test ingesters, sometimes there
will be hundreds of threads requesting service.  The thread pool grows slowly (one new thread
per second when falling behind).  Requests from the master starve waiting for new threads
to be created.
> Possible fixes:
>  * use a thread pool in the master to make status requests
>  * use asynchronous messaging (the old ping/pong) to make status requests
>  * modify the configuration to respond to large numbers of requests more quickly
>  * increase the minimum number of threads available to service requests
>  * replace the existing network infrastructure to use netty
>  * modify the thread-pool checker to increase the number of threads more quickly when
heavily loaded 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message