accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Newton (JIRA)" <j...@apache.org>
Subject [jira] [Created] (ACCUMULO-834) thrift server responds slowly to large numbers of requests
Date Fri, 26 Oct 2012 13:17:13 GMT
Eric Newton created ACCUMULO-834:
------------------------------------

             Summary: thrift server responds slowly to large numbers of requests
                 Key: ACCUMULO-834
                 URL: https://issues.apache.org/jira/browse/ACCUMULO-834
             Project: Accumulo
          Issue Type: Bug
          Components: tserver
    Affects Versions: 1.4.1
         Environment: large scale cluster
            Reporter: Eric Newton
            Assignee: Keith Turner
            Priority: Minor
             Fix For: 1.5.0


Testing on a large cluster, I found that the time to gather data from tablet servers got very
long, resulting in "blocky" graphs in the monitor.  After instrumenting the master, I found
that each request to a tablet server was taking longer, and there were many servers.  In the
tserver logs, I saw lots of adjustments to the number of threads in the tablet server.  At
a large scale, say, hundreds of servers, and hundreds of test ingesters, sometimes there will
be hundreds of threads requesting service.  The thread pool grows slowly (one new thread per
second when falling behind).  Requests from the master starve waiting for new threads to be
created.

Possible fixes:
 * use a thread pool in the master to make status requests
 * use asynchronous messaging (the old ping/pong) to make status requests
 * modify the configuration to respond to large numbers of requests more quickly
 * increase the minimum number of threads available to service requests
 * replace the existing network infrastructure to use netty
 * modify the thread-pool checker to increase the number of threads more quickly when heavily
loaded 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message