accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Newton (JIRA)" <>
Subject [jira] [Resolved] (ACCUMULO-834) thrift server responds slowly to large numbers of requests
Date Thu, 15 Nov 2012 21:31:12 GMT


Eric Newton resolved ACCUMULO-834.

    Resolution: Fixed

fixed with r1410025
> thrift server responds slowly to large numbers of requests
> ----------------------------------------------------------
>                 Key: ACCUMULO-834
>                 URL:
>             Project: Accumulo
>          Issue Type: Bug
>          Components: tserver
>    Affects Versions: 1.4.1
>         Environment: large scale cluster
>            Reporter: Eric Newton
>            Assignee: Keith Turner
>            Priority: Minor
>             Fix For: 1.5.0
> Testing on a large cluster, I found that the time to gather data from tablet servers
got very long, resulting in "blocky" graphs in the monitor.  After instrumenting the master,
I found that each request to a tablet server was taking longer, and there were many servers.
 In the tserver logs, I saw lots of adjustments to the number of threads in the tablet server.
 At a large scale, say, hundreds of servers, and hundreds of test ingesters, sometimes there
will be hundreds of threads requesting service.  The thread pool grows slowly (one new thread
per second when falling behind).  Requests from the master starve waiting for new threads
to be created.
> Possible fixes:
>  * use a thread pool in the master to make status requests
>  * use asynchronous messaging (the old ping/pong) to make status requests
>  * modify the configuration to respond to large numbers of requests more quickly
>  * increase the minimum number of threads available to service requests
>  * replace the existing network infrastructure to use netty
>  * modify the thread-pool checker to increase the number of threads more quickly when
heavily loaded 

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message