hc-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleg Kalnichevski <ol...@apache.org>
Subject Re: How to tell if Async dispatcher thread is busy?
Date Tue, 13 Dec 2016 09:59:07 GMT
On Mon, 2016-12-12 at 21:15 +0000, Idzerda, Edan wrote:
> Hello!  Our reverse proxy uses the Async Client pool to handle connections to backend
servers.  We've been tracking a problem for a while where we observe the initial TCP connection
is made, but no thread is available to handle the SSL setup before a 10 second timeout expires.
 We get into trouble because some of our backend servers are very slow, and some of our clients
download very slowly.
> 
> 
> I'm experimenting with a patch to AbstractMultiworkerIOReactor.addChannel() to determine
whether the next dispatcher thread is "busy."  My first try was to look at bufferedSessions
from the BaseIOReactor, and go through the list of dispatchers one time to see if I can find
a free one.
> 
> 
>         int i = Math.abs(this.currentWorker++ % this.workerCount);
> 
>         for (int j = 0; j < this.workerCount; j++) {
> 
>             if (this.dispatchers[i].getSessionCount() == 0) {
> 
>                 break;
> 
>             }
> 
>             i = Math.abs(this.currentWorker++ % this.workerCount);
> 
>         }
> 
>         this.dispatchers[i].addChannel(entry);
> 
> This seems to help us in MOST of the cases we see this issue in production, but there
still seem to be a small number of threads which collide.  I'm testing a different version
which looks at AbstractIOReactor "sessions" to determine thread busy state, but it never seems
to show more than "1" session if I look at the size after piling up slow connections on top
of each other.
> 
> I have two questions:
>     Is there a better way to determine whether a thread is busy?
>     Would you be willing to accept a patch to make the dispatchers array in AbstractMultiworkerIOReactor
"protected" so I can implement my own ConnectingIOReactor that overrides addChannel() with
my own thread selection model?
> 
> Thanks a lot for your help and for providing such a great library to the community!
> 
> - edan
> 

Hi Edan

What I do not quite understand is why i/o dispatch threads get blocked
for 10 seconds or longer. This sounds awfully suspicious.

I could imagine exposing the list of i/o dispatchers to subclasses of
AbstractMultiworkerIOReactor in 4.4.x branch but would rather prefer to
keep it as a last resort.

Oleg


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org


Mime
View raw message