tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rainer Jung <rainer.j...@kippdata.de>
Subject Re: jk_handler::mod_jk.c (2917): Could not get endpoint for worker ...
Date Fri, 21 Sep 2018 15:34:47 GMT
Am 15.09.2018 um 12:50 schrieb Clemens Wyss DEV:
> Hi all,
> we are seeing quite a few:
> "[Mon Sep 10 15:19:46 2018] [27562:140532026529536] [error] jk_handler::mod_jk.c (2917):
Could not get endpoint for worker=testAPJ"
> 
> errors in our md_jk.log. Worker properties are as follwos:
> 
> ...
> worker.list=testAPJ
> 
> worker.testAPJ.port=8009
> worker.testAPJ.host=127.0.0.1
> worker.testAPJ.type=ajp13
> worker.testAPJ.socket_keepalive=1
> worker.testAJP.connection_pool_timeout=600
> ...
> 
> At that point Apache seems to be stuck/struggling (but our tomcat does not seem to be
under pressure). Restarting Apache solves the issue ... till it pops up again ...
> 
> What is happening? What needs tob e tuned?
> 
> Apache 2.4.34, tried both event- and worker-MPM

Assuming this is mod_jk 1.2.44? Are there more setting for worker testAPJ?

Normally mod_jk creates as many local connection structures (named 
endpoints) in each Apache httpd child process, as that process has 
worker threads. When an httpd worker thread wants to talk to tomcat, it 
retrieves such an endpoint and uses it to create and handle the 
commnunication.

The error you observe means, that all endpoints were already in use. 
Since we create as many structures as there are worker threads - 
everything is per httpd process, this should not happen (and I don't 
remember any case were it did happen).

Ideas what could go wrong:

- setting the worker property connection_pool_size or the deprecated 
cachesize for worker testAPJ to a smaller value than your httpd 
ThreadsPerChild (32 from your config snippet). If not set, mod_jk 
automatically detects the number of httpd worker threads

- setting connection_acquire_timeout to a small value. By default it is 
equals to retries*retry_interval which in turn by default is equals to 
2*100 milliseconds. mod_jk will retry getting an endpoint before it 
shows you error message "retries" times with a sleep pause of 
"retry_interval" milliseconds but no longer than 
connection_acquire_timeout milliseconds.

- retrieving and endpoint must acquire a lock first. On some platforms 
locking can lead to problems like false positives in deadlock detection. 
But i think this can't happen here since the code doesn't check the 
return value of the locking.

- memory shortage leading to failing allocations (not likely but possible)

Do you see any other log messages? Any ones in the httpd error log or 
especially the mod_jk log? There should be a WARN message of type 
"Unable to get the free endpoint for worker %s from %u slots" but maybe 
more before that final problem happens? What do you see with JkLogLevel 
info?

Does the problem happen under high load or when your backend gets slow? 
What does "netstat -anp | grep 8009" show when the hang occurs?

Regards,

Rainer

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message