tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Samad - Yieldbroker <Alex.Sa...@yieldbroker.com>
Subject Issues with the tomcat connector (On W2k8 + IIS7.5)
Date Tue, 06 Mar 2012 05:35:55 GMT
Hi

This is my prod setup

Client -> Internet -> IIS7.5/NLB (2 node cluster) with the 1.2.32 tomcat connector ->
2 node tomcat cluster.

We had done a fair bit of testing on this setup enough to move into production.  But we ran
into a problem ... when IIS recycle its application pool. Luckily we had set this to manually
happen after hours

So I setup a test rig to see if I could simulate the problem.  The test rig is

Client -> 1 node of IIS7.5  + 1.2.32 tomcat connector -> 2 node of development tomcat
:)

The client works by creating a connection and keeping it open and making lots of requests.
 

We have jerry rigged a client to make multiple of these as the error occurs when the system
is under load.

So the test scenario is 
start client
login >>> this creates 1 connection 
strat stress test >>> this creates 500 connection and does a little bit of polling

My initial setup is 
4 process worker threads for the application pool
Allow overlapping recycle >>>> my understanding is this means the 4 threads are
not all terminated at once, but one at a time as a new thread is restarted

I use isapi_redirect.properties file
============================
extension_uri=/jakarta/isapi_redirect.dll
log_file=C:\ThisNow\Local\dev.ThisNow.com\Logs\ajpconfisapi_redirect.log
log_level=warn
log_filesize=20M
worker_file=C:\ThisNow\Shared\dev.ThisNow.com\ajpconfig\workers.properties
worker_mount_file=C:\ThisNow\Shared\dev.ThisNow.com\ajpconfig\uriworkermap.properties
rewrite_rule_file=C:\ThisNow\Shared\dev.ThisNow.com\ajpconfig\\rewrite.properties


C:\ThisNow\Shared\dev.ThisNow.com\ajpconfig\workers.properties
====================================================
worker.maintain=60
worker.list=jbclb
worker.jbclb.type=lb
worker.jbclb.balance_workers=worker1, worker2
worker.jbclb.sticky_session=true
worker.jbclb.method=Request
worker.jbclb.lock=pessimistic

# Template for all worker threads
worker.template.type=ajp13
worker.template.port=8009
worker.template.ping_mode=A
worker.template.ping_timeout=30000
worker.template.connection_pool_size=1000
worker.template.connection_pool_minsize=100
worker.template.socket_keepalive=true
worker.template.lock=pessimistic
worker.template.max_packet_size=65536

# worker 1
worker.worker1.host=10.172.204.30
worker.worker1.route=node1
worker.worker1.reference=worker.template

# worker 1
worker.worker2.route=node2
worker.worker2.host=10.172.204.31
worker.worker2.reference=worker.template

worker.list=jkstatus
# Define a 'jkstatus' worker using status
worker.jkstatus.type=status



C:\ThisNow\Shared\dev.ThisNow.com\ajpconfig\uriworkermap.properties
========================================================= 
/dev.ThisNow.com/ThisNowLive|/*=jbclb
/dev.ThisNow.com/wls|/*=jbclb
/dev.ThisNow.com/jkmanager|/*=jkstatus
/10.172.204.33/jkmanager|/*=jkstatus
/ThisNowodevrp01/jkmanager|/*=jkstatus




C:\ThisNow\Shared\dev.ThisNow.com\ajpconfig\rewrite.properties
=================================================== 
# rewrite rules
# Convert single node 
~/jb./(.*)=/$1
/web-debts-1.0.0-nossl/=/web-debts-1.0.0/


I can see there are 500 connection to IIS and ~500 connection through to tomcat. Check with
perfmon, netstat...

When I recycle the application pool. IIS starts to send 500's back to the server.... it complain
about the tomcat connector returning an error errorcode 2147942401, internal server error
Incorrect Function 0c80070001

What I can see from the network side of things. Is the client is trying to recreate all 500
connections at once.

What I can see in my tomcat connector logs... some times there is 4 for each process and sometimes
2 or 3, if its crashed into this 500 error loop


Examples of what I see (2 log files)
Ajp file 1
===
[Tue Mar 06 14:46:46.572 2012] [5872:5028] [error] ajp_worker_factory::jk_ajp_common.c (2997):
allocating ajp worker record from shared memory 
[Tue Mar 06 14:46:46.572 2012] [5872:5028] [error] wc_create_worker::jk_worker.c (151): factory
for ajp13 failed for worker2 
m shared memory 
[Tue Mar 06 14:46:46.572 2012] [5872:5028] [error] validate::jk_lb_worker.c (1615): Failed
creating worker worker2 
worker2 
[Tue Mar 06 14:46:46.587 2012] [2888:5708] [error] ajp_worker_factory::jk_ajp_common.c (2997):
allocating ajp worker record from shared memory 
[Tue Mar 06 14:46:46.603 2012] [5872:5028] [error] validate::jk_lb_worker.c (1663): NULL parameters

[Tue Mar 06 14:46:46.603 2012] [5872:5028] [error] wc_create_worker::jk_worker.c (163): validate
failed for jbclb 
 worker2 
[Tue Mar 06 14:46:46.619 2012] [5872:5028] [error] build_worker_map::jk_worker.c (262): failed
to create worker jbclb 
[Tue Mar 06 14:46:46.619 2012] [5872:5028] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.619 2012] [5872:5028] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jbclb' in uri map post processing. 
[Tue Mar 06 14:46:46.634 2012] [5872:5028] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.634 2012] [5872:5028] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jbclb' in uri map post processing. 
[Tue Mar 06 14:46:46.634 2012] [1536:1332] [error] validate::jk_lb_worker.c (1663): NULL parameters

d not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.634 2012] [5872:5028] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.634 2012] [5872:5028] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jbclb' in uri map post processing. 
[Tue Mar 06 14:46:46.634 2012] [1536:1332] [error] wc_create_worker::jk_worker.c (163): validate
failed for jbclb 
[Tue Mar 06 14:46:46.634 2012] [5872:5028] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jbclb' in uri map post processing. 
[Tue Mar 06 14:46:46.634 2012] [5872:5028] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.650 2012] [5872:5028] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.650 2012] [1536:1332] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.650 2012] [1536:1332] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jbclb' in uri map post processing. 
[Tue Mar 06 14:46:46.665 2012] [1536:1332] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.665 2012] [1536:1332] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jbclb' in uri map post processing. 
[Tue Mar 06 14:46:46.665 2012] [1536:1332] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.665 2012] [2888:5708] [error] build_worker_map::jk_worker.c (262): failed
to create worker jbclb 
with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.665 2012] [1536:1332] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jbclb' in uri map post processing. 
[Tue Mar 06 14:46:46.665 2012] [2888:5708] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.681 2012] [2888:5708] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jbclb' in uri map post processing. 
 
[Tue Mar 06 14:46:46.681 2012] [2888:5708] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.681 2012] [2888:5708] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jbclb' in uri map post processing. 
[Tue Mar 06 14:46:46.681 2012] [2888:5708] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.681 2012] [2888:5708] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.681 2012] [2888:5708] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jbclb' in uri map post processing. 
[Tue Mar 06 14:46:46.681 2012] [2888:5708] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jbclb' in uri map post processing. 
[Tue Mar 06 14:46:46.681 2012] [2888:5708] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.697 2012] [2888:5708] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing.  

Ajp log file 2
============== 
[Tue Mar 06 14:46:46.650 2012] [5872:3624] [warn] init_jk::jk_isapi_plugin.c (2738): The optimal
shared memory size can now be determined automatically. 
[Tue Mar 06 14:46:46.650 2012] [5872:3624] [warn] init_jk::jk_isapi_plugin.c (2740): You can
remove the shm_size attribute if you want to use the optimal size. 
[Tue Mar 06 14:46:46.665 2012] [5872:3624] [error] wc_create_worker::jk_worker.c (151): factory
for lb failed for jbclb 
[Tue Mar 06 14:46:46.665 2012] [5872:3624] [error] build_worker_map::jk_worker.c (262): failed
to create worker jbclb 
[Tue Mar 06 14:46:46.665 2012] [5872:3624] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.665 2012] [5872:3624] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jbclb' in uri map post processing. 
[Tue Mar 06 14:46:46.665 2012] [5872:3624] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.665 2012] [5872:3624] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jbclb' in uri map post processing. 
[Tue Mar 06 14:46:46.665 2012] [5872:3624] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.665 2012] [5872:3624] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.681 2012] [5872:3624] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jbclb' in uri map post processing. 
[Tue Mar 06 14:46:46.681 2012] [5872:3624] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jbclb' in uri map post processing. 
[Tue Mar 06 14:46:46.681 2012] [5872:3624] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.681 2012] [5872:3624] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.697 2012] [5872:2836] [warn] init_jk::jk_isapi_plugin.c (2738): The optimal
shared memory size can now be determined automatically. 
[Tue Mar 06 14:46:46.697 2012] [2888:4792] [warn] init_jk::jk_isapi_plugin.c (2738): The optimal
shared memory size can now be determined automatically. 
[Tue Mar 06 14:46:46.697 2012] [5872:2836] [warn] init_jk::jk_isapi_plugin.c (2740): You can
remove the shm_size attribute if you want to use the optimal size. 
[Tue Mar 06 14:46:46.712 2012] [2888:4792] [warn] init_jk::jk_isapi_plugin.c (2740): You can
remove the shm_size attribute if you want to use the optimal size. 
[Tue Mar 06 14:46:46.712 2012] [2888:4792] [error] wc_create_worker::jk_worker.c (151): factory
for lb failed for jbclb 
w be determined automatically. 
[Tue Mar 06 14:46:46.728 2012] [2888:4792] [error] build_worker_map::jk_worker.c (262): failed
to create worker jbclb 
te if you want to use the optimal size. 
[Tue Mar 06 14:46:46.728 2012] [2888:4792] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.728 2012] [5872:2836] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.728 2012] [2888:4792] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jbclb' in uri map post processing. 
[Tue Mar 06 14:46:46.728 2012] [1536:1492] [error] build_worker_map::jk_worker.c (262): failed
to create worker jbclb 
[Tue Mar 06 14:46:46.728 2012] [2888:4792] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.728 2012] [1536:1492] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.744 2012] [5872:2836] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.744 2012] [2888:4792] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jbclb' in uri map post processing. 
[Tue Mar 06 14:46:46.744 2012] [1536:1492] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jbclb' in uri map post processing. 
[Tue Mar 06 14:46:46.744 2012] [5872:2836] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jbclb' in uri map post processing. 
[Tue Mar 06 14:46:46.744 2012] [2888:4792] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.744 2012] [1536:1492] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.744 2012] [5872:2836] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.744 2012] [2888:4792] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.744 2012] [5872:2836] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.744 2012] [2888:4792] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jbclb' in uri map post processing. 
[Tue Mar 06 14:46:46.759 2012] [1536:1492] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.759 2012] [5872:2836] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jbclb' in uri map post processing. 
[Tue Mar 06 14:46:46.759 2012] [2888:4792] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jbclb' in uri map post processing. 
[Tue Mar 06 14:46:46.759 2012] [1536:1492] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.759 2012] [5872:2836] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jbclb' in uri map post processing. 
[Tue Mar 06 14:46:46.759 2012] [2888:4792] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.759 2012] [1536:1492] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jbclb' in uri map post processing. 
[Tue Mar 06 14:46:46.759 2012] [5872:2836] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.759 2012] [1536:1492] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jbclb' in uri map post processing. 
 
[Tue Mar 06 14:46:46.759 2012] [5872:2836] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.775 2012] [1536:1492] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.775 2012] [1536:1492] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.775 2012] [5872:3380] [warn] init_jk::jk_isapi_plugin.c (2738): The optimal
shared memory size can now be determined automatically. 
[Tue Mar 06 14:46:46.790 2012] [5872:3380] [warn] init_jk::jk_isapi_plugin.c (2740): You can
remove the shm_size attribute if you want to use the optimal size. 
[Tue Mar 06 14:46:46.790 2012] [1536:1272] [warn] init_jk::jk_isapi_plugin.c (2740): You can
remove the shm_size attribute if you want to use the optimal size. 
[Tue Mar 06 14:46:46.790 2012] [2888:1548] [warn] init_jk::jk_isapi_plugin.c (2740): You can
remove the shm_size attribute if you want to use the optimal size. 
[Tue Mar 06 14:46:46.790 2012] [1536:1272] [error] wc_create_worker::jk_worker.c (151): factory
for lb failed for jbclb 
[Tue Mar 06 14:46:46.790 2012] [2888:1548] [error] wc_create_worker::jk_worker.c (151): factory
for lb failed for jbclb 
[Tue Mar 06 14:46:46.806 2012] [1536:1272] [error] build_worker_map::jk_worker.c (262): failed
to create worker jbclb 
with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.806 2012] [2888:1548] [error] build_worker_map::jk_worker.c (262): failed
to create worker jbclb 
[Tue Mar 06 14:46:46.806 2012] [5872:3380] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jbclb' in uri map post processing. 
[Tue Mar 06 14:46:46.806 2012] [1536:1272] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.806 2012] [2888:1548] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.806 2012] [5872:3380] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.806 2012] [1536:1272] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jbclb' in uri map post processing. 
[Tue Mar 06 14:46:46.806 2012] [2888:1548] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jbclb' in uri map post processing. 
[Tue Mar 06 14:46:46.822 2012] [5872:3380] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jbclb' in uri map post processing. 
[Tue Mar 06 14:46:46.822 2012] [2888:1548] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.822 2012] [1536:1272] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jbclb' in uri map post processing. 
 
[Tue Mar 06 14:46:46.822 2012] [2888:1548] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jbclb' in uri map post processing. 
[Tue Mar 06 14:46:46.822 2012] [5872:3380] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.822 2012] [2888:1548] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.837 2012] [1536:1272] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.837 2012] [2888:1548] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.837 2012] [1536:1272] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jbclb' in uri map post processing. 
[Tue Mar 06 14:46:46.837 2012] [2888:1548] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jbclb' in uri map post processing. 
[Tue Mar 06 14:46:46.837 2012] [1536:1272] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jbclb' in uri map post processing. 
[Tue Mar 06 14:46:46.837 2012] [2888:1548] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jbclb' in uri map post processing. 
 
[Tue Mar 06 14:46:46.853 2012] [1536:1272] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.853 2012] [5872:3380] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.853 2012] [1536:1272] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.853 2012] [2888:1548] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.869 2012] [2888:3536] [warn] init_jk::jk_isapi_plugin.c (2738): The optimal
shared memory size can now be determined automatically. 
[Tue Mar 06 14:46:46.869 2012] [2888:3536] [warn] init_jk::jk_isapi_plugin.c (2740): You can
remove the shm_size attribute if you want to use the optimal size. 
[Tue Mar 06 14:46:46.869 2012] [1536:5072] [warn] init_jk::jk_isapi_plugin.c (2738): The optimal
shared memory size can now be determined automatically. 
[Tue Mar 06 14:46:46.884 2012] [1536:5072] [warn] init_jk::jk_isapi_plugin.c (2740): You can
remove the shm_size attribute if you want to use the optimal size. 
[Tue Mar 06 14:46:46.884 2012] [2888:3536] [error] build_worker_map::jk_worker.c (262): failed
to create worker jbclb 
[Tue Mar 06 14:46:46.884 2012] [1536:5072] [error] wc_create_worker::jk_worker.c (151): factory
for lb failed for jbclb 
[Tue Mar 06 14:46:46.884 2012] [1536:5072] [error] build_worker_map::jk_worker.c (262): failed
to create worker jbclb 
with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.884 2012] [1536:5072] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.884 2012] [2888:3536] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jbclb' in uri map post processing. 
[Tue Mar 06 14:46:46.884 2012] [1536:5072] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jbclb' in uri map post processing. 
[Tue Mar 06 14:46:46.900 2012] [2888:3536] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.900 2012] [1536:5072] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.900 2012] [2888:3536] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jbclb' in uri map post processing. 
[Tue Mar 06 14:46:46.900 2012] [2888:3536] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.900 2012] [2888:3536] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.915 2012] [2888:3536] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jbclb' in uri map post processing. 
[Tue Mar 06 14:46:46.915 2012] [2888:3536] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jbclb' in uri map post processing. 
[Tue Mar 06 14:46:46.915 2012] [2888:3536] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.915 2012] [1536:5072] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing. 
[Tue Mar 06 14:46:46.915 2012] [2888:3536] [error] uri_worker_map_ext::jk_uri_worker_map.c
(506): Could not find worker with name 'jkstatus' in uri map post processing.  



During normal startup I don't see anything logged at all.

My initial testing was to move from overlapping recycle to non-overlapping recycle. This seems
to have solved it for 99% of the time. But if I get enough connection trying to hit the server
at 1 time, in my case > ~450 it will cause this corruption some of the times :)

My best solution is to go to single process thread and non over lapping (although I would
have thought non overlapping shouldn't have an effect, but maybe that is how IIS decides to
kill first or afterwards!)


So my conclusion / presumption is there is an issue with the shared memory locking and also
what seems to be the initialisation code sequencing.

Under load, IIS will first start 1 process and then if enough request another and another
depending on load.  So what I see is 4 process starting at about the same time and the shared
memory is getting corrupted and from there it turns to poo.

I can presume it get corrupted for 2 reasons.  Firstly the connector logs faults unable to
map a URI and then stop warning about that, I can presume that, the process in question hasn't
reads its config and then eventual does !

If it hasn't read its config and it processing uri's and presumable touching the SHM, it is
doing so under an optimistic locking method... That in windows land using in process locking
methods, which don't work across processes ! (this is different to linux/unix world). 

So I am thinking 2 changes need to be made. The connector needs to default to pessimistic
locking (maybe the default for windows).... It also depends on how to fix the init code. 
If you can guarantee that the config file will be processed before the Shared memory is touched,
then maybe its okay to leave the default to optimistic.

The second bug, the connector needs to be fully initialised before it starts to process any
URI's... This is not good ... but I think it a problem with Web Gardens (this is the name
given by MS for more than 1 process in the application pool)
When it was architected I can guess this wouldn't have been the setup.  I could presume (haven't
looked at the init stuff yet), but if it find the SHM memory already created, it thinks it's
ready to go !


I don't mind digging around the code and supplying patches, but I am more used to the linux
world and well , I have tried to build the connector in VS 2010 and .....

Does somebody have a tar  ball of a tree what I can just point to and compile :) (or maybe
some instruction on how to get it to build from the svn tree in VS2010)


I have logged a bug report 
https://issues.apache.org/bugzilla/show_bug.cgi?id=52659, I think it might be linked to https://issues.apache.org/bugzilla/show_bug.cgi?id=52286


Thanks
Alex

Sorry for the long email




---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message