Mailing-List: contact users-help@tomcat.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Tomcat Users List" <users@tomcat.apache.org>
Received-SPF: pass (herse.apache.org: domain of lists@nabble.com designates
 72.21.53.35 as permitted sender)
Message-ID: <8896404.post@talk.nabble.com>
Date: Fri, 9 Feb 2007 16:56:56 -0800 (PST)
From: babu <babulalsatasiya@gmail.com>
To: users@tomcat.apache.org
Subject: Re: [OT] mod_jk -> tomcat sending delay
In-Reply-To: <45CD0CF5.7090302@kippdata.de>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
References: <epco8o$c8g$1@sea.gmane.org> <8894508.post@talk.nabble.com>
 <45CD0CF5.7090302@kippdata.de>


Hi,

We tried updating it once but we have problem in load balancing and we went
back to original mod_jk which we working fine. We upgraded to 1.2.15, but w=
e
are planing to try out new one 1.2.20, but I am watching this discussion
board and there issues even in 1.2.20 and I have posted because of thius.

With regards,

Babulal Satasiya


Rainer Jung-3 wrote:
>=20
> Version 1.2.6 of mod_jk is an antiquity. First update before=20
> investigating the problem.
>=20
> babu wrote:
>> Hi,
>>=20
>> We are also facing same problem and we are using mod_jk 1.2.6 with apach=
e
>> 2.0.53 and tomcat 5.0.27. Mod_jk is configured with loadbalancer. We hav=
e
>> Apache+mod_jk with LB +  9 tomcats in backend. Mod_jk is set with sticky
>> bit
>> set.
>>=20
>> Partial workers.properites
>>=20
>> worker.list=3Dloadbalancer
>> worker.loadbalancer.type=3Dlb
>> worker.loadbalancer.balanced_workers=3Dtc_1_1,tc_1_2,tc_1_3,tc_2_1,tc_2_=
2,tc_2_3,tc_3_1,tc_3_2,tc_3_3
>>=20
>>=20
>>=20
>> worker.tc_1_1.host=3Dcna-prod-app-25
>> worker.tc_1_1.port=3D8000
>> worker.tc_1_1.type=3Dajp13
>> worker.tc_1_1.lbfactor=3D1
>> worker.tc_1_1.cachesize=3D50
>> worker.tc_1_1.cache_timeout=3D180
>> worker.tc_1_1.socket_timeout=3D300
>> worker.tc_1_1.socket_keepalive=3D1
>> worker.tc_1_1.recovery_options=3D0
>> worker.tc_1_1.sticky_session=3D1
>>=20
>> server.xml
>>=20
>> <Connector port=3D"8000"
>>               protocol=3D"AJP/1.3"
>>               address=3D"cna-prod-app-25"
>>               minProcessors=3D"50"
>>               maxProcessors=3D"100"
>>               enableLookups=3D"false"
>>               debug=3D"0"
>>               URIEncoding=3D"UTF-8"
>>                           connectionTimeout=3D"180000"
>>               />
>>=20
>> I found in one of discussion board, that connection timeout in sever.xml
>> and
>> timeout parameters in workers.properties should be matched properly and
>> also
>> found that in server.xml its in miliseconds and in workers.properites it
>> is
>> seconds.
>>=20
>>=20
>> We do not find any issue at client side but erorr keeps going in mod_jk
>> log.
>> ( I am not sure of client side issues are there or not)
>>=20
>>=20
>> mod_jk log
>>=20
>> [Fri Feb 09 22:00:55 2007]  [jk_ajp_common.c (1250)]: Tomcat is down or
>> network problems. No response has been sent to the client (yet)
>> [Fri Feb 09 22:00:55 2007]  [jk_ajp_common.c (1449)]: ERROR: Receiving
>> from
>> tomcat failed, recoverable operation. err=3D2
>> [Fri Feb 09 22:00:55 2007]  [jk_ajp_common.c (1477)]: Error connecting t=
o
>> tomcat. Tomcat is probably not started or is listening on the wrong port=
.
>> worker=3Dtc_3_3 failed errno =3D 0
>> [Fri Feb 09 22:01:21 2007]  [jk_ajp_common.c (783)]: ERROR: can't receiv=
e
>> the response message from tomcat, network problems or tomcat is down
>> (128.107.229.109:8002), err=3D-1
>> [Fri Feb 09 22:01:21 2007]  [jk_ajp_common.c (1250)]: Tomcat is down or
>> network problems. No response has been sent to the client (yet)
>> [Fri Feb 09 22:01:21 2007]  [jk_ajp_common.c (1449)]: ERROR: Receiving
>> from
>> tomcat failed, recoverable operation. err=3D2
>> [Fri Feb 09 22:01:21 2007]  [jk_ajp_common.c (1477)]: Error connecting t=
o
>> tomcat. Tomcat is probably not started or is listening on the wrong port=
.
>> worker=3Dtc_1_3 failed errno =3D 0
>> [Fri Feb 09 22:01:41 2007]  [jk_ajp_common.c (783)]: ERROR: can't receiv=
e
>> the response message from tomcat, network problems or tomcat is down
>> (128.107.229.110:8001), err=3D-1
>> [Fri Feb 09 22:01:41 2007]  [jk_ajp_common.c (1250)]: Tomcat is down or
>> network problems. No response has been sent to the client (yet)
>> [Fri Feb 09 22:01:41 2007]  [jk_ajp_common.c (1449)]: ERROR: Receiving
>> from
>> tomcat failed, recoverable operation. err=3D2
>> [Fri Feb 09 22:01:41 2007]  [jk_ajp_common.c (1477)]: Error connecting t=
o
>> tomcat. Tomcat is probably not started or is listening on the wrong port=
.
>> worker=3Dtc_2_2 failed errno =3D 0
>> [Fri Feb 09 22:01:44 2007]  [jk_ajp_common.c (783)]: ERROR: can't receiv=
e
>> the response message from tomcat, network problems or tomcat is down
>> (128.107.229.109:8000), err=3D-1
>> [Fri Feb 09 22:01:44 2007]  [jk_ajp_common.c (1250)]: Tomcat is down or
>> network problems. No response has been sent to the client (yet)
>> [Fri Feb 09 22:01:44 2007]  [jk_ajp_common.c (1449)]: ERROR: Receiving
>> from
>> tomcat failed, recoverable operation. err=3D0
>> [Fri Feb 09 22:02:03 2007]  [jk_ajp_common.c (783)]: ERROR: can't receiv=
e
>> the response message from tomcat, network problems or tomcat is down
>> (128.107.229.110:8002), err=3D-1
>> [Fri Feb 09 22:02:03 2007]  [jk_ajp_common.c (1250)]: Tomcat is down or
>> network problems. No response has been sent to the client (yet)
>> [Fri Feb 09 22:02:03 2007]  [jk_ajp_common.c (1449)]: ERROR: Receiving
>> from
>> tomcat failed, recoverable operation. err=3D1
>> [Fri Feb 09 22:02:49 2007]  [jk_ajp_common.c (783)]: ERROR: can't receiv=
e
>> the response message from tomcat, network problems or tomcat is down
>> (128.107.229.110:8002), err=3D-1
>> [Fri Feb 09 22:02:49 2007]  [jk_ajp_common.c (1250)]: Tomcat is down or
>> network problems. No response has been sent to the client (yet)
>> [Fri Feb 09 22:02:49 2007]  [jk_ajp_common.c (1449)]: ERROR: Receiving
>> from
>> tomcat failed, recoverable operation. err=3D0
>> [Fri Feb 09 22:02:59 2007]  [jk_ajp_common.c (783)]: ERROR: can't receiv=
e
>> the response message from tomcat, network problems or tomcat is down
>> (128.107.229.110:8001), err=3D-1
>> [Fri Feb 09 22:02:59 2007]  [jk_ajp_common.c (1250)]: Tomcat is down or
>> network problems. No response has been sent to the client (yet)
>> [Fri Feb 09 22:02:59 2007]  [jk_ajp_common.c (1449)]: ERROR: Receiving
>> from
>> tomcat failed, recoverable operation. err=3D2
>> [Fri Feb 09 22:02:59 2007]  [jk_ajp_common.c (1477)]: Error connecting t=
o
>> tomcat. Tomcat is probably not started or is listening on the wrong port=
.
>> worker=3Dtc_2_2 failed errno =3D 0
>>=20
>> If you find any soluttion to this then please update me with your
>> solution.
>>=20
>> with regards,
>>=20
>> Babulal Satasiya
>>=20
>> Cisco System Inc.
>> Sanjose, CA.
>>=20
>>=20
>> Marcin Zaj=C4=85czkowski wrote:
>>> Hi,
>>>
>>>
>>> I observe in mod_jk's log errors:
>>> [error] ajp_get_reply::jk_ajp_common.c (1566): (ajp13w) Tomcat is down=
=20
>>> or refused connection. No response has been sent to the client (yet)
>>>
>>> In the rush hours (20-30 requests/second) up to several entries per
>>> hour.
>>> Sometimes it's followed by:
>>> [error] ajp_service::jk_ajp_common.c (1928): (ajp13w) Connecting to=20
>>> tomcat failed. Tomcat is probably not started or is listening on the=20
>>> wrong port
>>> or
>>> [error] ajp_connection_tcp_get_message::jk_ajp_common.c (948): (ajp13w)=
=20
>>> can't receive the response message from tomcat, network problems or=20
>>> tomcat is down (127.0.0.1:8009), err=3D-104
>>>
>>> but usually it is covered by another retries. It's corresponding to=20
>>> exception in tomcat:
>>> ERROR [org.apache.jk.common.HandlerRequest] Error decoding request
>>> java.net.SocketTimeoutException: Read timed out
>>>
>>> *Note* Post is quite long because of info from logs. To get know what i=
t=20
>>> is about it is enough to read descriptions next to numbers.
>>>
>>>
>>> I increased logs granularity (mod_jk and tomcat + tcpdump) and in a few=
=20
>>> checked issues it looks like:
>>>
>>> *1*. mod_jk talks with tomcat and some request is serviced.
>>>
>>> (...)
>>> [11:10:30] [debug] ajp_process_callback::jk_ajp_common.c (1491): AJP13=
=20
>>> protocol: Reuse is OK
>>> [debug] ajp_done::jk_ajp_common.c (2258): recycling connection pool=20
>>> slot=3D0 for worker ajp13w
>>> [debug] jk_handler::mod_jk.c (2126): Service finished with status=3D200=
=20
>>> for worker=3Dwlb
>>>
>>> *2*. A few seconds later thread is used again by an another request.
>>>
>>> [11:10:32] [debug] wc_get_worker_for_name::jk_worker.c (114): found a=
=20
>>> worker wlb
>>> [debug] wc_get_name_for_type::jk_worker.c (290): Found worker type 'lb'
>>> [debug] init_ws_service::mod_jk.c (584): Service protocol=3DHTTP/1.1=20
>>> method=3DPOST (...)
>>> [debug] service::jk_lb_worker.c (840): service sticky_session=3D1=20
>>> id=3D'8A49C0D934C8A115342BA5CBFC79C434'
>>> [debug] service::jk_lb_worker.c (860): service worker=3Dajp13w
>>> route=3Dajp13w
>>> [debug] ajp_get_endpoint::jk_ajp_common.c (2315): acquired connection=
=20
>>> pool slot=3D0
>>> [debug] ajp_marshal_into_msgb::jk_ajp_common.c (548): ajp marshaling
>>> done
>>> [debug] ajp_service::jk_ajp_common.c (1771): processing ajp13w with 2=
=20
>>> retries
>>> [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (893): sending=
=20
>>> to ajp13 pos=3D4 len=3D307 max=3D8192
>>> (...) [header is sent and it is received by tomcat]
>>>
>>> *3*.
>>> a) mod_jk tries to send body:
>>>
>>> [11:10:32] [debug] ajp_send_request::jk_ajp_common.c (1273): request=20
>>> body to send 646 - request body to resend 0
>>>
>>> b) tomcat is waiting for it:
>>> DEBUG [org.apache.jk.common.ChannelSocket] Call next 0=20
>>> org.apache.jk.common.HandlerRequest@1ca8b3d
>>> DEBUG [org.apache.jk.common.HandlerRequest] Handling 2
>>> DEBUG [org.apache.jk.common.ChannelSocket] receive()
>>>
>>> *4*. 5 seconds later (connector timeout is 5000ms) tomcat throws
>>> exception:
>>> [11:10:37] ERROR [org.apache.jk.common.HandlerRequest] Error decoding=
=20
>>> request
>>> java.net.SocketTimeoutException: Read timed out
>>> and resets connection (regarding to tcpdump log)
>>>
>>> *5*. A few more seconds later (e.g. 8 from point 3) mod_jk starts=20
>>> sending a body:
>>> [11:10:40] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c=20
>>> (893): sending to ajp13 pos=3D4 len=3D652 max=3D8192
>>> [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (893): 0000=20
>>> 12 34 02 88 02 86 3C 3F (...) [body's dump]
>>>
>>> *6*. Unfortunately connection was already reseted by tomcat and mod_jk=
=20
>>> got info about that:
>>> [11:10:40] [info]  ajp_connection_tcp_get_message::jk_ajp_common.c=20
>>> (942): (ajp13w) Tomcat has forced a connection close for socket 28
>>> [error] ajp_get_reply::jk_ajp_common.c (1566): (ajp13w) Tomcat is down=
=20
>>> or refused connection. No response has been sent to the client (yet)
>>> [info]  ajp_service::jk_ajp_common.c (1877): (ajp13w) receiving from=20
>>> tomcat failed, recoverable operation attempt=3D0
>>>
>>> *7*. mod_jk tries to do send request again to tomcat:
>>> [11:10:40] [info]  ajp_service::jk_ajp_common.c (1916): (ajp13w) sendin=
g=20
>>> request to tomcat failed,  recoverable operation attempt=3D1
>>> [debug] jk_open_socket::jk_connect.c (335): socket TCP_NODELAY set to O=
n
>>> [debug] jk_open_socket::jk_connect.c (433): trying to connect socket 28=
=20
>>> to 127.0.0.1:8009
>>> [debug] jk_open_socket::jk_connect.c (459): socket 28 connected to=20
>>> 127.0.0.1:8009
>>> [debug] ajp_connect_to_endpoint::jk_ajp_common.c (847): Connected socke=
t=20
>>> 28 to (127.0.0.1:8009)
>>> [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (893): sending=
=20
>>> to ajp13 pos=3D4 len=3D307 max=3D8192
>>>
>>> Usually (but not always) it is able to do that without errors.
>>>
>>> I sniffed into mod_jk source code, but I haven't found a place where it=
=20
>>> can stuck.
>>>
>>>
>>> *Question*
>>> What can be delay (about 8 seconds) in mod_jk between sending HTTP=20
>>> header and HTTP content caused by?
>>>
>>>
>>> I use Apache 2.0.52 (with MPM prefork), mod_jk 1.2.20, JBoss 3.2.5 with=
=20
>>> Tomcat 5.0 (probably 5.0.27) and Red Hat Enterprise Linux ES 4.
>>>
>>> mod_jk configuration is quite simple, without load balancing (and wasn'=
t=20
>>> changed for years).
>>>
>>> <workers.properties>
>>> workers.tomcat_home=3D/jboss/deploy/jbossweb-tomcat50.sar
>>> workers.java_home=3D/jboss/java
>>> ps=3D/
>>>
>>> worker.list=3Dwlb,jkstatus
>>>
>>> worker.ajp13w.type=3Dajp13
>>> worker.ajp13w.host=3Dlocalhost
>>> worker.ajp13w.port=3D8009
>>>
>>> worker.wlb.type=3Dlb
>>> worker.wlb.balance_workers=3Dajp13w
>>> </workers.properties>
>>>
>>> The only change in mod_jk config was:
>>> JkShmSize 1024k
>>>
>>> <servel.xml>
>>>        <Connector port=3D"8009" address=3D"127.0.0.1"
>>>         minProcessors=3D"5" maxProcessors=3D"100"
>>>         enableLookups=3D"false" acceptCount=3D"100" debug=3D"0"
>>>         connectionTimeout=3D"5000" useURIValidationHack=3D"false"
>>>         protocol=3D"AJP/1.3"/>
>>> </servel.xml>
>>>
>>>
>>> Thanks for your advice
>>> Marcin
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To start a new topic, e-mail: users@tomcat.apache.org
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>>>
>>=20
>=20
> --=20
> kippdata informationstechnologie GmbH
> Bornheimer Str. 33a
> 53111 Bonn
>=20
> Tel.: 0228/98549-0
> Fax:  0228/98549-50
> www.kippdata.de
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> kippdata informationstechnologie GmbH
> Bornheimer Str. 33a
> D-53111 Bonn
>=20
> Tel.: +49/0228/98549-0
> Fax:  +49/0228/98549-50
> www.kippdata.de
>=20
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>=20
>=20
>=20

--=20
View this message in context: http://www.nabble.com/mod_jk--%3E-tomcat-send=
ing-delay-tf3121799.html#a8896404
Sent from the Tomcat - User mailing list archive at Nabble.com.


---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org