tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Emmanuel Courreges <ecourre...@orb.com>
Subject mod_jk request freeze with load balancing
Date Sat, 20 Jan 2007 02:44:53 GMT
Hello all,
Some answers to requests freeze and never return, which is a problem for 
setting up the load balancing in production environment. Usually if the 
user refreshes and the request is made again it doesn't freeze again, so 
it's pretty random.
This seems to happen only:
- when the answer is big (>16kBytes)
- with the balanced tomcat which is not on localhost

My ethereal and mod_jk traces (cf. later) show me that data for the next 
chunk is received by the server but mod_jk doesn't see it.

My setup is basically 2 Redhat Enterprise 4 32bits up to date,
- mod_jk 1.2.20 compiled by myself
- tomcat 5.5.17
- jdk1.6

Apache+mod_jk load balances to the local server and to a remote server. 
port 8009 is open in iptables and they are on the same switch AND data 
keeps getting received so a network/firewall thing is not the issue 
here. It balances well 50/50 except that some requests that get balanced 
to the remote server never return.

I have noticed this problem for more than a year now (with older 
versions of everything), which is why I did hold off on load balancing.
Does anyone experience the same problem/have a solution?

Thank you!
Emmanuel Courreges.



The first chunks arrive fine, then you'll see from the last characters 
of the last chunk displayed in mod_jk (e.g. search for QOe8), that it's 
not the last thing sent at all in this tcp connection.
It's definitely blocked somewhere, because when the request works fine I 
get a message: "AJP13 protocol: Reuse is OK" at the end, and here 
nothing happens after this chunk.


[Fri Jan 19 13:05:10 2007] [11579:64192] [debug] 
ajp_connection_tcp_send_message::jk_ajp_common.c (893): 0300    62 37 00 
FF 00 00 00 00 00 00 00 00 00 00
 00 00  - b7..............
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug] 
ajp_send_request::jk_ajp_common.c (1273): request body to send 0 - 
request body to resend 0
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug] 
ajp_connection_tcp_get_message::jk_ajp_common.c (1029): received from 
ajp13 pos=0 len=146 max=8192
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug] 
ajp_connection_tcp_get_message::jk_ajp_common.c (1029): 0000    04 00 C8 
00 02 4F 4B 00 00 05 00 0D 43 61
 63 68  - .....OK.....Cach
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug] 
ajp_connection_tcp_get_message::jk_ajp_common.c (1029): 0010    65 2D 43 
6F 6E 74 72 6F 6C 00 00 11 6E 6F
 2D 63  - e-Control...no-c
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug] 
ajp_connection_tcp_get_message::jk_ajp_common.c (1029): 0020    61 63 68 
65 2C 20 70 72 69 76 61 74 65 00
 00 06  - ache,.private...
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug] 
ajp_connection_tcp_get_message::jk_ajp_common.c (1029): 0030    50 72 61 
67 6D 61 00 00 08 4E 6F 2D 43 61
 63 68  - Pragma...No-Cach
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug] 
ajp_connection_tcp_get_message::jk_ajp_common.c (1029): 0040    65 00 00 
07 45 78 70 69 72 65 73 00 00 02
 2D 31  - e...Expires...-1
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug] 
ajp_connection_tcp_get_message::jk_ajp_common.c (1029): 0050    00 00 0C 
43 6F 6E 74 65 6E 74 2D 54 79 70
 65 00  - ...Content-Type.
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug] 
ajp_connection_tcp_get_message::jk_ajp_common.c (1029): 0060    00 17 74 
65 78 74 2F 68 74 6D 6C 3B 63 68
 61 72  - ..text/html;char
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug] 
ajp_connection_tcp_get_message::jk_ajp_common.c (1029): 0070    73 65 74 
3D 55 54 46 2D 38 00 00 10 43 6F
 6E 74  - set=UTF-8...Cont
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug] 
ajp_connection_tcp_get_message::jk_ajp_common.c (1029): 0080    65 6E 74 
2D 4C 61 6E 67 75 61 67 65 00 00
 02 65  - ent-Language...e
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug] 
ajp_connection_tcp_get_message::jk_ajp_common.c (1029): 0090    6E 00 00 
00 00 00 00 00 00 00 00 00 00 00
 00 00  - n...............
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug] 
ajp_unmarshal_response::jk_ajp_common.c (603): status = 200
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug] 
ajp_unmarshal_response::jk_ajp_common.c (610): Number of headers is = 5
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug] 
ajp_unmarshal_response::jk_ajp_common.c (666): Header[0] [Cache-Control] 
= [no-cache, private]
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug] 
ajp_unmarshal_response::jk_ajp_common.c (666): Header[1] [Pragma] = 
[No-Cache]
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug] 
ajp_unmarshal_response::jk_ajp_common.c (666): Header[2] [Expires] = [-1]
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug] 
ajp_unmarshal_response::jk_ajp_common.c (666): Header[3] [Content-Type] 
= [text/html;charset=UTF-8]
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug] 
ajp_unmarshal_response::jk_ajp_common.c (666): Header[4] 
[Content-Language] = [en]
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug] 
ajp_connection_tcp_get_message::jk_ajp_common.c (1029): received from 
ajp13 pos=0 len=8188 max=8192
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug] 
ajp_connection_tcp_get_message::jk_ajp_common.c (1029): 0000    03 1F F8 
3C 68 74 6D 6C 3E 0D 0A 3C 68 65
 61 64  - ...<html>..<head
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug] 
ajp_connection_tcp_get_message::jk_ajp_common.c (1029): 0010    3E 0D 0A 
09 3C 74 69 74 6C 65 3E 4F 72 62
 3C 2F  - >...<title>Orb</
...
 7F 20  - +..!G&5sSJT?p=..
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug] 
ajp_connection_tcp_get_message::jk_ajp_common.c (1029): 03f0    42 3D 7F 
20 28 3E 33 7F 20 40 3D 4F 7F 20
 21 3F  - B=..(>3..@=O..!?
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug] ws_write::mod_jk.c 
(436): written 8184 out of 8184
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug] 
ajp_connection_tcp_get_message::jk_ajp_common.c (1029): received from 
ajp13 pos=0 len=8188 max=8192
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug] 
ajp_connection_tcp_get_message::jk_ajp_common.c (1029): 0000    03 1F F8 
65 76 65 39 68 21 74 62 3E 40 5B
 28 7F  - ...eve9h!tb>@[(.

...
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug] 
ajp_connection_tcp_get_message::jk_ajp_common.c (1029): 03c0    2A 3D 4C 
3E 24 76 47 7A 71 71 55 3D 2C 29
 3A 31  - *=L>$vGzqqU=,):1
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug] 
ajp_connection_tcp_get_message::jk_ajp_common.c (1029): 03d0    55 7F 39 
5F 3B 4D 6A 6C 4F 22 71 34 32 7F
 24 5C  - U.9_;MjlO"q42.$\
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug] 
ajp_connection_tcp_get_message::jk_ajp_common.c (1029): 03e0    40 23 5E 
32 32 29 34 3D 32 2C 7A 3E 3F 5C
 67 35  - @#^22)4=2,z>?\g5
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug] 
ajp_connection_tcp_get_message::jk_ajp_common.c (1029): 03f0    64 3E 5C 
51 4F 65 38 23 34 4E 23 26 31 28
 39 3F  - d>\QOe8#4N#&1(9?
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug] ws_write::mod_jk.c 
(436): written 8184 out of 8184


ETHEREAL:

45311 13:05:10.600576 208.96.4.147 -> 208.96.4.140 AJP13 143:RSP:SEND 
BODY CHUNK[Unreassembled Packet]

0000  00 11 11 3e c5 8d 00 30 48 56 4e c6 08 00 45 00   ...>...0HVN...E.
0010  05 dc 0f d7 40 00 40 06 7b 65 d0 60 04 93 d0 60   ....@.@.{e.`...`
0020  04 8c 1f 49 98 1a b7 2d 80 27 05 b6 71 2b 80 10   ...I...-.'..q+..
0030  21 4f ce cd 00 00 01 01 08 0a c4 8d 79 10 d8 0f   !O..........y...
0040  c2 2d 41 42 1f fc 03 1f f8 65 76 65 39 68 21 74   .-AB.....eve9h!t
0050  62 3e 40 5b 28 7f 63 6a 59 5d 24 26 6d 21 54 37   b>@[(.cjY]$&m!T7
0060  27 3d 38 2e 69 55 65 2e 54 51 55 42 2f 32 6d 51   '=8.iUe.TQUB/2mQ
0070  28 25 55 76 38 20 73 21 2d 26 3b 65 27 29 63 29   (%Uv8 s!-&;e')c)
0080  2b 45 21 21 56 34 77 2f 3e 62 2b 29 36 2c 6a 38   +E!!V4w/>b+)6,j8
0090  5c 27 39 24 7f 20 4a 2a 33 24 2a 28 4d 5a 47 7d   \'9$. J*3$*(MZG}
00a0  47 38 5a 32 25 37 2e 2f 34 25 3e 7c 70 38 4b 42   G8Z2%7./4%>|p8KB
00b0  24 46 26 6c 71 31 31 3c 67 71 26 46 3c 79 30 3e   $F&lq11<gq&F<y0>
00c0  25 28 3a 34 2d 33 66 6c 7f 79 3d 2b 2c 4c 70 5f   %(:4-3fl.y=+,Lp_
00d0  35 24 55 3a 7c 31 70 33 39 41 6e 26 2e 65 20 62   5$U:|1p39An&.e b
00e0  22 33 23 3e 36 34 72 7e 6e 3a 36 6d 24 3b 23 2c   "3#>64r~n:6m$;#,
00f0  2b 6b 77 2c 7b 3a 5d 24 24 20 3a 3a 2f 66 2b 7a   +kw,{:]$$ ::/f+z
0100  53 23 43 2a 27 37 2b 3d 38 35 5b 36 7b 3a 51 42   S#C*'7+=85[6{:QB
0110  4b 6e 37 2a 5a 49 5e 4e 42 21 31 73 47 4f 3a 33   Kn7*ZI^NB!1sGO:3
0120  57 62 69 39 22 49 62 33 77 68 39 60 25 4a 67 27   Wbi9"Ib3wh9`%Jg'
0130  27 71 3d 79 67 2a 71 6b 57 27 65 25 3d 59 6b 46   'q=yg*qkW'e%=YkF
0140  55 29 24 7b 2e 3e 58 56 40 47 66 2a 79 2c 31 3e   U)${.>XV@Gf*y,1>
0150  77 63 5b 3d 6b 36 46 40 63 20 48 32 45 7f 20 5f   wc[=k6F@c H2E. _
0160  36 3d 2c 4f 50 35 27 6e 2f 51 58 52 3b 5c 4d 3a   6=,OP5'n/QXR;\M:
0170  55 27 2a 55 35 7b 2b 71 24 2d 24 26 35 34 38 63   U'*U5{+q$-$&548c
0180  67 4c 6a 54 33 68 4c 78 77 48 2a 76 38 29 64 72   gLjT3hLxwH*v8)dr
0190  27 39 29 20 39 74 2e 45 2f 24 23 54 2f 67 5b 42   '9) 9t.E/$#T/g[B
01a0  56 70 26 77 31 2d 52 52 24 39 39 2b 42 38 2a 2a   Vp&w1-RR$99+B8**
01b0  44 2e 44 73 20 25 40 6b 3f 71 29 56 72 2c 25 27   D.Ds %@k?q)Vr,%'
01c0  44 57 34 28 66 64 3b 21 39 60 7b 71 31 5c 5e 6d   DW4(fd;!9`{q1\^m
01d0  36 7d 24 37 3a 23 4d 24 25 6d 24 37 6b 4d 24 2c   6}$7:#M$%m$7kM$,
01e0  55 43 39 28 30 22 2f 39 2e 41 2c 3f 51 57 35 61   UC9(0"/9.A,?QW5a
01f0  2a 21 2d 3f 5a 39 44 3f 3c 2c 29 20 77 27 3d 61   *!-?Z9D?<,) w'=a
0200  29 27 5a 2b 3a 32 3a 65 37 28 29 41 3c 56 34 24   )'Z+:2:e7()A<V4$
0210  5c 33 25 47 5a 64 2b 27 2f 66 30 2e 38 7e 68 77   \3%GZd+'/f0.8~hw
0220  2d 3d 23 4a 48 21 35 4c 4e 62 2d 5b 50 2d 68 47   -=#JH!5LNb-[P-hG
0230  63 5c 60 3f 4a 3b 62 2a 3c 3a 26 55 4e 3a 36 78   c\`?J;b*<:&UN:6x
0240  2e 54 75 2e 37 3b 72 24 4b 2b 29 36 3b 28 3c 34   .Tu.7;r$K+)6;(<4
0250  24 36 39 61 4f 20 7d 55 35 2e 22 24 5b 32 39 64   $69aO }U5."$[29d
0260  42 26 22 3b 42 35 23 54 77 2f 2a 2f 5c 64 47 33   B&";B5#Tw/*/\dG3
0270  4e 3a 2b 79 2e 23 2e 45 55 36 39 77 6e 37 6c 4e   N:+y.#.EU69wn7lN
0280  2a 5b 6c 4b 2e 24 7f 31 53 29 40 29 3e 59 22 31   *[lK.$.1S)@)>Y"1
0290  22 2a 21 30 32 2c 60 3e 25 47 7f 7a 29 54 27 22   "*!02,`>%G.z)T'"
02a0  51 5a 21 53 67 27 26 53 39 63 4c 25 41 68 54 49   QZ!Sg'&S9cL%AhTI
02b0  24 60 43 76 36 66 51 30 39 4e 41 61 2e 7a 3a 2b   $`Cv6fQ09NAa.z:+
02c0  70 40 2c 30 48 31 32 3c 38 4f 34 4d 76 31 54 64   p@,0H12<8O4Mv1Td
02d0  2f 3a 20 2f 4e 39 54 39 6a 6b 39 24 6a 37 4d 73   /: /N9T9jk9$j7Ms
02e0  2d 4b 76 31 7d 27 74 3e 53 3c 59 65 35 77 35 44   -Kv1}'t>S<Ye5w5D
02f0  51 5d 21 64 73 5b 5a 26 2f 49 5d 30 45 6c 56 50   Q]!ds[Z&/I]0ElVP
0300  2f 34 2e 4e 6a 2e 2e 7a 2f 7e 7a 3a 4a 50 7d 6b   /4.Nj..z/~z:JP}k
0310  6b 53 33 3e 4e 32 2d 29 20 2d 22 34 47 3c 2b 30   kS3>N2-) -"4G<+0
0320  26 5f 21 38 6b 33 6b 2f 7a 30 32 79 30 2c 49 3b   &_!8k3k/z02y0,I;
0330  34 31 3b 41 2b 4e 2b 32 23 23 2c 72 7d 2a 27 52   41;A+N+2##,r}*'R
0340  32 4a 2c 21 2a 35 27 48 5d 74 31 2f 39 5a 69 22   2J,!*5'H]t1/9Zi"
0350  4b 71 39 78 46 54 39 76 6c 5c 73 48 6c 2a 44 30   Kq9xFT9vl\sHl*D0
0360  47 75 3f 59 59 64 65 49 2b 5b 5d 6f 20 30 3d 7d   Gu?YYdeI+[]o 0=}
0370  4f 78 57 6c 63 75 34 56 28 4a 3b 3d 39 2a 6d 6e   OxWlcu4V(J;=9*mn
0380  2d 42 46 3e 66 65 26 31 22 3f 39 20 20 37 62 36   -BF>fe&1"?9  7b6
0390  7d 68 20 2f 6e 38 3a 5c 23 2b 3f 2d 26 70 3e 3f   }h /n8:\#+?-&p>?
03a0  52 3a 2a 72 2f 2a 54 5a 32 55 76 3a 24 7e 43 5c   R:*r/*TZ2Uv:$~C\
03b0  52 32 2c 76 6e 43 4d 2f 3c 21 24 46 33 75 47 3f   R2,vnCM/<!$F3uG?
03c0  4e 2a 34 2b 2f 39 36 52 3f 3c 49 39 7b 42 41 27   N*4+/96R?<I9{BA'
03d0  36 26 3b 32 7a 73 49 2f 2a 35 2c 64 65 72 2a 34   6&;2zsI/*5,der*4
03e0  5f 3a 34 39 39 66 56 3d 6d 2b 23 31 36 2e 45 5e   _:499fV=m+#16.E^
03f0  64 33 73 30 3d 7f 20 3a 75 68 64 7f 20 29 32 32   d3s0=. :uhd. )22
0400  41 22 4b 65 37 6e 2a 3d 4c 3e 24 76 47 7a 71 71   A"Ke7n*=L>$vGzqq
0410  55 3d 2c 29 3a 31 55 7f 39 5f 3b 4d 6a 6c 4f 22   U=,):1U.9_;MjlO"
0420  71 34 32 7f 24 5c 40 23 5e 32 32 29 34 3d 32 2c   q42.$\@#^22)4=2,
0430  7a 3e 3f 5c 67 35 64 3e 5c 51 4f 65 38 23 34 4e   z>?\g5d>\QOe8#4N
0440  23 26 31 28 39 3f 79 5b 2e 4d 7e 25 30 25 4a 35   #&1(9?y[.M~%0%J5
0450  7b 3f 38 71 67 35 27 29 4e 5d 4f 44 50 6d 56 20   {?8qg5')N]ODPmV
0460  40 2d 20 48 30 65 5f 36 21 29 23 21 7f 29 2b 2e   @- H0e_6!)#!.)+.
0470  4d 26 72 30 49 25 33 3f 78 78 66 76 55 6c 2f 3c   M&r0I%3?xxfvUl/<
0480  23 74 7a 3a 62 4c 35 27 28 32 3b 3a 3e 3f 24 34   #tz:bL5'(2;:>?$4
0490  5d 25 20 49 5e 35 36 63 65 36 7c 6d 2f 66 36 24   ]% I^56ce6|m/f6$
04a0  23 5b 3c 23 51 45 73 3b 61 20 2a 50 4b 3b 7e 63   #[<#QEs;a *PK;~c
04b0  2c 4e 3c 21 5e 2a 4a 4f 4d 31 6e 55 2b 2c 54 6d   ,N<!^*JOM1nU+,Tm
04c0  23 23 33 40 4b 4a 37 2a 71 3f 3d 24 6c 4e 47 5c   ##3@KJ7*q?=$lNG\
04d0  63 52 32 5b 37 5e 63 32 48 5d 48 2f 3e 32 2e 54   cR2[7^c2H]H/>2.T
04e0  2e 48 5e 7d 38 2d 5e 3a 35 7f 20 78 4e 7a 34 5a   .H^}8-^:5. xNz4Z
04f0  37 2a 2d 5a 29 34 31 71 32 39 38 38 28 5d 41 42   7*-Z)41q2988(]AB
0500  63 20 33 53 27 74 27 46 38 62 60 4a 44 3f 34 6b   c 3S't'F8b`JD?4k
0510  26 37 22 29 42 22 4f 2d 4d 73 53 2c 63 51 24 2d   &7")B"O-MsS,cQ$-
0520  79 3d 3c 3e 25 79 4c 28 71 3b 27 59 3c 35 5d 36   y=<>%yL(q;'Y<5]6
0530  6c 32 43 2e 73 53 58 54 5b 52 56 72 30 62 4c 28   l2C.sSXT[RVr0bL(
0540  3c 4f 3b 2b 67 57 53 2f 73 21 72 53 6e 7b 39 4e   <O;+gWS/s!rSn{9N
0550  23 29 25 27 22 46 24 4b 24 5b 24 46 43 3b 4b 66   #)%'"F$K$[$FC;Kf
0560  49 2b 36 57 27 3a 25 3d 27 21 27 35 56 2c 30 59   I+6W':%='!'5V,0Y
0570  48 21 4b 46 32 31 5e 4a 33 2c 64 71 6b 53 37 28   H!KF21^J3,dqkS7(
0580  46 23 72 36 26 5c 23 6e 31 2a 29 3b 34 72 2d 69   F#r6&\#n1*);4r-i
0590  45 27 39 3c 3d 7b 31 46 2e 7a 2c 55 38 76 23 28   E'9<={1F.z,U8v#(
05a0  2d 77 40 7d 5c 6a 42 4a 3d 3d 4d 4f 34 32 44 37   -w@}\jBJ==MO42D7
05b0  53 5e 47 45 3a 35 64 79 6a 32 5a 49 32 49 2a 4a   S^GE:5dyj2ZI2I*J
05c0  2c 4c 64 2e 2b 32 44 7f 6e 3a 2e 7b 3d 39 2c 23   ,Ld.+2D.n:.{=9,#
05d0  49 3c 31 7c 69 2a 36 58 43 27 66 3e 7e 3d 2a 69   I<1|i*6XC'f>~=*i
05e0  53 23 53 30 26 5d 2d 2a 55 32                     S#S0&]-*U2

45312 13:05:10.600592 208.96.4.147 -> 208.96.4.140 TCP [Continuation to 
#45312] 8009 > 38938 [ACK] Seq=2204919 Ack=85681 Win=34108 Len=1448 
TSV=3297605904 T
SER=3624911405

0000  00 11 11 3e c5 8d 00 30 48 56 4e c6 08 00 45 00   ...>...0HVN...E.
0010  05 dc 0f d9 40 00 40 06 7b 63 d0 60 04 93 d0 60   ....@.@.{c.`...`
0020  04 8c 1f 49 98 1a b7 2d 85 cf 05 b6 71 2b 80 10   ...I...-....q+..
0030  21 4f f2 aa 00 00 01 01 08 0a c4 8d 79 10 d8 0f   !O..........y...


---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message