tomcat-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yefym Dmukh <Yefym.Dm...@icw-global.com>
Subject Feature request /Discussion: JK loadbalancer improvements for high load
Date Wed, 04 Jul 2007 14:05:52 GMT
Hi all,
sorry for the stress but it seems that it is a time to come back to  the 
discussion related to the load balancing for JVM (Tomcat).

Prehistory:
Recently we made benchmark and smoke tests of our product at the sun high 
tech centre in Langen (Germany). 
 
As the webserver apache2.2.4 has been used, container -10xTomcat 5.5.25 
and as load balancer - JK connector 1.2.23 with busyness algorithm. 

        Under the high load the strange behaviour was  observed: some 
tomcat workers temporary got the non-proportional load, often 10 times 
higher then the others for the relatively long periods.  As the result the 
response times that usually stay under 500ms went up to 20+ sec, that in 
its turn  made the overall test results almost two time worst as 
estimated. 

                At the beginning we were quite confused, because we were 
sure that it was not the problem of JVM configuration and supposed that 
the reason is in LB logic of mod_jk, and the both suggestions were right. 

Actually the following was happening: the LB sends requests and gets the 
session sticky, continuously sending the upcoming requests to the same 
cluster node. At the certain period of time the JVM started the major 
garbage collection (full gc) and spent, mentioned above, 20 seconds. At 
the same time jk continued to send new requests and the sticky to node 
requests that led us to the situation where the one node broke the SLA on 
response times. 

I ^ve been searching the web for awhile to find the LoadBalancer 
implementation that takes an account the GC activity and reduces the load 
accordingly case JVM is close to the major collection, but nothing found.

Once again the LB of JVMs under the load is really an issue for production 
and with optimally distributed load you are able not only to lower the 
costs, but also able to prevent bad customer experience, not to mention 
broken SLAs. 

Feature request:

        All lb algorithms have to be extended with the bidirectional 
connection with jvm:
             Jvm -> Lb: old gen size and the current occupancy
         Lb -> Jvm: prevent node overload and advice gc on dependent on 
parameterized free old gen space in %. 
 

All the ideas and comments are appreciated.

Regards,
Yefym.
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message