tomcat-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Henri Gomez" <henri.go...@gmail.com>
Subject Re: Feature request /Discussion: JK loadbalancer improvements for high load
Date Thu, 05 Jul 2007 09:41:21 GMT
Well many beans are allready available :

java.lang.Theading
java.lang.MemoryPool

I could see also some beans related to CPU load but using a Sun JVM
(not on IBM).

2007/7/5, jean-frederic clere <jfclere@gmail.com>:
> Henri Gomez wrote:
> > Something we should also check is the CPU load of the Tomcat instance.
> > May be it will be usefull also to let users/admin add their own
> > counters in the load estimation.
>
> If you want to add this to Tomcat remeber that stuff needs a JNI module
> to collect information from the OS/hardware and that is an OS depend code.
>
> Cheers
>
> Jean-Frederic
>
> >
> > For example, if some admins considers we should base the
> > load-balancing on HTTP requests or SQL access, and they have these
> > counters on their webapp applications, it will be usefull to be able
> > to get them from Tomcat to send them back to jk balancer.
> >
> > It shouldn't be too hard and very welcome for many Tomcat sites
> >
> > 2007/7/4, Rainer Jung <rainer.jung@kippdata.de>:
> >> Hi,
> >>
> >> implementing a management communication between the lb and the backend
> >> is on the roadmap for jk3. It is somehow unlikely, that this will help
> >> in your situation, because when doing a GC the jvm will no longer
> >> respond to the management channel. Traditional Mark Sweep Compact GC is
> >> not distinguishable from the outside from a halt in the backend. Of
> >> course we could think of a webapp trying to use the JMX info on memory
> >> consumption to estimate GC activity in advance, but I doubt that this
> >> will be a stable solution. There are notifications, when GCs happen, but
> >> at the moment I'm not sure, if such events exist, before, or only after
> >> a GC.
> >>
> >> I think a first step (and a better solution) would be to use modern GC
> >> algorithms like Concurrent Mark Sweep, which will most of the time
> >> reduce the GC stop times to some 10s or 100s of milliseconds (depending
> >> on heap size). CMS comes with a cost, a little more memory needed and a
> >> little more CPU needed, but the dramatically decreased stop times are
> >> worth it. Also it is quite robust since about 1-2 years.
> >>
> >> Other components will not like long GC pauses as well, like for instance
> >> cluster replication. There you configure the longest pause you accept
> >> for missing heartbeat packets before assuming a node is dead. Assuming a
> >> node being dead because of GC pauses and then the node suddenly works
> >> without having noticed itself that it outer world has changed is a very
> >> bad situation too.
> >>
> >> What we plan as a first step for jk3 is putting mod_jk on the basis of
> >> the apache APR libraries. Then we can relatively easily use our own
> >> management threads to monitor the backend status and influence the
> >> balancing decisions. As long as we do everything on top of the request
> >> handling threads we can't do complex things in a stable way.
> >>
> >> Getting jk3 out of the door will take some longer time (maybe 6 to 12
> >> months'for a release). People willing to help are welcome.
> >>
> >> Concerning the SLAs: it always makes sense to put a percentage limit on
> >> the maximum response times and error rates. A 100% below some limit
> >> clause will always be to expensive. But of course, if you can't reduce
> >> GC times and the GC runs to often, there will be no acceptable
> >> percentage for long running requests.
> >>
> >> Thank you for sharing your experiences at Langen with us!
> >>
> >> Regards,
> >>
> >> Rainer
> >>
> >> Yefym Dmukh wrote:
> >> > Hi all,
> >> > sorry for the stress but it seems that it is a time to come back to
> >> the
> >> > discussion related to the load balancing for JVM (Tomcat).
> >> >
> >> > Prehistory:
> >> > Recently we made benchmark and smoke tests of our product at the sun
> >> high
> >> > tech centre in Langen (Germany).
> >> >
> >> > As the webserver apache2.2.4 has been used, container -10xTomcat 5.5.25
> >> > and as load balancer - JK connector 1.2.23 with busyness algorithm.
> >> >
> >> >         Under the high load the strange behaviour was  observed: some
> >> > tomcat workers temporary got the non-proportional load, often 10 times
> >> > higher then the others for the relatively long periods.  As the
> >> result the
> >> > response times that usually stay under 500ms went up to 20+ sec,
> >> that in
> >> > its turn  made the overall test results almost two time worst as
> >> > estimated.
> >> >
> >> >                 At the beginning we were quite confused, because we
> >> were
> >> > sure that it was not the problem of JVM configuration and supposed that
> >> > the reason is in LB logic of mod_jk, and the both suggestions were
> >> right.
> >> >
> >> > Actually the following was happening: the LB sends requests and gets
> >> the
> >> > session sticky, continuously sending the upcoming requests to the same
> >> > cluster node. At the certain period of time the JVM started the major
> >> > garbage collection (full gc) and spent, mentioned above, 20 seconds. At
> >> > the same time jk continued to send new requests and the sticky to node
> >> > requests that led us to the situation where the one node broke the
> >> SLA on
> >> > response times.
> >> >
> >> > I ^ve been searching the web for awhile to find the LoadBalancer
> >> > implementation that takes an account the GC activity and reduces the
> >> load
> >> > accordingly case JVM is close to the major collection, but nothing
> >> found.
> >> >
> >> > Once again the LB of JVMs under the load is really an issue for
> >> production
> >> > and with optimally distributed load you are able not only to lower the
> >> > costs, but also able to prevent bad customer experience, not to mention
> >> > broken SLAs.
> >> >
> >> > Feature request:
> >> >
> >> >         All lb algorithms have to be extended with the bidirectional
> >> > connection with jvm:
> >> >              Jvm -> Lb: old gen size and the current occupancy
> >> >          Lb -> Jvm: prevent node overload and advice gc on dependent
on
> >> > parameterized free old gen space in %.
> >> >
> >> >
> >> > All the ideas and comments are appreciated.
> >> >
> >> > Regards,
> >> > Yefym.
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
> >> For additional commands, e-mail: dev-help@tomcat.apache.org
> >>
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: dev-help@tomcat.apache.org
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: dev-help@tomcat.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org


Mime
View raw message