tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Leon Rosenberg <rosenberg.l...@gmail.com>
Subject Re: VERY HIGH TRAFFIC TUNING
Date Wed, 09 Jul 2014 07:16:02 GMT
On Wed, Jul 9, 2014 at 4:47 AM, Hernán Marsili <hernan@cmsmedios.com> wrote:

> Hi,
>

Hello Hernán,


>
> For the past 4 years we has been working with a 'stable' configuration in
> which we put APACHE in front of TOMCAT7 (previously Tomcat6) with mod_jk
> connector. We usually serve high traffic sites with about 7000 to 10.000
> concurrent users per box (8gb RAM / 4 vcpu) (50.000 active users total).
>
> We are OK with the performance, but sometimes we notice Tomcat stops
> responding normally while there are at least 2 full CPU left to be consumed
> (JAVA memory is fine).
>

Hard to tell from here, but dropping performance while still having
resources is often an indicator for synchronization issues. You should
analyze your thread usage. You could do it with jstack (save multiple
stacks in short slots, like every 10 seconds for 2-3 minutes). Check what
threads are in what states:
 - do you have any threads in BLOCKED state? If yes, what they are waiting
for?
 - what are you RUNNABLE threads doing? Are they waiting for something,
even not blocked - for example reading the database or reading the incoming
request.
 - is your amount of TIMED_WAITING threads sufficient? If you have non,
your thread pool is probably out of threads.



>
> This is the configuration we use for the connector:
>
>  <Connector port="8009" protocol="AJP/1.3" address="127.0.0.1"
> emptySessionPath="true" redirectPort="8443" maxThreads="1024"
> minSpareThreads="32" enableLookups="false" request.registerRequests="false"
> />
>

Generally removing apache httpd can increase performance. I assume you have
a hardware loadbalancer in front of things, so httpd doesn't do you any
good.


>
> I have a couple of questions:
> 1) should we set a particular connector or let Tomcat7 decide? I understand
> using protocol="AJP/1.3" the auto-switch kicks in. But, for non-SSL high
> concurrency sites maybe is best to fixed on APR?
>

Warning, flame war is about to begin, but personally I always found that
the plain old java connector is the best option for speed. And the simplest
to use too. If you insist of having apache httpd in front, you may want to
try mod_proxy (or was it mod_proxy_ajp?) instead.


>
> 2) how many THREADS can we have? can we go beyond the 1024?
>

Depends on your OS. On modern linuxes sky is the limit. However context
switches can kill you too. If you have very fast connections, got for a
smaller amount. If you have keepalive and slow connections, remember that
every connection can hold 1-2 threads without doing anything at all. But
you can go up to 4096 without second thought, if you are really out of
threads. However, if your problem are blocked threads or a slow DB, you
will make things much much much worse.


>
> 3) is there any advantage on using processorCache?
>
> 4) We are not defining a CONNECTION TIMEOUT not a KEEP ALIVE. Any advice on
> this one? The average user session is 7 minutes.
>

If playing with CONNECTION_TIMEOUT, than better on OS level. But again that
depends on what is really happening within your application and with the
connections. You could monitor it with netstat and see if you have too many
CLOSE_WAITs or something. That it's easier to decide what to do.



>
> Thanks for the help!
> Hernán.
>

Leon.

P.S. needless to say, but having a monitoring tool like
http://www.moskito.org will help either ;-)

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message