tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christopher Schultz <>
Subject Re: Understanding tomcat + apache and I/O
Date Thu, 30 Nov 2017 17:06:04 GMT
Hash: SHA256

To whom it may concern,

On 11/29/17 3:16 PM, TurboChargedDad . wrote:
>>> So now all you have to do is upgrade to Tomcat 8.0 or, even
>>> better, Tomcat 8.5 :)
> That's the plan but it's kind of like pulling teeth.
>>> Can you expand on the "weirdness"? I see you have some more
>>> details below but I think you could be more specific.
> Let's say that there are 12 users on a given system all running a
> tomcat server that has SSL terminated on the same host. user01
> user02 user03 and son on all the way to user12.  Each user has
> their own /home/userNN directory.  Each user has their own own
> environment file in /etc/sysconfig/ '/etc/sysconfig/tomcat7@userNN
> .  In each of those files contains the various settings that are
> required for each user.  CATALINA_HOME Java path, PID etc. Each
> user starts it's own JVM in a work directory in their home 
> directory.

So... separate Tomcat processes for each user, right?

> Now imagine that user10's application starts to experience a
> database issue and the app stops responding..  It used to be true
> that everyone would stop responding because the AJP connectors were
> BIO.

That doesn't make any sense: the proxy configuration is much more
important than whether or not BIO is being used with Tomcat. If one
Tomcat is seized-up (which would happen e.g. if all db connections
were used and all BIO threads were blocking), there should be no
problem with the other workers on the proxy reaching other Tomcat

> Then the HTTP connections would stack up across the board.  The 
> stacking of the HTTP connections was expected given the situation. 
> Eventually the reverse proxy servers would die from running out of 
> memory if were didn't get the outage under control quickly enough.
Now, /that/ I can understand.

But you said you were using prefork. :/

> Now that we switched that we have had 2 outages.  In both cases the
> only tenants impacted from a performance perspective were the
> tenants experiencing the failures.

Good: if an application fails, it should expect to suffer downtime,
but it shouldn't affect the other applications. Assuming we are
talking about separate Tomcat JVM instances.

> No other alarms were detected during these outages for any other 
> tenants. Something odd does happen however. The Apache HTTP 
> connections rise for everyone along with the offending site.

> Please see the shared graph.
> This is caclulated by doing a netstat and grepping for EST then
> httpd then the AJP port that would have connections passed back to
> it. ( sudo -tt > /bin/netstat -ntp | grep EST | grep httpd | grep
> ':8125' | wc -l )

So what's :8125? The failing application or one of the good ones?

If httpd is over-provisioned (which is usually okay) and Tomcat is
stalled, then the number of hung connections from httpd -> Tomcat
rising is exactly what I'd expect: the web server can accept more
connections so it's doing that. Tomcat can't make progress so everyone

> It is the example above that determines the connection counts for
> each tenant.
> I cannot for the life of me understand how or why this is
> happening.. The only rise in connections should be detected in the
> offending application? Right?

Yes. What you've shown me is that a failing application experiences
high connection counts which makes sense. Unless I'm reading your text
above incorrectly. Or the graph?

> I can't say beyond a shadow of a doubt that the AJP connector
> threads aren't being wonky.  I am having trouble getting JMX to
> tell me that information through zabbix.

How are you querying JMX?

- -chris
Comment: GPGTools -
Comment: Using GnuPG with Thunderbird -


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message