Mailing-List: contact users-help@tomcat.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Tomcat Users List" <users@tomcat.apache.org>
Subject: Re: Understanding tomcat + apache and I/O
To: users@tomcat.apache.org
References: <CAJaHpydDUeQ5evP2gp=woNwRkhAJjD7C2E0tXT=32oWZ2Wsf=A@mail.gmail.com>
 <35790bfb-7421-ee1d-0da3-5174bb75b8a8@christopherschultz.net>
 <CAJaHpycubdb5E2y+7syWg0OUhPMDZA35owC5Et+VbBPf5YeiTg@mail.gmail.com>
From: Christopher Schultz <chris@christopherschultz.net>
Message-ID: <7f59ebc7-98ef-880d-303b-d1042b1e7360@christopherschultz.net>
Date: Thu, 30 Nov 2017 12:06:04 -0500
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:52.0)
 Gecko/20100101 Thunderbird/52.4.0
MIME-Version: 1.0
In-Reply-To: <CAJaHpycubdb5E2y+7syWg0OUhPMDZA35owC5Et+VbBPf5YeiTg@mail.gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 8bit
archived-at: Thu, 30 Nov 2017 17:06:17 -0000

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

To whom it may concern,

On 11/29/17 3:16 PM, TurboChargedDad . wrote:
>>> So now all you have to do is upgrade to Tomcat 8.0 or, even
>>> better, Tomcat 8.5 :)
> That's the plan but it's kind of like pulling teeth.
> 
>>> Can you expand on the "weirdness"? I see you have some more
>>> details below but I think you could be more specific.
> 
> Let's say that there are 12 users on a given system all running a
> tomcat server that has SSL terminated on the same host. user01
> user02 user03 and son on all the way to user12.  Each user has
> their own /home/userNN directory.  Each user has their own own
> environment file in /etc/sysconfig/ '/etc/sysconfig/tomcat7@userNN
> .  In each of those files contains the various settings that are
> required for each user.  CATALINA_HOME Java path, PID etc. Each
> user starts it's own JVM in a work directory in their home 
> directory.

So... separate Tomcat processes for each user, right?

> Now imagine that user10's application starts to experience a
> database issue and the app stops responding..  It used to be true
> that everyone would stop responding because the AJP connectors were
> BIO.

That doesn't make any sense: the proxy configuration is much more
important than whether or not BIO is being used with Tomcat. If one
Tomcat is seized-up (which would happen e.g. if all db connections
were used and all BIO threads were blocking), there should be no
problem with the other workers on the proxy reaching other Tomcat
instances.

> Then the HTTP connections would stack up across the board.  The 
> stacking of the HTTP connections was expected given the situation. 
> Eventually the reverse proxy servers would die from running out of 
> memory if were didn't get the outage under control quickly enough.
Now, /that/ I can understand.

But you said you were using prefork. :/

> Now that we switched that we have had 2 outages.  In both cases the
> only tenants impacted from a performance perspective were the
> tenants experiencing the failures.

Good: if an application fails, it should expect to suffer downtime,
but it shouldn't affect the other applications. Assuming we are
talking about separate Tomcat JVM instances.

> No other alarms were detected during these outages for any other 
> tenants. Something odd does happen however. The Apache HTTP 
> connections rise for everyone along with the offending site.

> Please see the shared graph.
> 
> https://photos.app.goo.gl/ZzEgpQUdbv9L84X82
> 
> This is caclulated by doing a netstat and grepping for EST then
> httpd then the AJP port that would have connections passed back to
> it. ( sudo -tt > /bin/netstat -ntp | grep EST | grep httpd | grep
> ':8125' | wc -l )

So what's :8125? The failing application or one of the good ones?

If httpd is over-provisioned (which is usually okay) and Tomcat is
stalled, then the number of hung connections from httpd -> Tomcat
rising is exactly what I'd expect: the web server can accept more
connections so it's doing that. Tomcat can't make progress so everyone
waits.

> It is the example above that determines the connection counts for
> each tenant.
> 
> I cannot for the life of me understand how or why this is
> happening.. The only rise in connections should be detected in the
> offending application? Right?

Yes. What you've shown me is that a failing application experiences
high connection counts which makes sense. Unless I'm reading your text
above incorrectly. Or the graph?

> I can't say beyond a shadow of a doubt that the AJP connector
> threads aren't being wonky.  I am having trouble getting JMX to
> tell me that information through zabbix.

How are you querying JMX?

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQJRBAEBCAA7FiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlogOnwdHGNocmlzQGNo
cmlzdG9waGVyc2NodWx0ei5uZXQACgkQHPApP6U8pFignBAAjU3tpDu/4YqpMOAy
kPKgD9/U8gWBuxJpDbEbMwErPjv0gFGGwYKz/VE0d6of+DEJtVn/ZQKaaTSHJ+f1
FwHmJLYr+H8GI3Nc9dobDOtgE5fyEekqWByHA6r+4jU3hBWjqYzcT+ihcEhHhs1v
xy3cOaUd/4G/XON9HtoxsQ3wdiOlBT06GNQpYQNJl3/zlYrS+5I4qoYIazQ21dq9
qpNVRuRID5ltIjbq/G95S+y+srMW2xX7O7XUz520gc546C8p+29hmppEoUTjyxch
wVn46UEAkIbWaCOCCAYikUlwLPMPLhBhCE/vldCqjqq1VMtPV5fhhLUxm/kNac26
mvIFZuCrAv2jD+SDpB+trQK696STDYlCNrn3s3mMw+fkWVnIhiI3OZPXDzieD+Bf
MXO26uAMwAnjsWuJ6M8u+OGhufWW6YetVpJZpibJWCOcx5NLDkm+7BReeVEn1Vv4
p5zjXT/rePAlhz3T+7oUCCUN/O+X7BPIcjdeoh8mp+W3t30dYWWQJI4hHt+ATuq1
7J1fUzyjJNE7iP5kvxnG5lA3Kc0clobhwyMH8D5dgqwR32Hm+7IPE2PQbtx4SU8O
pqNTIdi2l2h1TTAjmey1TtP85GZHsl6Rwy0gP8nRso4UtGZp5EPPsUhrRfPSS85D
MD3low32ZefGl1Omb1lDCFSEbDg=
=qCJH
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org