Mailing-List: contact users-help@tomcat.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Tomcat Users List" <users@tomcat.apache.org>
Subject: Re: Understanding tomcat + apache and I/O
To: users@tomcat.apache.org
References: <CAJaHpydDUeQ5evP2gp=woNwRkhAJjD7C2E0tXT=32oWZ2Wsf=A@mail.gmail.com>
From: Christopher Schultz <chris@christopherschultz.net>
Message-ID: <35790bfb-7421-ee1d-0da3-5174bb75b8a8@christopherschultz.net>
Date: Wed, 29 Nov 2017 09:51:41 -0500
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:52.0)
 Gecko/20100101 Thunderbird/52.4.0
MIME-Version: 1.0
In-Reply-To: <CAJaHpydDUeQ5evP2gp=woNwRkhAJjD7C2E0tXT=32oWZ2Wsf=A@mail.gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 8bit
archived-at: Wed, 29 Nov 2017 14:51:50 -0000

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Big Papa,

On 11/29/17 12:06 AM, TurboChargedDad . wrote:
> So.. Thank you for those help me understand the NIO vs BIO in
> tomcat 7..

So now all you have to do is upgrade to Tomcat 8.0 or, even better,
Tomcat 8.5 :)

> I made those changes things have improved quite a bit.  I am still 
> experiencing some weirdness that I have tried to understand but
> can't get a handle on it.

Can you expand on the "weirdness"? I see you have some more details
below but I think you could be more specific.

> Quick overview.. --Proxies-- Apache Proxies (2) - The end user
> terminates SSL at the proxy/edge The proxies use HTTPS/SSL to
> reverse proxy back to the tomcat server. --/Proxies--
> 
> PXY1 & 2 configs for prefork mode. <IfModule prefork.c> 
> StartServers 30 MinSpareServers 15 MaxSpareServers 30 ServerLimit
> 400 MaxClients 400 MaxRequestsPerChild 4000 </IfModule>

If you want high performance, you have to abandon the prefork model
and move to event. Some modules (e.g. mod_php IIRC) don't work
properly with the event model. Think about using your lb with PHP
running on another server as Jim Riggs suggests[1]. You may get better
performance, stability, and fault-tolerance.

> --Tomcat server-- (1) Apache terminates SSL over the top of Tomcat
> on the same server. Reverse proxies to the tomcat server using NIO
> AJP connectors. --/Tomcat server--

Above you say that you are using HTTPS/SSL to connect httpd -> Tomcat.
If you are using AJP then this is not true. So which is it? Are you
using HTTP or AJP as your protocol?

> Tomcat apache prefork mode config: <IfModule prefork.c> 
> StartServers       8 MinSpareServers    5 MaxSpareServers   20 
> ServerLimit      800 MaxClients       800 MaxRequestsPerChild
> 4000 </IfModule>

What does "Tomcat apache prefork mode" mean? The above is an httpd
configuration, not a Tomcat one.

> Typical vhost config for a given tenant would look like this.. 
> <someuser.conf> <VirtualHost 10.10.10.26:443
> <http://10.10.10.26:443/>> ServerAdmin admin@company.com
> <mailto:admin@company.com> ServerName somewhere.somedomain.com
> <http://somewhere.somedomain.com/> ProxyPass /
> ajp://localhost:8126/ retry=3

Okay, now you are using AJP. I think there's definitely some confusion
here as to what is being configured with what.

> Typical tomcat connector thread config : <Connector port="8126" 
> protocol="org.apache.coyote.ajp.AjpNioProtocol"
> redirectPort="8443" maxThreads="300" />

If this is the only <Connector> in Tomcat, then you are 100% using AJP
and not HTTP as your protocol.

Using NIO is the best practice here IMO.

> We are operating a multi-tenant environment.  As of right now, we
> have somewhere around 20 tomcat instances on a large machine of
> which only a handful are "busy".

Good.

> It used to be that when any one of them experienced a blocking
> issue. Every one of them went down.  All of their AJP connector
> threads would rise until the system because tomcat was
> unresponsive.

That would be a capacity-planning problem with the httpd proxies. You
probably didn't do your math correctly.

> So far that appears for the most part to be addressed...
Good. Maybe your math is better but it may still be wrong.

> However...  When an issue is experienced.  The site(s)
> experiencing the issue(s) going down doesn't seem to bring down any
> of the other sites. (w00t! w00t!)

Good.

> But the httpd connections for each site all still climb together.

That shouldn't happen (of course!).

> (Please see attached graph) Again no outage is experienced buy as 
> demonstrated by the graph attached to this message.

Attachments are stripped. Either post your graph elsewhere or describe
it in words.

> That graph is from zabbix using a custom metric that checks every
> 3 mins..  It does the following for each virtual host / tomcat
> instances
> 
> For user25 : UserParameter=somewebsite.constats,sudo -tt
> /bin/netstat -ntp | grep EST | grep httpd | grep ':8125' | wc -l 
> UserParameter=somewebsite2.constats,sudo -tt /bin/netstat -ntp |
> grep EST | grep httpd | grep ':8126' | wc -l
> 
> So there is virtually no way they can be getting mixed up.  Not to 
> mention that there are a few that do not experience a rise in
> connections.

So the "Weirdness" is that your AJP connection count on the httpd
proxy instances increases across all web servers (or all workers?).
What does mod_proxy's status page say for *each worker*? THAT'S what
you need to compare, not just the total number of connections/threads
on the proxy.

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQJRBAEBCAA7FiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAloeyX0dHGNocmlzQGNo
cmlzdG9waGVyc2NodWx0ei5uZXQACgkQHPApP6U8pFgAZQ/+OyyDIEaWzgF5zG1o
amUGjCUackktehlpW9STa5kRhIj9REYT4Cql64Cwqvw8ciZVQXAOsYJBACXFKcfa
fvegRQ03YeLy9LDXhPtsx4Nr+qT17ySiFo/MckEIkxCR9mBbFokUb1bVes9kkYQu
yJjQ7AV8SWDWKGdAkbRk4WTuJ23bvRwZ2g4MNb0sDg5dJEQIOY7JYhlFJQLPm/1a
Yeeo/xRMLfY4FBI0zpA1DAXEwiLyXup4SOztHnoxbK5h0YgrRGMOKvAwZXs5/u/2
NbiqCnsA80OzUrSXd5sDBYzsuR2yOfnnUMcUJh6LmL8XG1Fh+QeJdCDM2KCC0hJL
HSyqEVfl9EyIXj/Bu0DzU6lL8z5lhxoEYeYBE+pMfJKZ3Skb3EHLImuI/Fwv/+hk
T13o1irPIiqq0N0pTvjLtbJbapNZ9Nz8aIzTBLR3TRY6Cf79jI49QRHBYMsth7e/
2Ub+WmkbphcoaC1oJEpXu8fbcSuDrdTAzZ9VSvUFsqpUw5b/9OSB2DjKx5GtnHzR
f8bEg682xw8VKlOV7EsD4RvyILmFqisrrTAfE8/3F3IGMYqXV8Is+8BtdScvUgT5
MzaLVVyT650pv1NJxBjz/UF8BdhJ2Rpj0r5jhqB64Q1F1ZuNO6aLI2qEwRNgwYMV
IoMgNCcqcB03pYbpuYxHKoX5Cfc=
=imbH
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org