Mailing-List: contact users-help@tomcat.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Tomcat Users List" <users@tomcat.apache.org>
Date: Wed, 04 Oct 2017 15:33:27 +0100
User-Agent: K-9 Mail for Android
In-Reply-To: <26cf79e5-cf4b-e101-fbfe-5c67945fab15@apache.org>
References: <CAJaHpyd4oKXYqiB_rOc1TjRGML55XQbLm+snLuEaUJM3f3YDeQ@mail.gmail.com> <26cf79e5-cf4b-e101-fbfe-5c67945fab15@apache.org>
MIME-Version: 1.0
Content-Type: text/plain;
 charset=utf-8
Content-Transfer-Encoding: quoted-printable
Subject: Re: AJP connection pool issue bug?
To: Tomcat Users List <users@tomcat.apache.org>
From: Mark Thomas <markt@apache.org>
Message-ID: <7A29D23B-BA3F-4319-B5BE-4FA844DF4E78@apache.org>
archived-at: Wed, 04 Oct 2017 14:33:34 -0000

On 4 October 2017 15:17:25 BST, Mark Thomas <markt@apache=2Eorg> wrote:
>On 04/10/17 13:51, TurboChargedDad =2E wrote:
>>  Hello all=2E=2E
>> I am going to do my best to describe my problem=2E  Hopefully someone
>will
>> have some sort of insight=2E
>>=20
>> Tomcat 7=2E0=2E41 (working on updating that)
>> Java 1=2E6 (Working on getting this updated to the latest minor
>release)
>> RHEL Linux
>>=20
>> I inherited an opti-tenant setup=2E  Individual user accounts on the
>system
>> each have their own Tomcat instance, each is started using sysinit=2E=
=20
>This
>> is done to keep each website in its own permissible world so one
>website
>> can't interfere with a others data=2E
>>=20
>> There are two load balanced apache proxies at the edge that point to
>one
>> Tomcat server (I know I know but again I inherited this)
>>=20
>> Apache lays over the top of tomcat to terminate SSL and uses AJP to
>> proxypass to each tomcat instance based on the users assigned port=2E
>>=20
>> Things have run fine for years (so I am being told anyway) until
>recently=2E
>> Let me give an example of an outage=2E
>>=20
>> User1, user2 and user3 all use unique databases on a shared database
>> server, SQL server 10=2E
>>=20
>> User 4 runs on a windows jboss server and also has a database on
>shared
>> database server 10=2E
>>=20
>> Users 5-50 all run in the mentioned Linux server using tomcat and
>have
>> databases on *other* various shared databases servers but have
>nothing to
>> do with database server 10=2E
>>=20
>> User 4 had a stored proc go wild on database server 10 basically
>knocking
>> it offline=2E
>>=20
>>   Now one would expect sites 1-4 to experience interruption of
>service
>> because they use a shared DBMS platform=2E  However=2E
>>=20
>> Every single site goes down=2E I monitor the connections for each site
>with a
>> custom tool=2E  When this happens, the connections start stacking up
>across
>> all the components=2E (Proxies all the way through the stack)
>> Looking at the AJP connection pool threads for user 9 shows that user
>has
>> exhausted their AJP connection pool threads=2E  They are maxed out at
>300 yet
>> that user doesn't have high activity at all=2E The CPU load, memory
>usage and
>> traffic for everything except SQL server 10 is stable during this
>outrage=2E
>> The proxies start consuming more and more memory the longer the
>outrage
>> occurs but that's expected as the connection counts stack up into the
>> thousands=2E  After a short time all the sites apache / ssl termination
>later
>> start throwing AJP timeout errors=2E  Shortly after that the edge
>proxies
>> will naturally also starting throwing timeout errors of their own=2E
>>=20
>> I am only watching user 9 using a tool that allows me to have insight
>into
>> what's going on using JMX metrics but I suspect that once I get all
>the
>> others instrumented that I will see the same thing=2E Maxed out AJP
>> connection pools=2E
>>=20
>> Aren't those supposed to be unique per user/ JVM? Am I missing
>something in
>> the docs?
>>=20
>> Any assistance from the tomcat gods is much appreciated=2E
>
>TL;DR - Try switching to the NIO AJP connector on Tomcat=2E
>
>Take a look at this session I just uploaded from TomcatCon London last
>week=2E You probably want to start around 35:00 and the topic of thread
>exhaustion=2E

Whoops=2E Here is the link=2E

https://youtu=2Ebe/2QYWp1k5QQM

Mark


>
>HTH,
>
>Mark
>
>P=2ES=2E The other sessions we have are on the way=2E I plan to update th=
e
>site and post links once I have them all uploaded=2E
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: users-unsubscribe@tomcat=2Eapache=2Eorg
>For additional commands, e-mail: users-help@tomcat=2Eapache=2Eorg


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org