Mailing-List: contact users-help@tomcat.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Tomcat Users List" <users@tomcat.apache.org>
Subject: Re: Performance settings for Multiple Hosts
To: users@tomcat.apache.org
References: <a9981c8d-4037-b727-6148-ace457f543e0@malcolms.com>
 <CAP-JJ3rzRf4FMmzm82L0pHH-7fD6nf-LvgdgnKXD=AdE2N-rBg@mail.gmail.com>
 <59D757D7.8040709@ice-sa.com>
 <0b9100de-b8be-4eda-0c18-6e62bfdb919d@malcolms.com>
From: =?UTF-8?Q?Andr=c3=a9_Warnier_=28tomcat=29?= <aw@ice-sa.com>
Message-ID: <59D8B0F5.2010409@ice-sa.com>
Date: Sat, 7 Oct 2017 12:48:21 +0200
User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:38.0) Gecko/20100101
 Thunderbird/38.5.0
MIME-Version: 1.0
In-Reply-To: <0b9100de-b8be-4eda-0c18-6e62bfdb919d@malcolms.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
archived-at: Sat, 07 Oct 2017 10:48:37 -0000

On 07.10.2017 00:05, Jerry Malcolm wrote:
> On 10/6/2017 5:15 AM, André Warnier (tomcat) wrote:
>> On 06.10.2017 11:57, Suvendu Sekhar Mondal wrote:
>>> On Fri, Oct 6, 2017 at 2:06 AM, Jerry Malcolm <techstuff@malcolms.com> wrote:
>>>> I am running TC 8.0 on WinServer8 on a commercially hosted platform with a
>>>> WAMP environment.   I am running around 10 virtual hosts.   2 hosts are
>>>> dedicated to JSPWiki.  The other 8 are running variations of the same custom
>>>> application with around 10-15 individual webapps each.
>>>>
>>>> When I am running 7 of the 10 hosts, performance is great.  I get JSP
>>>> response time under a second.  But when I add just a couple of more of the
>>>> hosts, my page response time on all of the apps goes from an acceptable
>>>> under a second to horrible at around 15-20 seconds per page. It doesn't
>>>> seem to be a specific host causing the problem.  Reducing overall hosts in
>>>> any order makes the problem go away.
>>>>
>>>> I've looked at the the processor utilization during the good times and slow
>>>> times, and don't see a significant difference.  I have 16GB of memory, and
>>>> it consistently shows about 35% utilization.  I also checked mySQL response
>>>> time, and the per-query SQL response does not vary.  So it doesn't appear to
>>>> be a db problem.
>>>>
>>>> I suspect there is some TC configuration parameter such as heap, etc that I
>>>> need to tweak. (But I'm not getting OutOfMemory errors).  But I don't know
>>>> which one, and I don't know a formula to use to figure out what to set it
>>>> to.  So I just need a little education.  What tools can I use to help me
>>>> figure out what is going south slowing everything to a crawl when the extra
>>>> hosts come online?  And what parameters should I be looking at (and how
>>>> should I calculate the proper values based on number of hosts)?
>>>>
>>>> Suggestions?
>>>>
>>> Interesting problem. Couple of questions:
>>> Is the slowness sporadic or persistent? What happens when you again
>>> shutdown some of the hosts? Does response time comes back to normal?
>>> Is all transactions are slow or some of them which are routed to
>>> specific host(s)?
>>> Are you fronting Tomcat instances with Apache? If yes, then please
>>> post BalancerMember configuration of the Tomcat cluster here.
>>>
>>
>> Taking the OP's observations above at face value, I think that first thing that I would
>> do, is to configure the Java JVM that runs this tomcat, to do some logging of the
>> Garbage Collection events. And then I would look at that logfile, and find out if there
>> is some significant difference (in frequency of GC, or in the mounts of memory mentioned
>> for each event), depending on the number of Hosts enabled.
>>
>> You would have to look at the appropriate command-line switches to use for the
>> particular JVM being used, for how to set this up.
>> And then (since this seems to be on a Windows platform, running tomcat as a service),
>> you should use the "tomcat(version)w.exe" program (a GUI), to set these parameters
>> properly.
>> See e.g.
>> https://tomcat.apache.org/tomcat-8.5-doc/windows-service-howto.html#Tomcat_service_application
>>
>>
>> --> "Tomcat monitor application"
>>
> Thanks for the responses. Answers to several questions in responses:
>
> 1) The 16GB is physical OS memory, not JVM memory

Ok. And how much /is/ the JVM memory ? (size of the Heap e.g.).
(for this, you may need to examine the command-line switches used for this JVM, using the 
above-mentioned tomcat8w.exe.)

> 2) Re: 'balancermember config'.... I am running one instance of Apache with mod_jk to one
> one instance of Tomcat containing all of the 10 virtual hosts.  I'm not familiar with
> 'balancermember'.  But it sounds like something to do with a cluster config.  If it still
> applies to my config, let me know where I can learn more about it.
> 3) I'll look into turning on garbage collection logging
> 4) The problem is constant when I have 'too many' hosts.  And the problem is pervasive to
> all running hosts. I've found the magic number is 7 hosts.  7 hosts and life is good.  8+
> hosts  and everything clogs up big-time.
>
> I failed to mention that none of these host sites are anywhere close to 'high-volume'.
> When a user hits the site, there will be flurry of page activity for a few minutes.  But
> I'd say each of the sites averages one or two user hits per hour on average.  So bringing
> sites 8, 9, and 10 online is not bringing massive additional tomcat/mySQL/network
> activity.  It's basically bringing 3 more 'effectively dormant' web sites online.  This
> seems to me that it's got to have something to with the size of the basic inactive
> footprint of each host.
>
Yes, that's what your observations would tend to suggest. And that's why I was 
recommending to set up, and have a look at, the GC logs.
Apart from showing how often GC runs happen (with 7 hosts, and beyond), they will also 
show important parameters about the allocated memory.
You have not indicated so far which JVM you are using, nor its startup parameters.
It could be that some exotic memory-related parameter there introduces a constraint which 
hits precisely at the 7-8 Host boundary in your case.
(There are better and more precise ways to find out about memory usage, but they are also 
a bit more complicated to set up; that's why I am recommending the GC logging first).

To look at this in another way : I am quite sure that there are a lot of people out there 
running tomcat's with a lot more than 7 Hosts (and a lot more traffic than what you say 
you have). If there was something within the tomcat code itself which does not scale 
properly beyond 7 Hosts or so, this support list would be swamped with reports about it. 
It isn't. Ergo, it is unlikely; and thus, it has to be the influence of something outside 
tomcat itself. And JVM memory settings and usage would be a convenient first-line suspect.


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org