Mailing-List: contact users-help@tomcat.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Tomcat Users List" <users@tomcat.apache.org>
Received-SPF: pass (nike.apache.org: local policy)
From: Talha Fazal <tfazal@credera.com>
To: Tomcat Users List <users@tomcat.apache.org>, R Batchelor
	<rsbatch8@yahoo.com>
Subject: RE: Tomcat Performance Turning.
Thread-Topic: Tomcat Performance Turning.
Thread-Index: 
 AcxpdKifMHRjGX6tTmGfCAchYIxF+AAKm2QAAApG/UD//7ZtgIAAUqFQ//+7CYCAAAQrAIAARHmA
Date: Fri, 2 Sep 2011 16:16:11 +0000
Message-ID: <FBE4B43BAA728B4990A091267429FD081A58DB60@exchsrv.credera.com>
References: <FBE4B43BAA728B4990A091267429FD081A58D9DF@exchsrv.credera.com>
 <CAE2_di0LjCvFDmtih38-yAQ3npSdDBPqaY8hUjZyz-iSV0ZVZg@mail.gmail.com>
 <FBE4B43BAA728B4990A091267429FD081A58DA08@exchsrv.credera.com>
 <CAE2_di2uw_Cy=x7GcaR2MzGQH4U2=HdJ60J2SgZc+2TLmh0mbQ@mail.gmail.com>
 <FBE4B43BAA728B4990A091267429FD081A58DAB7@exchsrv.credera.com>
 <CAE2_di3G00kDA1v53QVg2xKd_TDfXCGuvtyP2YFZRyk72PXvYw@mail.gmail.com>
 <1314976214.66810.YahooMailNeo@web162017.mail.bf1.yahoo.com>
In-Reply-To: <1314976214.66810.YahooMailNeo@web162017.mail.bf1.yahoo.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

PLEASE SEE MY ANSWERS BELOW IN UPPER CAPS.

ONE IMPORTANT OBSERVATION: Even though, tomcat's response time is decent, f=
or some reason, apache is taking a lot longer to serve requests. We are usi=
ng mod_jk with Tomcat using AJP1.3 protocol in the connector setting in ser=
ver.xml. We plan to try mod_proxy_ajp and mod_proxy instead of mod_jk. Any =
thoughts here?

Thanks,


-----Original Message-----
From: R Batchelor [mailto:rsbatch8@yahoo.com]=20
Sent: Friday, September 02, 2011 10:10 AM
To: Tomcat Users List
Subject: Re: Tomcat Performance Turning.

Have you ruled out issues with db connection pooling? NO. IN FACT, THERE AR=
E KNOWN DBCP CONNECTION POOLING DEADLOCK ISSUES IN TOMCAT 6.0.26 [https://i=
ssues.apache.org/jira/browse/DBCP-270]. I AM LOBBYING TO UPGRADE TO TOMCAT =
7.0.20 AND USE THE JDBC CONNECTION POOL INSTEAD OF DBCP CONNECTION POOL.

You might consider setting your dbcp maxWait to 8000 or less and watch=20
for "timeout waiting for idle object" exceptions.=A0 Also, you could=20
monitor database connections/active-users on the DB side to see if your=20
dbcp pools are max'd but all the database users/connections are idle.=A0=20
These can be symptoms of an application mis-managing its dbcp pools.
I WILL DEFINITELY TRY THIS OUT. THANKS!

________________________________
From: Leon Rosenberg <rosenberg.leon@gmail.com>
To: Tomcat Users List <users@tomcat.apache.org>
Sent: Friday, September 2, 2011 7:55 AM
Subject: Re: Tomcat Performance Turning.

Hello Talha,

well... the quickshots are through.
You should really create some threaddumps after each other (for
example with jstack) and try to find out which thread is slowing the
app down.
For starters you could try with code you changed for tomcat
adaptations if any. The problem seems to lie beneath the surface, so
you will have to start
performance monitoring and look into the inside.
However, time difference of 17 seconds must be something very very
obvious like a synchronized block in a valve/filter, db lock (have you
checked the db locks?), or an if (tomcat)
Thread.sleep(1000L*60*17);....

regards
Leon

On Fri, Sep 2, 2011 at 4:21 PM, Talha Fazal <tfazal@credera.com> wrote:
> Hi Leon,
>
> Please see any answers in CAPS below.
>
>
>
> -----Original Message-----
> From: Leon Rosenberg [mailto:rosenberg.leon@gmail.com]
> Sent: Friday, September 02, 2011 9:06 AM
> To: Tomcat Users List
> Subject: Re: Tomcat Performance Turning.
>
> Hello Talha,
>
> seems that your tomcats are fine.
> Question, you mention that the tomcat do not perform, but you don't
> tell us how you come to this conclusion except for cpu load. ANSWER: LOOK=
ING INTO THE TOMCAT ACCESS LOGS, THE RESPONSE TIME (%D) IS HORRIBLE).
> What is the difference in response times between weblogic and tomcat? ANS=
WER: FOR PAGES THAT WEBLOGIC SERVES IN 500 ms ON AVERAGE, TOMCAT IS TAKING =
17000 ms ON AVERAGE.
> Maybe your tomcat just perform the job MUCH faster ;-)
> The thread dump you posted in another reply indicates that your
> tomcats are idling.
> Do you have manager installed? NO. With manager you could monitor active
> connections and requests.
> Do you have probe installed? YES, WE HAVE IT INSTALLED. A LOT OF THREADS =
ARE IDLING IS WHAT WE OBSERVE.
> Finally, if you really want to profile your application, do you have
> moskito installed? :-) (the one i initially wrote
> -moskito.anotheria.net, therefore the shameless advertisement). :-) SURE =
WILL GIVE IT A TRY [TIME PERMITTING].
>
> In the last case I can offer you to provide support getting it running. W=
ILL LET YOU KNOW IF WE ARE INTERESTED. THANKS.
>
> regards
> Leon
>
> On Fri, Sep 2, 2011 at 3:50 PM, Talha Fazal <tfazal@credera.com> wrote:
>> Plz. see my answers below in UPPERCAPS.
>>
>> Thanks!
>>
>> -----Original Message-----
>> From: Leon Rosenberg [mailto:rosenberg.leon@gmail.com]
>> Sent: Friday, September 02, 2011 8:35 AM
>> To: Tomcat Users List
>> Subject: Re: Tomcat Performance Turning.
>>
>> Hello Talha,
>>
>> from a quick glance at your post, do you have the same 30 max threads
>> limit in weblogic? YES.
>> Because sending 525 users through 2x30=3D60 max threads seems a little
>> bottlenecky. I FORGOT TO MENTION THAT ON EACH TOMCAT SERVER, WE ARE RUNN=
ING 3 INSTANCES OF TOMCAT ON EACH OF OUR TOMCAT SERVERS (2 SERVERS), THUS 2=
*3*30=3D180 maxThreads. WE EVEN INCREASED THE maxThreads TO 120 FOR EACH IN=
STANCE (thus 120*6=3D720 maxThreads), AND MATCHED maxActive IN JNDI RESOURC=
E TO 120, BUT WE STILL DIDN'T SEE CONSIDERABLE IMPROVEMENT. THE CPU USAGE I=
NCREASED FROM 7% to 10%.
>>
>> Which software are you using to produce the load? LOADRUNNER. Does it ke=
epalive the connections? YES.
>>
>> regards
>> Leon
>>
>> On Fri, Sep 2, 2011 at 3:31 PM, Talha Fazal <tfazal@credera.com> wrote:
>>> Background:
>>>
>>> We have a moderately high traffic web application (between 8 to 21 mill=
ion hits/day) running Apache to serve static content (also to load balance =
and create a DMZ) and Weblogic to serve dynamic content (Struts 1.1 based J=
ava web application).
>>> We are trying to replace Weblogic with Tomcat and we have ported our co=
de to work with Tomcat. All works well in Tomcat in the DEV, QA, and STAGIN=
G environment as long as there is no real load.
>>>
>>> The Issue - Load Testing:
>>>
>>> In our staging environment for load testing, when we run the load test =
using 525 concurrent users, the app doesn't perform at all. The CPU usage (=
on Apache and Tomcat Servers) hovers between 7% to 8%. The database server =
CPU usage is also between 4 and 5%.
>>>
>>> Setup for Load Testing:
>>>
>>> We have setup 2 apache web servers (4 Quad Processors i.e. 16 CPUs each=
), 2 Tomcat (version 6.0.29) servers (4 Quad Processors i.e. 16 CPUs each).=
 Each server has 32 Gb ram. We are using AJP 1.3 to connect Tomcat and Apac=
he.
>>>
>>> Mentioned below is the version information:
>>> Apache Version 2.2.14 (with mod_jk module)
>>> Tomcat: 6.0.29
>>> Database: Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 6=
4bits
>>> Connection Pool: DBCP.
>>>
>>> Mentioned below are connector settings in conf/server.xml:
>>>
>>> =A0<Connector
>>> =A0 =A0 =A0 =A0address=3D"stagingTCserver01"
>>> =A0 =A0 =A0 =A0backlog=3D"300"
>>> =A0 =A0 =A0 =A0connectionTimeout=3D"60000"
>>> =A0 =A0 =A0 =A0enableLookups=3D"false"
>>> =A0 =A0 =A0 =A0maxPostSize=3D"2097152"
>>> =A0 =A0 =A0 =A0maxSpareThreads=3D"10"
>>> =A0 =A0 =A0 =A0maxThreads=3D"30"
>>> =A0 =A0 =A0 =A0minSpareThreads=3D"5"
>>> =A0 =A0 =A0 =A0port=3D"8006"
>>> =A0 =A0 =A0 =A0protocol=3D"AJP/1.3"
>>> =A0 =A0 =A0 =A0tcpNoDelay=3D"true"
>>> =A0 =A0/>
>>>
>>> Mentioned below are the settings for JNDI resource configured in conf/c=
ontext.xml:
>>>
>>> =A0<Resource
>>> =A0 =A0 =A0 =A0name=3D"jdbc/onlinedb"
>>> =A0 =A0 =A0 =A0auth=3D"Container"
>>> =A0 =A0 =A0 =A0type=3D"javax.sql.DataSource"
>>> =A0 =A0 =A0 =A0driverClassName=3D"oracle.jdbc.OracleDriver"
>>> =A0 =A0 =A0 =A0url=3D"jdbc:oracle:thin:@192.168.12.10:1521:WEBDB"
>>> =A0 =A0 =A0 =A0initialSize=3D"1"
>>> =A0 =A0 =A0 =A0maxActive=3D"30"
>>> =A0 =A0 =A0 =A0minIdle=3D"1"
>>> =A0 =A0 =A0 =A0maxIdle=3D"5"
>>> =A0 =A0 =A0 =A0maxWait=3D"300000"
>>> =A0 =A0 =A0 =A0poolPreparedStatements=3D"true"
>>> =A0 =A0 =A0 =A0maxOpenPreparedStatements=3D"300"
>>> =A0 =A0 =A0 =A0validationQuery=3D"SELECT 1 FROM BB_DUAL"
>>> =A0 =A0 =A0 =A0testOnBorrow=3D"true"
>>> =A0 =A0 =A0 =A0validationInterval=3D"10000"
>>> =A0 =A0 =A0 =A0testWhileIdle=3D"true"
>>> =A0 =A0/>
>>>
>>> JVM Parameters:
>>>
>>> -Xms512m -Xmx2048m
>>> -XX:PermSize=3D128m -XX:MaxPermSize=3D256m
>>> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=3D/logs/WL2TC/app/
>>> -verbose:gc -Xloggc:/logs/WL2TC/app/WL2TC_1-gc.log
>>>
>>> It must be noted that Weblogic setup performs very well using similar s=
ettings.
>>>
>>> Garbage Collection: While the load test is running, Garbage collection =
works just fine i.e. Young GC occurring every 2-3 minutes and takes less th=
an half a second. Full GC occurs every hour and takes a little over 2 secon=
ds.
>>>
>>> Any tips/pointers will be greatly appreciated.
>>>
>>> Talha.
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org