cocoon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric E. Meyer" <eric.meyer.coc...@quoininc.com>
Subject Cocoon Performance Woes, Is it flow? I don't know!
Date Thu, 17 Mar 2005 17:30:56 GMT
Cocoon Performance Woes, Is it flow? I don't know!

Hello all,

My team developed and deployed a web application for a client which is 
built on top of Cocoon (www.fivestaralliance.com).

I have spent over two weeks now attempting to improve the
performance and scalability of the application – with some real
improvements. However, I continue to feel like I'm flying blind –
because the apparent bottlenecks are somewhere outside of my code. I
have read a number of previous debates about javascript flow being slow,
and we are using CForms with Javascript flow, but my main concern is
that I cannot determine where the bottlenecks are.

Even when the application is bogging down, the components that my team
wrote are performing their tasks (generation, transformation,
flowscripting) at very fast rates (as seen by tracing/logging), but
something else in the framework/process is bogging down the requests.
The analysis tools don't show any obvious resource limitation even at
the highest loading levels.

Some pages in the site scale quite well, while one in particular does
not scale well at all. While I know that I'm close to having a system
that runs blindingly fast, I'm currently faced with a situation where I
cannot effectively argue that the architecture isn't "fundamentally
flawed" and I'm unable to address a major scalability concern for my
client. I would welcome any concrete suggestions on how to better
determine my bottlenecks and any additional tuning advice.

Brief description of the application architecture

CForms, Javascript flow, mix of JX template generators, XInclude
transformer and custom generator transformers. Core application
components implemented in Java. Hibernate persistence.

Profiling and Monitoring

My biggest problem is that I've only been able to determine where the
problem isn't at this point. I've used a variety of tools to attempt to
see what's going on, and why the application is bogging down, but I
cannot seem to get a comprehensive picture of what delays/bottlenecks
there are within Cocoon.

Specifically, it would be extremely helpful to monitor the number of
generators/transformers/other pooled components in use, allocated, freed
while under load. Additionally, it would be useful to see the time taken
up by each of the steps in the process of servicing a request – not just
the set-up and generation/transform times as shown by the Cocoon profiler.

These are the tools/approaches that I have used:

Multi-thread load test with JMeter.
Profiling of application code using JProbe (CPU and memory analysis).
Profiling of Cocoon components using Cocoon pipeline profiler.
Monitoring of Cocoon components using the Cocoon instrumentation client.
Monitoring of Cocoon server using Status generator.
Monitored Linux system activity with SAR, iostat, mpstat and vmstat
during load-testing.
Profiling of custom generators, transformers and flowscript using
Jakarata Commons StopWatch and log statements.

Tweaks made thus far

Adjusted Java virtual machine parameters
	-server -Xms512M -Xmx512M
Adjust logging levels - turned down logging
Adjusted thread pool sizes in Tomcat
	150 -> 350 max
Adjusted database connection pool size up to 50
Adjusted sitemap component pool sizes up
Optimized some Java code based upon JProbe profiling
Added additional objects to the in-memory cache to reduce database queries
Turned off reloading of sitemaps and javascript files
Replaced default Cocoon JCS cache with Whirlycache
Replaced default Cocoon Xalan XSL transformer with faster Saxon XSL
transformer
Configured Cocoon to reuse XML parsers
Removed Cocoon store janitor
Preloading key OO javascript flowscript at server startup

Observations

Windows XP
Pentium 4 1.8Ghz
JDK 1.4.2_06
Tomcat 5.0.28
Cocoon 2.1.5.1
512MB physical RAM
JVM -server -Xms256M -Xmx256M

Load test with num users threads each making 5 successive request in a
loop with approximately 3 second think time between requests. No derived
resources – only the main page.

users overall home  search1 search2 search3  detail  total
       avg ms  page                                   num reqs

10      486    208      637    588     644     355    500
20     1704    378     2684   1837    2875     745    500
30     3725    682     5987   4626    6270    1059    450
40    19461   1411    36021  23089   34726    2059    600
50    72942   3213   130482  90993  131666    8356    500

home page: /
search1: /luxury_hotels/europe__france__paris/index.html
search2: /luxury_hotels/bahamas_%26_the_caribbean/beach_resort/index.html
search3:
/luxury_hotels/europe__france__paris/city_centre_location/index.html
detail: /luxury_hotel/new_york,_ny/the_carlyle

Platform:

Development
Windows XP
JDK 1.4.2_06
Tomcat 5.0.28
Cocoon 2.1.5.1

Deployment
Linux 2.6.x

We see similar degradation on Linux as on Windows.

The home page has no flowscript or cforms, but does have jxtemplate
generation, xinclude, xslt, and a custom generator.

The search and detail pages include a cform, and are therefore driven
with flowscript at the top-level matching (and create continuations in
the process of displaying their forms). These pages use jxtemplate
generation, xinclude, xslt, custom generation, custom transformation,
and internal-only sub pipelines. When looking at the pipeline times with
a profiling pipeline, the total times (while under load) are much higher
that the displayed times for the setup and generation steps -- so where
is the time going?

Regards,
Eric Meyer





Mime
View raw message