cocoon-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Christofer.Dutz@c-ware.de" <christofer.d...@c-ware.de>
Subject Cocoon 2.1.10 Application Freezing
Date Wed, 02 Jul 2008 09:50:24 GMT
Hi,

 

I am sort of going to get a heart attack in the next hours, if I can’t
resolve this quickly.

 

We recently deployed a new Version of a intranet-application of a large
logistic company here in Germany. It is completely written in cocoon and
performs extraordinary nicely. Until at one point the System starts
thrashing. CPU Usage climbs from 12% to 80% and no request are served
anymore.

 

Here a little more details on the Setup:

-          Windows 2003 Server 

-          4 GB Ram … 1,5 GB reserved for the appserver

-          Bea Weblogic 9.2

-          JRockit 1.5 VM

-          MySQL 5.1 DB Backend

 

During normal operation a thread dump is about 30k big … during thrashing
this increases to 1,5MB.

 

Yesterday we had similar Problems which were unfortunately related to too
much logging (100 Threads blocked because they were locked from the Cocoon
Log system). After removing a lot of the Log Output the system performed
nicely even under heavier load. Until the whole thing blew up again. This
time the Thread Dump showed a lot of Threads blocked at:

 

"[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default
(self-tuning)'" id=35 idx=0x80 tid=5092 prio=5 alive, in native, blocked,
daemon

    -- Blocked trying to get lock: java/lang/Object@0x011A8D38[unlocked]

    at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)

    at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1630)[optimized]

    at jrockit/vm/Locks.lockFat(Locks.java:1731)[optimized]

    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1277)[optimized]

    at
org/apache/avalon/excalibur/pool/InstrumentedResourceLimitingPool.put(Instru
mentedResourceLimitingPool.java:411)[inlined]

    at
org/apache/avalon/excalibur/component/PoolableComponentHandler.doPut(Poolabl
eComponentHandler.java:212)[optimized]

    at
org/apache/avalon/excalibur/component/ComponentHandler.put(ComponentHandler.
java:425)[inlined]

    at
org/apache/avalon/excalibur/component/ExcaliburComponentSelector.release(Exc
aliburComponentSelector.java:307)[inlined]

    at
org/apache/cocoon/components/ExtendedComponentSelector.release(ExtendedCompo
nentSelector.java:301)[optimized]

    at
org/apache/cocoon/components/pipeline/AbstractProcessingPipeline.recycle(Abs
tractProcessingPipeline.java:743)[optimized]

    at
org/apache/avalon/excalibur/pool/InstrumentedResourceLimitingPool.put(Instru
mentedResourceLimitingPool.java:407)[inlined]

    at
org/apache/avalon/excalibur/component/PoolableComponentHandler.doPut(Poolabl
eComponentHandler.java:212)[optimized]

    at
org/apache/avalon/excalibur/component/ComponentHandler.put(ComponentHandler.
java:425)[inlined]

    at
org/apache/avalon/excalibur/component/ExcaliburComponentSelector.release(Exc
aliburComponentSelector.java:307)[inlined]

    at
org/apache/cocoon/components/ExtendedComponentSelector.release(ExtendedCompo
nentSelector.java:301)[optimized]

    at
org/apache/cocoon/components/treeprocessor/InvokeContext.dispose(InvokeConte
xt.java:283)

    at
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(Con
creteTreeProcessor.java:179)[inlined]

    at
org/apache/cocoon/components/treeprocessor/TreeProcessor.process(TreeProcess
or.java:253)[inlined]

    at
org/apache/cocoon/components/treeprocessor/sitemap/MountNode.invoke(MountNod
e.java:118)[optimized]

    at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:47)[inlined]

    at
org/apache/cocoon/components/treeprocessor/sitemap/MatchNode.invoke(MatchNod
e.java:108)[optimized]

    at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:69)[inlined]

    at
org/apache/cocoon/components/treeprocessor/sitemap/PipelineNode.invoke(Pipel
ineNode.java:143)[optimized]

    at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:69)[inlined]

    at
org/apache/cocoon/components/treeprocessor/sitemap/PipelinesNode.invoke(Pipe
linesNode.java:93)[optimized]

    at
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(Con
creteTreeProcessor.java:235)[optimized]

    at
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(Con
creteTreeProcessor.java:177)[inlined]

    at
org/apache/cocoon/components/treeprocessor/TreeProcessor.process(TreeProcess
or.java:253)[inlined]

    at
org/apache/cocoon/components/treeprocessor/sitemap/MountNode.invoke(MountNod
e.java:118)[optimized]

    at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:47)[inlined]

    at
org/apache/cocoon/components/treeprocessor/sitemap/ActTypeNode.invoke(ActTyp
eNode.java:139)[optimized]

    at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:47)[inlined]

    at
org/apache/cocoon/components/treeprocessor/sitemap/MatchNode.invoke(MatchNod
e.java:108)[optimized]

    at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:69)[inlined]

    at
org/apache/cocoon/components/treeprocessor/sitemap/PipelineNode.invoke(Pipel
ineNode.java:143)[optimized]

    at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:69)[inlined]

    at
org/apache/cocoon/components/treeprocessor/sitemap/PipelinesNode.invoke(Pipe
linesNode.java:93)[optimized]

    at
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(Con
creteTreeProcessor.java:235)[optimized]

    at
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(Con
creteTreeProcessor.java:177)[inlined]

    at
org/apache/cocoon/components/treeprocessor/TreeProcessor.process(TreeProcess
or.java:253)[optimized]

    at org/apache/cocoon/Cocoon.process(Cocoon.java:699)[optimized]

    at
org/apache/cocoon/servlet/CocoonServlet.service(CocoonServlet.java:1154)[opt
imized]

    at
javax/servlet/http/HttpServlet.service(HttpServlet.java:856)[optimized]

    at
weblogic/servlet/internal/StubSecurityHelper$ServletServiceAction.run(StubSe
curityHelper.java:227)[optimized]

    at
weblogic/servlet/internal/StubSecurityHelper.invokeServlet(StubSecurityHelpe
r.java:125)[inlined]

    at
weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:283)[
optimized]

    at
weblogic/servlet/internal/TailFilter.doFilter(TailFilter.java:26)[optimized]

    at
weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:42)[
optimized]

    at
de/transa/disco/filter/SetCharacterEncodingFilter.doFilter(SetCharacterEncod
ingFilter.java:104)[optimized]

    at
weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:42)[
optimized]

    at
de/transa/disco/database/hibernate/CloseHibernateSessionFilter.doFilter(Clos
eHibernateSessionFilter.java:25)[optimized]

    at
weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:42)[
inlined]

    at
weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run(W
ebAppServletContext.java:3242)[optimized]

    at
weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubjec
t.java:321)[optimized]

    at
weblogic/security/service/SecurityManager.runAs(SecurityManager.java:121)[in
lined]

    at
weblogic/servlet/internal/WebAppServletContext.securedExecute(WebAppServletC
ontext.java:2010)[inlined]

    at
weblogic/servlet/internal/WebAppServletContext.execute(WebAppServletContext.
java:1916)[optimized]

    at
weblogic/servlet/internal/ServletRequestImpl.run(ServletRequestImpl.java:136
6)[optimized]

    at
weblogic/work/ExecuteThread.execute(ExecuteThread.java:209)[optimized]

    at weblogic/work/ExecuteThread.run(ExecuteThread.java:181)

    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)

    -- end of trace

 

As far as I understand the thing, the problem are pool limitations to some
of the sitemap components. In general this should not be a problem.
Unfortunately my application uses JavaFlow and a lot of Flows have to send
Data to other Systems. They all use internal pipelines to generate this
output. My guess now is, that if a Flow is executed to produce output, it
blocks some resources. Maybe some resources it needs itself to finish the
request, causing a deadlock. This would be the case in the following
Thread-Dump:

 

"[ACTIVE] ExecuteThread: '4' for queue: 'weblogic.kernel.Default
(self-tuning)'" id=36 idx=0x84 tid=5100 prio=5 alive, in native,
native_blocked, daemon

    at
jrockit/vm/Allocator.allocLargeObjectOrArray(IIIZ)Ljava/lang/Object;(Native
Method)

    at
jrockit/vm/Allocator.allocLargerThanFreeTLA(Allocator.java:831)[inlined]

    at jrockit/vm/Allocator.allocSlowCaseInner(Allocator.java:947)[inlined]

    at jrockit/vm/Allocator.allocSlowCase(Allocator.java:793)[optimized]

    at
org/apache/cocoon/util/BufferedOutputStream.<init>(BufferedOutputStream.java
:63)[optimized]

    at
org/apache/cocoon/util/BufferedOutputStream.<init>(BufferedOutputStream.java
:46)

    at
org/apache/cocoon/environment/AbstractEnvironment.getOutputStream(AbstractEn
vironment.java:490)

    at
org/apache/cocoon/environment/http/HttpEnvironment.getOutputStream(HttpEnvir
onment.java:263)

    at
org/apache/cocoon/environment/wrapper/EnvironmentWrapper.getOutputStream(Env
ironmentWrapper.java:311)

    at
org/apache/cocoon/components/pipeline/AbstractProcessingPipeline.processXMLP
ipeline(AbstractProcessingPipeline.java:577)[optimized]

    at
org/apache/cocoon/components/pipeline/AbstractProcessingPipeline.process(Abs
tractProcessingPipeline.java:481)[optimized]

    at
org/apache/cocoon/components/treeprocessor/sitemap/SerializeNode.invoke(Seri
alizeNode.java:121)[optimized]

    at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:47)[inlined]

    at
org/apache/cocoon/components/treeprocessor/sitemap/MatchNode.invoke(MatchNod
e.java:108)[optimized]

    at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:69)[inlined]

    at
org/apache/cocoon/components/treeprocessor/sitemap/PipelineNode.invoke(Pipel
ineNode.java:143)[optimized]

    at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:69)[inlined]

    at
org/apache/cocoon/components/treeprocessor/sitemap/PipelinesNode.invoke(Pipe
linesNode.java:93)[optimized]

    at
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(Con
creteTreeProcessor.java:235)[optimized]

    at
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.handleCocoo
nRedirect(ConcreteTreeProcessor.java:299)[optimized]

    at
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.access$000(
ConcreteTreeProcessor.java:48)[inlined]

    at
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor$TreeProcess
orRedirector.cocoonRedirect(ConcreteTreeProcessor.java:340)[inlined]

    at
org/apache/cocoon/environment/ForwardRedirector.redirect(ForwardRedirector.j
ava:60)[optimized]

    at
org/apache/cocoon/components/flow/java/AbstractContinuable.sendPage(Abstract
Continuable.java:99)[optimized]

    at
de/transa/disco/flow/im/InstantMessengerFlow.doCheckMessageClosed(InstantMes
sengerFlow.java:88)

    at
sun/reflect/GeneratedMethodAccessor1853.invoke(Ljava/lang/Object;[Ljava/lang
/Object;)Ljava/lang/Object;(Unknown Source)

    at
sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)[optimized]

    at java/lang/reflect/Method.invoke(Method.java:585)[optimized]

    at
org/apache/cocoon/components/flow/java/JavaInterpreter.callFunction(JavaInte
rpreter.java:173)[optimized]

    at
org/apache/cocoon/components/treeprocessor/sitemap/CallFunctionNode.invoke(C
allFunctionNode.java:139)[optimized]

    at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:47)[inlined]

    at
org/apache/cocoon/components/treeprocessor/sitemap/MatchNode.invoke(MatchNod
e.java:108)[optimized]

    at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:69)[inlined]

    at
org/apache/cocoon/components/treeprocessor/sitemap/PipelineNode.invoke(Pipel
ineNode.java:143)[optimized]

    at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:69)[inlined]

    at
org/apache/cocoon/components/treeprocessor/sitemap/PipelinesNode.invoke(Pipe
linesNode.java:93)[optimized]

    at
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(Con
creteTreeProcessor.java:235)[optimized]

    at
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(Con
creteTreeProcessor.java:177)[inlined]

    at
org/apache/cocoon/components/treeprocessor/TreeProcessor.process(TreeProcess
or.java:253)[inlined]

    at
org/apache/cocoon/components/treeprocessor/sitemap/MountNode.invoke(MountNod
e.java:118)[optimized]

    at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:47)[inlined]

    at
org/apache/cocoon/components/treeprocessor/sitemap/MatchNode.invoke(MatchNod
e.java:108)[optimized]

    at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:69)[inlined]

    at
org/apache/cocoon/components/treeprocessor/sitemap/PipelineNode.invoke(Pipel
ineNode.java:143)[optimized]

    at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:69)[inlined]

    at
org/apache/cocoon/components/treeprocessor/sitemap/PipelinesNode.invoke(Pipe
linesNode.java:93)[optimized]

    at
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(Con
creteTreeProcessor.java:235)[optimized]

    at
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(Con
creteTreeProcessor.java:177)[inlined]

    at
org/apache/cocoon/components/treeprocessor/TreeProcessor.process(TreeProcess
or.java:253)[inlined]

    at
org/apache/cocoon/components/treeprocessor/sitemap/MountNode.invoke(MountNod
e.java:118)[optimized]

    at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:47)[inlined]

    at
org/apache/cocoon/components/treeprocessor/sitemap/ActTypeNode.invoke(ActTyp
eNode.java:139)[optimized]

    at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:47)[inlined]

    at
org/apache/cocoon/components/treeprocessor/sitemap/MatchNode.invoke(MatchNod
e.java:108)[optimized]

    at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:69)[inlined]

    at
org/apache/cocoon/components/treeprocessor/sitemap/PipelineNode.invoke(Pipel
ineNode.java:143)[optimized]

    at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:69)[inlined]

    at
org/apache/cocoon/components/treeprocessor/sitemap/PipelinesNode.invoke(Pipe
linesNode.java:93)[optimized]

    at
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(Con
creteTreeProcessor.java:235)[optimized]

    at
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(Con
creteTreeProcessor.java:177)[inlined]

    at
org/apache/cocoon/components/treeprocessor/TreeProcessor.process(TreeProcess
or.java:253)[optimized]

    at org/apache/cocoon/Cocoon.process(Cocoon.java:699)[optimized]

    at
org/apache/cocoon/servlet/CocoonServlet.service(CocoonServlet.java:1154)[opt
imized]

    at
javax/servlet/http/HttpServlet.service(HttpServlet.java:856)[optimized]

    at
weblogic/servlet/internal/StubSecurityHelper$ServletServiceAction.run(StubSe
curityHelper.java:227)[optimized]

    at
weblogic/servlet/internal/StubSecurityHelper.invokeServlet(StubSecurityHelpe
r.java:125)[inlined]

    at
weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:283)[
optimized]

    at
weblogic/servlet/internal/TailFilter.doFilter(TailFilter.java:26)[optimized]

    at
weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:42)[
optimized]

    at
de/transa/disco/filter/SetCharacterEncodingFilter.doFilter(SetCharacterEncod
ingFilter.java:104)[optimized]

    at
weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:42)[
optimized]

    at
de/transa/disco/database/hibernate/CloseHibernateSessionFilter.doFilter(Clos
eHibernateSessionFilter.java:25)[optimized]

    at
weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:42)[
inlined]

    at
weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run(W
ebAppServletContext.java:3242)[optimized]

    at
weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubjec
t.java:321)[optimized]

    at
weblogic/security/service/SecurityManager.runAs(SecurityManager.java:121)[in
lined]

    at
weblogic/servlet/internal/WebAppServletContext.securedExecute(WebAppServletC
ontext.java:2010)[inlined]

    at
weblogic/servlet/internal/WebAppServletContext.execute(WebAppServletContext.
java:1916)[optimized]

    at
weblogic/servlet/internal/ServletRequestImpl.run(ServletRequestImpl.java:136
6)[optimized]

    at
weblogic/work/ExecuteThread.execute(ExecuteThread.java:209)[optimized]

    at weblogic/work/ExecuteThread.run(ExecuteThread.java:181)

    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)

    -- end of trace

 

Causing the System to immediately run out of resources. 

 

The only solution I can think of would be to turn off pooling of components.
Where and how can I do this? Is this simply done by setting the pool-max
properties in the core.properties to 0 or do I have to do more?

We currently have 4 CPUs and 4GB Ram, but this could be extended if we need
to (Virtual environment).

 

Regards,

    Chris

 

 

[ C h r i s t o f e r  D u t z ]

C-Ware IT-Service
Inhaber
Dipl. Inf. Christofer Dutz
Karlstraße. 104, 64285 Darmstadt

fon:  0 61 51 / 27315 - 61
fax:  0 61 51 / 27315 - 64
mobil:  0171 / 7 444 2 33
email:   <mailto:christofer.dutz@c-ware.de> christofer.dutz@c-ware.de
 <http://www.c-ware.de/> http://www.c-ware.de

FA Darmstadt: 07 813 60581

 


Mime
View raw message