cocoon-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Christofer Dutz" <christofer.d...@c-ware.de>
Subject AW: [english 84%] Re: Cocoon 2.1.10 Application Freezing
Date Thu, 03 Jul 2008 09:24:09 GMT
Hi,

well I turned off the pooling and the system is behaving nicely. The problem
I was having, was that tons of threads were blocked, which all blocked
resources themselves.
All resources were blocked by threads, that were blocked and hereby could
not release them.

I didn't turn off the pooling of DB-Connections and so on, but I turned it
off for all Sitemap Components.

In theory modern Garbage Collections work a lot better than pooling for
small and cheap objects. In Spring they even suggest leaving it away
completely. Since Yesterday I have to confirm this. 
My application is now surviving load-peaks it wouldn't have survived
yesterday and the general average system load dropped about 5%.

Regards,
    Chris



-----Ursprüngliche Nachricht-----
Von: Ralph Goers [mailto:Ralph.Goers@dslextreme.com] 
Gesendet: Donnerstag, 3. Juli 2008 05:53
An: users@cocoon.apache.org
Betreff: [english 84%] Re: Cocoon 2.1.10 Application Freezing

A couple of thoughts.
1. In my experience turning off pooling will make matters worse, not better.
2. You show threads blocked, but they aren't using CPU. What is using 
all the CPU? Look at the threads that are actually running. Chances are 
the clue to the bottleneck will be there. The two threads you have shown 
say "blocked" so I doubt either of them is the cause.
3. If the bottleneck can't be alleviated then you need to increase your 
pool size to allow for that.

Christofer.Dutz@c-ware.de wrote:
>
> Hi,
>
> I am sort of going to get a heart attack in the next hours, if I can’t 
> resolve this quickly.
>
> We recently deployed a new Version of a intranet-application of a 
> large logistic company here in Germany. It is completely written in 
> cocoon and performs extraordinary nicely. Until at one point the 
> System starts thrashing. CPU Usage climbs from 12% to 80% and no 
> request are served anymore.
>
> Here a little more details on the Setup:
>
> - Windows 2003 Server
>
> - 4 GB Ram … 1,5 GB reserved for the appserver
>
> - Bea Weblogic 9.2
>
> - JRockit 1.5 VM
>
> - MySQL 5.1 DB Backend
>
> During normal operation a thread dump is about 30k big … during 
> thrashing this increases to 1,5MB.
>
> Yesterday we had similar Problems which were unfortunately related to 
> too much logging (100 Threads blocked because they were locked from 
> the Cocoon Log system). After removing a lot of the Log Output the 
> system performed nicely even under heavier load. Until the whole thing 
> blew up again. This time the Thread Dump showed a lot of Threads 
> blocked at:
>
> "[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default 
> (self-tuning)'" id=35 idx=0x80 tid=5092 prio=5 alive, in native, 
> blocked, daemon
>
> -- Blocked trying to get lock: java/lang/Object@0x011A8D38[unlocked]
>
> at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)
>
> at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1630)[optimized]
>
> at jrockit/vm/Locks.lockFat(Locks.java:1731)[optimized]
>
> at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1277)[optimized]
>
> at 
>
org/apache/avalon/excalibur/pool/InstrumentedResourceLimitingPool.put(Instru
mentedResourceLimitingPool.java:411)[inlined]
>
> at 
>
org/apache/avalon/excalibur/component/PoolableComponentHandler.doPut(Poolabl
eComponentHandler.java:212)[optimized]
>
> at 
>
org/apache/avalon/excalibur/component/ComponentHandler.put(ComponentHandler.
java:425)[inlined]
>
> at 
>
org/apache/avalon/excalibur/component/ExcaliburComponentSelector.release(Exc
aliburComponentSelector.java:307)[inlined]
>
> at 
>
org/apache/cocoon/components/ExtendedComponentSelector.release(ExtendedCompo
nentSelector.java:301)[optimized]
>
> at 
>
org/apache/cocoon/components/pipeline/AbstractProcessingPipeline.recycle(Abs
tractProcessingPipeline.java:743)[optimized]
>
> at 
>
org/apache/avalon/excalibur/pool/InstrumentedResourceLimitingPool.put(Instru
mentedResourceLimitingPool.java:407)[inlined]
>
> at 
>
org/apache/avalon/excalibur/component/PoolableComponentHandler.doPut(Poolabl
eComponentHandler.java:212)[optimized]
>
> at 
>
org/apache/avalon/excalibur/component/ComponentHandler.put(ComponentHandler.
java:425)[inlined]
>
> at 
>
org/apache/avalon/excalibur/component/ExcaliburComponentSelector.release(Exc
aliburComponentSelector.java:307)[inlined]
>
> at 
>
org/apache/cocoon/components/ExtendedComponentSelector.release(ExtendedCompo
nentSelector.java:301)[optimized]
>
> at 
>
org/apache/cocoon/components/treeprocessor/InvokeContext.dispose(InvokeConte
xt.java:283)
>
> at 
>
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(Con
creteTreeProcessor.java:179)[inlined]
>
> at 
>
org/apache/cocoon/components/treeprocessor/TreeProcessor.process(TreeProcess
or.java:253)[inlined]
>
> at 
>
org/apache/cocoon/components/treeprocessor/sitemap/MountNode.invoke(MountNod
e.java:118)[optimized]
>
> at 
>
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:47)[inlined]
>
> at 
>
org/apache/cocoon/components/treeprocessor/sitemap/MatchNode.invoke(MatchNod
e.java:108)[optimized]
>
> at 
>
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:69)[inlined]
>
> at 
>
org/apache/cocoon/components/treeprocessor/sitemap/PipelineNode.invoke(Pipel
ineNode.java:143)[optimized]
>
> at 
>
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:69)[inlined]
>
> at 
>
org/apache/cocoon/components/treeprocessor/sitemap/PipelinesNode.invoke(Pipe
linesNode.java:93)[optimized]
>
> at 
>
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(Con
creteTreeProcessor.java:235)[optimized]
>
> at 
>
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(Con
creteTreeProcessor.java:177)[inlined]
>
> at 
>
org/apache/cocoon/components/treeprocessor/TreeProcessor.process(TreeProcess
or.java:253)[inlined]
>
> at 
>
org/apache/cocoon/components/treeprocessor/sitemap/MountNode.invoke(MountNod
e.java:118)[optimized]
>
> at 
>
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:47)[inlined]
>
> at 
>
org/apache/cocoon/components/treeprocessor/sitemap/ActTypeNode.invoke(ActTyp
eNode.java:139)[optimized]
>
> at 
>
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:47)[inlined]
>
> at 
>
org/apache/cocoon/components/treeprocessor/sitemap/MatchNode.invoke(MatchNod
e.java:108)[optimized]
>
> at 
>
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:69)[inlined]
>
> at 
>
org/apache/cocoon/components/treeprocessor/sitemap/PipelineNode.invoke(Pipel
ineNode.java:143)[optimized]
>
> at 
>
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:69)[inlined]
>
> at 
>
org/apache/cocoon/components/treeprocessor/sitemap/PipelinesNode.invoke(Pipe
linesNode.java:93)[optimized]
>
> at 
>
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(Con
creteTreeProcessor.java:235)[optimized]
>
> at 
>
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(Con
creteTreeProcessor.java:177)[inlined]
>
> at 
>
org/apache/cocoon/components/treeprocessor/TreeProcessor.process(TreeProcess
or.java:253)[optimized]
>
> at org/apache/cocoon/Cocoon.process(Cocoon.java:699)[optimized]
>
> at 
>
org/apache/cocoon/servlet/CocoonServlet.service(CocoonServlet.java:1154)[opt
imized]
>
> at javax/servlet/http/HttpServlet.service(HttpServlet.java:856)[optimized]
>
> at 
>
weblogic/servlet/internal/StubSecurityHelper$ServletServiceAction.run(StubSe
curityHelper.java:227)[optimized]
>
> at 
>
weblogic/servlet/internal/StubSecurityHelper.invokeServlet(StubSecurityHelpe
r.java:125)[inlined]
>
> at 
>
weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:283)[
optimized]
>
> at 
>
weblogic/servlet/internal/TailFilter.doFilter(TailFilter.java:26)[optimized]
>
> at 
>
weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:42)[
optimized]
>
> at 
>
de/transa/disco/filter/SetCharacterEncodingFilter.doFilter(SetCharacterEncod
ingFilter.java:104)[optimized]
>
> at 
>
weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:42)[
optimized]
>
> at 
>
de/transa/disco/database/hibernate/CloseHibernateSessionFilter.doFilter(Clos
eHibernateSessionFilter.java:25)[optimized]
>
> at 
>
weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:42)[
inlined]
>
> at 
>
weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run(W
ebAppServletContext.java:3242)[optimized]
>
> at 
>
weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubjec
t.java:321)[optimized]
>
> at 
>
weblogic/security/service/SecurityManager.runAs(SecurityManager.java:121)[in
lined]
>
> at 
>
weblogic/servlet/internal/WebAppServletContext.securedExecute(WebAppServletC
ontext.java:2010)[inlined]
>
> at 
>
weblogic/servlet/internal/WebAppServletContext.execute(WebAppServletContext.
java:1916)[optimized]
>
> at 
>
weblogic/servlet/internal/ServletRequestImpl.run(ServletRequestImpl.java:136
6)[optimized]
>
> at weblogic/work/ExecuteThread.execute(ExecuteThread.java:209)[optimized]
>
> at weblogic/work/ExecuteThread.run(ExecuteThread.java:181)
>
> at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
>
> -- end of trace
>
> As far as I understand the thing, the problem are pool limitations to 
> some of the sitemap components. In general this should not be a 
> problem. Unfortunately my application uses JavaFlow and a lot of Flows 
> have to send Data to other Systems. They all use internal pipelines to 
> generate this output. My guess now is, that if a Flow is executed to 
> produce output, it blocks some resources. Maybe some resources it 
> needs itself to finish the request, causing a deadlock. This would be 
> the case in the following Thread-Dump:
>
> "[ACTIVE] ExecuteThread: '4' for queue: 'weblogic.kernel.Default 
> (self-tuning)'" id=36 idx=0x84 tid=5100 prio=5 alive, in native, 
> native_blocked, daemon
>
> at 
>
jrockit/vm/Allocator.allocLargeObjectOrArray(IIIZ)Ljava/lang/Object;(Native 
> Method)
>
> at 
> jrockit/vm/Allocator.allocLargerThanFreeTLA(Allocator.java:831)[inlined]
>
> at jrockit/vm/Allocator.allocSlowCaseInner(Allocator.java:947)[inlined]
>
> at jrockit/vm/Allocator.allocSlowCase(Allocator.java:793)[optimized]
>
> at 
>
org/apache/cocoon/util/BufferedOutputStream.<init>(BufferedOutputStream.java
:63)[optimized]
>
> at 
>
org/apache/cocoon/util/BufferedOutputStream.<init>(BufferedOutputStream.java
:46)
>
> at 
>
org/apache/cocoon/environment/AbstractEnvironment.getOutputStream(AbstractEn
vironment.java:490)
>
> at 
>
org/apache/cocoon/environment/http/HttpEnvironment.getOutputStream(HttpEnvir
onment.java:263)
>
> at 
>
org/apache/cocoon/environment/wrapper/EnvironmentWrapper.getOutputStream(Env
ironmentWrapper.java:311)
>
> at 
>
org/apache/cocoon/components/pipeline/AbstractProcessingPipeline.processXMLP
ipeline(AbstractProcessingPipeline.java:577)[optimized]
>
> at 
>
org/apache/cocoon/components/pipeline/AbstractProcessingPipeline.process(Abs
tractProcessingPipeline.java:481)[optimized]
>
> at 
>
org/apache/cocoon/components/treeprocessor/sitemap/SerializeNode.invoke(Seri
alizeNode.java:121)[optimized]
>
> at 
>
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:47)[inlined]
>
> at 
>
org/apache/cocoon/components/treeprocessor/sitemap/MatchNode.invoke(MatchNod
e.java:108)[optimized]
>
> at 
>
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:69)[inlined]
>
> at 
>
org/apache/cocoon/components/treeprocessor/sitemap/PipelineNode.invoke(Pipel
ineNode.java:143)[optimized]
>
> at 
>
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:69)[inlined]
>
> at 
>
org/apache/cocoon/components/treeprocessor/sitemap/PipelinesNode.invoke(Pipe
linesNode.java:93)[optimized]
>
> at 
>
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(Con
creteTreeProcessor.java:235)[optimized]
>
> at 
>
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.handleCocoo
nRedirect(ConcreteTreeProcessor.java:299)[optimized]
>
> at 
>
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.access$000(
ConcreteTreeProcessor.java:48)[inlined]
>
> at 
>
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor$TreeProcess
orRedirector.cocoonRedirect(ConcreteTreeProcessor.java:340)[inlined]
>
> at 
>
org/apache/cocoon/environment/ForwardRedirector.redirect(ForwardRedirector.j
ava:60)[optimized]
>
> at 
>
org/apache/cocoon/components/flow/java/AbstractContinuable.sendPage(Abstract
Continuable.java:99)[optimized]
>
> at 
>
de/transa/disco/flow/im/InstantMessengerFlow.doCheckMessageClosed(InstantMes
sengerFlow.java:88)
>
> at 
>
sun/reflect/GeneratedMethodAccessor1853.invoke(Ljava/lang/Object;[Ljava/lang
/Object;)Ljava/lang/Object;(Unknown 
> Source)
>
> at 
>
sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)[optimized]
>
> at java/lang/reflect/Method.invoke(Method.java:585)[optimized]
>
> at 
>
org/apache/cocoon/components/flow/java/JavaInterpreter.callFunction(JavaInte
rpreter.java:173)[optimized]
>
> at 
>
org/apache/cocoon/components/treeprocessor/sitemap/CallFunctionNode.invoke(C
allFunctionNode.java:139)[optimized]
>
> at 
>
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:47)[inlined]
>
> at 
>
org/apache/cocoon/components/treeprocessor/sitemap/MatchNode.invoke(MatchNod
e.java:108)[optimized]
>
> at 
>
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:69)[inlined]
>
> at 
>
org/apache/cocoon/components/treeprocessor/sitemap/PipelineNode.invoke(Pipel
ineNode.java:143)[optimized]
>
> at 
>
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:69)[inlined]
>
> at 
>
org/apache/cocoon/components/treeprocessor/sitemap/PipelinesNode.invoke(Pipe
linesNode.java:93)[optimized]
>
> at 
>
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(Con
creteTreeProcessor.java:235)[optimized]
>
> at 
>
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(Con
creteTreeProcessor.java:177)[inlined]
>
> at 
>
org/apache/cocoon/components/treeprocessor/TreeProcessor.process(TreeProcess
or.java:253)[inlined]
>
> at 
>
org/apache/cocoon/components/treeprocessor/sitemap/MountNode.invoke(MountNod
e.java:118)[optimized]
>
> at 
>
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:47)[inlined]
>
> at 
>
org/apache/cocoon/components/treeprocessor/sitemap/MatchNode.invoke(MatchNod
e.java:108)[optimized]
>
> at 
>
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:69)[inlined]
>
> at 
>
org/apache/cocoon/components/treeprocessor/sitemap/PipelineNode.invoke(Pipel
ineNode.java:143)[optimized]
>
> at 
>
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:69)[inlined]
>
> at 
>
org/apache/cocoon/components/treeprocessor/sitemap/PipelinesNode.invoke(Pipe
linesNode.java:93)[optimized]
>
> at 
>
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(Con
creteTreeProcessor.java:235)[optimized]
>
> at 
>
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(Con
creteTreeProcessor.java:177)[inlined]
>
> at 
>
org/apache/cocoon/components/treeprocessor/TreeProcessor.process(TreeProcess
or.java:253)[inlined]
>
> at 
>
org/apache/cocoon/components/treeprocessor/sitemap/MountNode.invoke(MountNod
e.java:118)[optimized]
>
> at 
>
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:47)[inlined]
>
> at 
>
org/apache/cocoon/components/treeprocessor/sitemap/ActTypeNode.invoke(ActTyp
eNode.java:139)[optimized]
>
> at 
>
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:47)[inlined]
>
> at 
>
org/apache/cocoon/components/treeprocessor/sitemap/MatchNode.invoke(MatchNod
e.java:108)[optimized]
>
> at 
>
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:69)[inlined]
>
> at 
>
org/apache/cocoon/components/treeprocessor/sitemap/PipelineNode.invoke(Pipel
ineNode.java:143)[optimized]
>
> at 
>
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:69)[inlined]
>
> at 
>
org/apache/cocoon/components/treeprocessor/sitemap/PipelinesNode.invoke(Pipe
linesNode.java:93)[optimized]
>
> at 
>
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(Con
creteTreeProcessor.java:235)[optimized]
>
> at 
>
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(Con
creteTreeProcessor.java:177)[inlined]
>
> at 
>
org/apache/cocoon/components/treeprocessor/TreeProcessor.process(TreeProcess
or.java:253)[optimized]
>
> at org/apache/cocoon/Cocoon.process(Cocoon.java:699)[optimized]
>
> at 
>
org/apache/cocoon/servlet/CocoonServlet.service(CocoonServlet.java:1154)[opt
imized]
>
> at javax/servlet/http/HttpServlet.service(HttpServlet.java:856)[optimized]
>
> at 
>
weblogic/servlet/internal/StubSecurityHelper$ServletServiceAction.run(StubSe
curityHelper.java:227)[optimized]
>
> at 
>
weblogic/servlet/internal/StubSecurityHelper.invokeServlet(StubSecurityHelpe
r.java:125)[inlined]
>
> at 
>
weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:283)[
optimized]
>
> at 
>
weblogic/servlet/internal/TailFilter.doFilter(TailFilter.java:26)[optimized]
>
> at 
>
weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:42)[
optimized]
>
> at 
>
de/transa/disco/filter/SetCharacterEncodingFilter.doFilter(SetCharacterEncod
ingFilter.java:104)[optimized]
>
> at 
>
weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:42)[
optimized]
>
> at 
>
de/transa/disco/database/hibernate/CloseHibernateSessionFilter.doFilter(Clos
eHibernateSessionFilter.java:25)[optimized]
>
> at 
>
weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:42)[
inlined]
>
> at 
>
weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run(W
ebAppServletContext.java:3242)[optimized]
>
> at 
>
weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubjec
t.java:321)[optimized]
>
> at 
>
weblogic/security/service/SecurityManager.runAs(SecurityManager.java:121)[in
lined]
>
> at 
>
weblogic/servlet/internal/WebAppServletContext.securedExecute(WebAppServletC
ontext.java:2010)[inlined]
>
> at 
>
weblogic/servlet/internal/WebAppServletContext.execute(WebAppServletContext.
java:1916)[optimized]
>
> at 
>
weblogic/servlet/internal/ServletRequestImpl.run(ServletRequestImpl.java:136
6)[optimized]
>
> at weblogic/work/ExecuteThread.execute(ExecuteThread.java:209)[optimized]
>
> at weblogic/work/ExecuteThread.run(ExecuteThread.java:181)
>
> at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
>
> -- end of trace
>
> Causing the System to immediately run out of resources.
>
> The only solution I can think of would be to turn off pooling of 
> components. Where and how can I do this? Is this simply done by 
> setting the pool-max properties in the core.properties to 0 or do I 
> have to do more?
>
> We currently have 4 CPUs and 4GB Ram, but this could be extended if we 
> need to (Virtual environment).
>
> Regards,
>
> Chris
>
> [ C h r i s t o f e r D u t z ]
>
> C-Ware IT-Service
> Inhaber
> Dipl. Inf. Christofer Dutz
> Karlstraße. 104, 64285 Darmstadt
>
> fon: 0 61 51 / 27315 - 61
> fax: 0 61 51 / 27315 - 64
> mobil: 0171 / 7 444 2 33
> email: christofer.dutz@c-ware.de <mailto:christofer.dutz@c-ware.de>
> http://www.c-ware.de <http://www.c-ware.de/>
>
> FA Darmstadt: 07 813 60581
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org




---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org


Mime
View raw message