cocoon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alexander Daniel (JIRA)" <j...@apache.org>
Subject [jira] Updated: (COCOON-2173) AbstractCachingProcessingPipeline: Two requests can deadlock each other
Date Fri, 21 Mar 2008 17:25:24 GMT

     [ https://issues.apache.org/jira/browse/COCOON-2173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Alexander Daniel updated COCOON-2173:
-------------------------------------

    Attachment: patchFor2.1.11.txt

For applying the patch to Cocoon 2.1.11 use patch -p6 <$DownloadFolder/patchFor2.1.11.txt

I considered three possibilities to fix this issue
(1) using a timed wait
(2) removing the synchronization of requests with pipe locks completely
(3) redesign of the synchronization of requests

I chose (1) for simplicity. The current waitForLock implementation returns true when the lock
is the current thread (request for 2.1-dev and 2.2-dev). For the calling method validatePipeline
this means that there was no cached response found. When there is a timeout on the timed wait
it behaves the same way. The default timeout is 250ms but it can be configured with the timeoutForSynchronizingRequests
parameter.

Unit tests for the new method timedWait are included in patchFor2.1.11.txt

> AbstractCachingProcessingPipeline: Two requests can deadlock each other
> -----------------------------------------------------------------------
>
>                 Key: COCOON-2173
>                 URL: https://issues.apache.org/jira/browse/COCOON-2173
>             Project: Cocoon
>          Issue Type: Bug
>          Components: - Components: Sitemap
>    Affects Versions: 2.1.9, 2.1.10, 2.1.11, 2.2-dev (Current SVN)
>            Reporter: Alexander Daniel
>         Attachments: patchFor2.1.11.txt, reproduceMultipleThreads.tar.gz, reproduceMultipleThreads2.2RC3-SNAPSHOT.tar.gz
>
>
> Two requests can deadlock each other when they depend on the same resources which they
acquire in a different order. I can reproduce that in Cocoon 2.1.11 and Cocoon 2.2-RC3-SNAPSHOT:
> * request A: generating lock for 55933 
> * request B: generating lock for 58840 
> * request B: waiting for lock 55933 which is hold by request A 
> * request A: waiting for lock 58840 which is hold by request B 
> I can reproduce this behaviour with Apache Bench and following pipeline: 
> * terminal 1: Apache Bench request A (ab -k -n 10000 -c 25 http://localhost:8888/samples/reproduceMultipleThreads/productOfferForDevice/55933/)

> * terminal 2: Apache Bench request B (ab -k -n 10000 -c 25 http://localhost:8888/samples/reproduceMultipleThreads/productOfferForDevice/58840/)

> * terminal 3: touching the two data files every second to invalidate the cache (while
true; do echo -n "."; touch 55933.xml 58840.xml; sleep 1; done) 
> * pipeline: 
> <map:pipeline type="caching"> 
>   <map:match pattern="productOfferForDevice*/*/"> 
>     <map:generate src="cocoon:/exists/{2}.xml" label="a"/> 
>     <map:transform type="xsltc" src="productOfferIncludeDevice.xsl" label="b">

>         <map:parameter name="noInc" value="{1}"/> 
>     </map:transform> 
>     <map:transform type="include" label="c"/> 
>     <map:serialize type="xml"/> 
> </map:match> 
>   <map:match pattern="exists/**"> 
>     <map:act type="resource-exists"> 
>         <map:parameter name="url" value="{1}" /> 
>         <map:generate src="{../1}" /> 
>         <map:serialize type="xml" /> 
>     </map:act> 
>     <!-- not found --> 
>     <map:generate src="dummy.xml" /> 
>     <map:serialize type="xml" /> 
>   </map:match> 
> </map:pipeline> 
> After some seconds the deadlock occurs ==> 
> * Apache Bench requests run into a timeout 
> * I can see following pipe locks in the default transient store: 
> PIPELOCK:PK_G-file-cocoon://samples/reproduceMultipleThreads/exists/55933.xml?pipelinehash=-910770960103935149_T-xsltc-file:///Users/alex/dev/cocoon/cocoon-2.1.11/build/webapp/samples/reproduceMultipleThreads/productOfferIncludeDevice.xsl;noInc=_T-include-I_S-xml-1
(class: org.mortbay.util.ThreadPool$PoolThread) 
> PIPELOCK:PK_G-file-cocoon://samples/reproduceMultipleThreads/exists/58840.xml?pipelinehash=-4996088883111986478_T-xsltc-file:///Users/alex/dev/cocoon/cocoon-2.1.11/build/webapp/samples/reproduceMultipleThreads/productOfferIncludeDevice.xsl;noInc=_T-include-I_S-xml-1
(class: org.mortbay.util.ThreadPool$PoolThread) 
> PIPELOCK:PK_G-file-file:///Users/alex/dev/cocoon/cocoon-2.1.11/build/webapp/samples/reproduceMultipleThreads/55933.xml
(class: org.mortbay.util.ThreadPool$PoolThread) 
> PIPELOCK:PK_G-file-file:///Users/alex/dev/cocoon/cocoon-2.1.11/build/webapp/samples/reproduceMultipleThreads/58840.xml
(class: org.mortbay.util.ThreadPool$PoolThread) 
> I added some logging to AbstractCachingProcessingPipeline.java which reconfirms the explanations
above: 
> INFO (2008-03-13) 13:50.16:072 [sitemap] (/samples/reproduceMultipleThreads/productOfferForDevice/55933/)
PoolThread-47/AbstractCachingProcessingPipeline: generating lock PIPELOCK:PK_G-file-cocoon://samples/reproduceMultipleThreads/exists/55933.xml?pipelinehash=-910770960103935149_T-xsltc-file:///Users/alex/dev/cocoon/cocoon-2.1.11/build/webapp/samples/reproduceMultipleThreads/productOfferIncludeDevice.xsl;noInc=_T-include-I_S-xml-1

> INFO (2008-03-13) 13:50.16:074 [sitemap] (/samples/reproduceMultipleThreads/productOfferForDevice/55933/)
PoolThread-47/AbstractCachingProcessingPipeline: generating lock PIPELOCK:PK_G-file-file:///Users/alex/dev/cocoon/cocoon-2.1.11/build/webapp/samples/reproduceMultipleThreads/55933.xml

> INFO (2008-03-13) 13:50.16:075 [sitemap] (/samples/reproduceMultipleThreads/productOfferForDevice/58840/)
PoolThread-6/AbstractCachingProcessingPipeline: generating lock PIPELOCK:PK_G-file-cocoon://samples/reproduceMultipleThreads/exists/58840.xml?pipelinehash=-4996088883111986478_T-xsltc-file:///Users/alex/dev/cocoon/cocoon-2.1.11/build/webapp/samples/reproduceMultipleThreads/productOfferIncludeDevice.xsl;noInc=_T-include-I_S-xml-1

> INFO (2008-03-13) 13:50.16:075 [sitemap] (/samples/reproduceMultipleThreads/productOfferForDevice/58840/)
PoolThread-6/AbstractCachingProcessingPipeline: generating lock PIPELOCK:PK_G-file-file:///Users/alex/dev/cocoon/cocoon-2.1.11/build/webapp/samples/reproduceMultipleThreads/58840.xml

> INFO (2008-03-13) 13:50.16:281 [sitemap] (/samples/reproduceMultipleThreads/productOfferForDevice/58840/)
PoolThread-6/AbstractCachingProcessingPipeline: waiting for lock PIPELOCK:PK_G-file-file:///Users/alex/dev/cocoon/cocoon-2.1.11/build/webapp/samples/reproduceMultipleThreads/55933.xml

> INFO (2008-03-13) 13:50.16:304 [sitemap] (/samples/reproduceMultipleThreads/productOfferForDevice/55933/)
PoolThread-47/AbstractCachingProcessingPipeline: waiting for lock PIPELOCK:PK_G-file-file:///Users/alex/dev/cocoon/cocoon-2.1.11/build/webapp/samples/reproduceMultipleThreads/58840.xml

> Reproduce yourself with Cocoon 2.1.11:
> * download and extract Cocoon 2.1.11 
> * cd $CocoonHome 
> * ./build.sh 
> * cd build/webapp/samples 
> * tar -xzf $DownloadFolder/reproduceMultipleThreads.tar.gz 
> * cd ../../.. 
> * ./cocoon.sh 
> * open 3 terminals and cd into $CocoonHome/build/webapp/samples/reproduceMultipleThreads
in each 
> * dry run without invalidating the cache to see that everything is working: 
>   - terminal 1: ./terminal1.sh 
>   - terminal 2: ./terminal2.sh 
> * run with invalidating the cache every seconds: 
>   - terminal 1: ./terminal1.sh 
>   - terminal 2: ./terminal2.sh 
>   - terminal 3: ./terminal3.sh 
> * When Apache Bench has run into a timeout you can view the pipelocks with http://localhost:8888/samples/reproduceMultipleThreads/pipelocks

> Reproduce yourself with Cocoon 2.2RC3-SNAPSHOT:
> * svn checkout http://svn.apache.org/repos/asf/cocoon/trunk
> * mvn clean install
> * ./cocoon.sh
> * cd core/cocoon-webapp/target/work/blocks/cocoon-core-main-sample
> * tar -xzf $DownloadFolder/reproduceMultipleThreads2.2RC3-SNAPSHOT.tar.gz 
> * open 3 terminals and cd into $CocoonTrunk/core/cocoon-webapp/target/work/blocks/cocoon-core-main-sample/reproduceMultipleThreads/
in each
> * dry run without invalidating the cache to see that everything is working: 
>   - terminal 1: ./terminal1.sh 
>   - terminal 2: ./terminal2.sh 
> * run with invalidating the cache every second: 
>   - terminal 1: ./terminal1.sh 
>   - terminal 2: ./terminal2.sh 
>   - terminal 3: ./terminal3.sh 
> * When Apache Bench has run into a timeout you can view the pipelocks with http://localhost:8888/samples/core/reproduceMultipleThreads/pipelocks

> Since we are currently facing this issue on our production servers I would appreciate
a hint on how to fix that issue in a good way.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message