cocoon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeremy Aston" <jeremyas...@yahoo.co.uk>
Subject RE: Lucene Problems
Date Tue, 05 Nov 2002 02:49:17 GMT
Hi,

This can be solved on Tomcat 4.x & Windows by tweaking the JAVA_OPTS in
catalina.bat to include a -Xmx setting of -Xmx300000000 and a
<store-janitor> heapsize parameter setting of 250000000 (in cocoon.xconf).
There is no science to these figures, I just made them super large to avoid
mucking around!  I suspect there could be just a _little_ optimising done
here ;-)

Please note that this applies to the use of the JispFilesystemStore as
covered in the <cache-persistent> section of cocoon.xconf.  IMHO this should
be used in preference to the original FilesystemStore for lots of reasons,
not least of which is that the original will invariably bomb on a Windows
platform thanks to it generating hella long filenames.

rgds

Jeremy

 -----Original Message-----
From: 	Peter Neumcke [mailto:pneumcke@virbus.de]
Sent:	04 November 2002 10:26
To:	'cocoon-dev@xml.apache.org'
Subject:	AW: Lucene Problems

Hi Jeremy,

we had this problem too (on Linux)- solved it by setting the maximum number
of open file descriptors to 2048:

ulimit -n 2048


> -----Ursprungliche Nachricht-----
> Von: Jeremy Quinn [mailto:jeremy@media.demon.co.uk]
> Gesendet: Montag, 4. November 2002 11:06
> An: cocoon-dev@xml.apache.org
> Betreff: Re: Lucene Problems
>
>
>
> On Sunday, Nov 3, 2002, at 20:34 Europe/London, Bernhard Huber wrote:
>
> > hi,
> >
>
> Many thanks for your reply, Bernhard.
>
> >>  IOException in index()
>
> >> This causes the index to be incomplete, and causes further
> problems
> >> when you try to search using the index.
> >>
> >> If I use the Lucene Samples in Cocoon, and index my Cocoon
> >> documentation, I always get this error.
> >
> > Hm, i once got this kind of error, too,
> > but i can't remember exactly how i did solve it,
> > perhaps you must set the mergefactor of lucene differently.
> > As much as i remember lucene has some options about how
> many files are
> > kept opened in case of indexing,
> > it later merges the indexes files, ....
>
> Ah Ha! I was beginning to think it was some weakness of
> MacOSX JVM (or
> maybe it still is ....).
>
> Have you any idea where this is configured?
>
> >> Before I started to have the indexing failures, I had another
> >> problem,  this time much more difficult to replicate.
> >>
> >> About every 10 searches, I would get a "Bad Resource" error from
> >> LuceneGenerator. A simple reload of the page would give me
> the search
> >>  hits I expected.
> >>
> >> Can anyone think of a Cocoon cause for these problems, or
> should I
> >> get  onto the Lucene team about them?
> >
> > Do you have anymore detailed exception stack traces?
>
> I do for the indexing, but not for the intermittent fault, which has
> not happened again since the indexing started failing.
>
> Here is the indexing exception report:
>
> Cocoon 2 - Internal server error
>
> type fatal
>
> message IOException in index()
>
> description org.apache.cocoon.ProcessingException: IOException in
> index(): java.io.FileNotFoundException:
> /Users/jermq/Library/TomCat/work/Standalone/localhost/cocoon/cocoon-
> files/index/_1d.f30 (Too many open files)
>
> sender org.apache.cocoon.servlet.CocoonServlet
>
> source Cocoon servlet
>
> stack-trace
>
> org.apache.cocoon.ProcessingException: IOException in index():
> java.io.FileNotFoundException:
> /Users/jermq/Library/TomCat/work/Standalone/localhost/cocoon/cocoon-
> files/index/_1d.f30 (Too many open files)
> 	at
> org.apache.cocoon.components.search.SimpleLuceneCocoonIndexerI
> mpl.index(
> SimpleLuceneCocoonIndexerImpl.java:261)
> 	at
> org.apache.cocoon.www.samples.search.create_index_xsp.createIndex(/
> Users/jermq/Library/TomCat/work/Standalone/localhost/cocoon/cocoon-
> files/org/apache/cocoon/www/samples/search/create_index_xsp.java:106)
> 	at
> org.apache.cocoon.www.samples.search.create_index_xsp.generate
> (/Users/
> jermq/Library/TomCat/work/Standalone/localhost/cocoon/cocoon-f
> iles/org/
> apache/cocoon/www/samples/search/create_index_xsp.java:190)
> 	at
> org.apache.cocoon.generation.ServerPagesGenerator.generate(Ser
> verPagesGe
> nerator.java:269)
> 	at
> org.apache.cocoon.components.pipeline.AbstractProcessingPipeli
> ne.process
> XMLPipeline(AbstractProcessingPipeline.java:512)
> 	at
> org.apache.cocoon.components.pipeline.impl.AbstractCachingProc
> essingPipe
> line.processXMLPipeline(AbstractCachingProcessingPipeline.java:204)
> 	at
> org.apache.cocoon.components.pipeline.AbstractProcessingPipeli
> ne.process
> (AbstractProcessingPipeline.java:483)
> 	at
> org.apache.cocoon.components.treeprocessor.sitemap.SerializeNo
> de.invoke(
> SerializeNode.java:149)
> 	at
> org.apache.cocoon.components.treeprocessor.AbstractParentProce
> ssingNode.
> invokeNodes(AbstractParentProcessingNode.java:85)
> 	at
> org.apache.cocoon.components.treeprocessor.sitemap.PreparableM
> atchNode.i
> nvoke(PreparableMatchNode.java:166)
> 	at
> org.apache.cocoon.components.treeprocessor.AbstractParentProce
> ssingNode.
> invokeNodes(AbstractParentProcessingNode.java:109)
> 	at
> org.apache.cocoon.components.treeprocessor.sitemap.PipelineNod
> e.invoke(P
> ipelineNode.java:153)
> 	at
> org.apache.cocoon.components.treeprocessor.AbstractParentProce
> ssingNode.
> invokeNodes(AbstractParentProcessingNode.java:109)
> 	at
> org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNo
> de.invoke(
> PipelinesNode.java:143)
> 	at
> org.apache.cocoon.components.treeprocessor.TreeProcessor.proce
> ss(TreePro
> cessor.java:326)
> 	at
> org.apache.cocoon.components.treeprocessor.TreeProcessor.proce
> ss(TreePro
> cessor.java:308)
> 	at
> org.apache.cocoon.components.treeprocessor.sitemap.MountNode.i
> nvoke(Moun
> tNode.java:131)
> 	at
> org.apache.cocoon.components.treeprocessor.AbstractParentProce
> ssingNode.
> invokeNodes(AbstractParentProcessingNode.java:85)
> 	at
> org.apache.cocoon.components.treeprocessor.sitemap.PreparableM
> atchNode.i
> nvoke(PreparableMatchNode.java:166)
> 	at
> org.apache.cocoon.components.treeprocessor.AbstractParentProce
> ssingNode.
> invokeNodes(AbstractParentProcessingNode.java:109)
> 	at
> org.apache.cocoon.components.treeprocessor.sitemap.PipelineNod
> e.invoke(P
> ipelineNode.java:153)
> 	at
> org.apache.cocoon.components.treeprocessor.AbstractParentProce
> ssingNode.
> invokeNodes(AbstractParentProcessingNode.java:109)
> 	at
> org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNo
> de.invoke(
> PipelinesNode.java:143)
> 	at
> org.apache.cocoon.components.treeprocessor.TreeProcessor.proce
> ss(TreePro
> cessor.java:326)
> 	at
> org.apache.cocoon.components.treeprocessor.TreeProcessor.proce
> ss(TreePro
> cessor.java:308)
> 	at
> org.apache.cocoon.components.treeprocessor.sitemap.MountNode.i
> nvoke(Moun
> tNode.java:131)
> 	at
> org.apache.cocoon.components.treeprocessor.AbstractParentProce
> ssingNode.
> invokeNodes(AbstractParentProcessingNode.java:85)
> 	at
> org.apache.cocoon.components.treeprocessor.sitemap.PreparableM
> atchNode.i
> nvoke(PreparableMatchNode.java:166)
> 	at
> org.apache.cocoon.components.treeprocessor.AbstractParentProce
> ssingNode.
> invokeNodes(AbstractParentProcessingNode.java:109)
> 	at
> org.apache.cocoon.components.treeprocessor.sitemap.PipelineNod
> e.invoke(P
> ipelineNode.java:153)
> 	at
> org.apache.cocoon.components.treeprocessor.AbstractParentProce
> ssingNode.
> invokeNodes(AbstractParentProcessingNode.java:109)
> 	at
> org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNo
> de.invoke(
> PipelinesNode.java:143)
> 	at
> org.apache.cocoon.components.treeprocessor.TreeProcessor.proce
> ss(TreePro
> cessor.java:326)
> 	at
> org.apache.cocoon.components.treeprocessor.TreeProcessor.proce
> ss(TreePro
> cessor.java:308)
> 	at org.apache.cocoon.Cocoon.process(Cocoon.java:595)
> 	at
> org.apache.cocoon.servlet.CocoonServlet.service(CocoonServlet.
> java:1069)
> 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
> 	at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilt
> er(Applica
> tionFilterChain.java:247)
> 	at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(Appli
> cationFilt
> erChain.java:193)
> 	at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardW
> rapperValv
> e.java:260)
> 	at
> org.apache.catalina.core.StandardPipeline$StandardPipelineValv
> eContext.i
> nvokeNext(StandardPipeline.java:643)
> 	at
> org.apache.catalina.core.StandardPipeline.invoke(StandardPipel
> ine.java:4
> 80)
> 	at
> org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
> 	at
> org.apache.catalina.core.StandardContextValve.invoke(StandardC
> ontextValv
> e.java:191)
> 	at
> org.apache.catalina.core.StandardPipeline$StandardPipelineValv
> eContext.i
> nvokeNext(StandardPipeline.java:643)
> 	at
> org.apache.catalina.core.StandardPipeline.invoke(StandardPipel
> ine.java:4
> 80)
> 	at
> org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
> 	at
> org.apache.catalina.core.StandardContext.invoke(StandardContex
> t.java:239
> 6)
> 	at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHost
> Valve.java
> :180)
> 	at
> org.apache.catalina.core.StandardPipeline$StandardPipelineValv
> eContext.i
> nvokeNext(StandardPipeline.java:643)
> 	at
> org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDi
> spatcherVa
> lve.java:170)
> 	at
> org.apache.catalina.core.StandardPipeline$StandardPipelineValv
> eContext.i
> nvokeNext(StandardPipeline.java:641)
> 	at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReport
> Valve.java
> :172)
> 	at
> org.apache.catalina.core.StandardPipeline$StandardPipelineValv
> eContext.i
> nvokeNext(StandardPipeline.java:641)
> 	at
> org.apache.catalina.core.StandardPipeline.invoke(StandardPipel
> ine.java:4
> 80)
> 	at
> org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
> 	at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEn
> gineValve.
> java:174)
> 	at
> org.apache.catalina.core.StandardPipeline$StandardPipelineValv
> eContext.i
> nvokeNext(StandardPipeline.java:643)
> 	at
> org.apache.catalina.core.StandardPipeline.invoke(StandardPipel
> ine.java:4
> 80)
> 	at
> org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
> 	at
> org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.
> java:223)
> 	at
> org.apache.coyote.http11.Http11Processor.process(Http11Process
> or.java:40
> 5)
> 	at
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandle
> r.processC
> onnection(Http11Protocol.java:380)
> 	at
> org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoi
> nt.java:50
> 8)
> 	at
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(
> ThreadPool
> .java:533)
> 	at java.lang.Thread.run(Thread.java:491)
> java.io.FileNotFoundException:
> /Users/jermq/Library/TomCat/work/Standalone/localhost/cocoon/cocoon-
> files/index/_1d.f30 (Too many open files)
> 	at java.io.RandomAccessFile.open(Native Method)
> 	at java.io.RandomAccessFile.(RandomAccessFile.java:93)
> 	at java.io.RandomAccessFile.(RandomAccessFile.java:138)
> 	at
> org.apache.lucene.store.FSInputStream$Descriptor.(Unknown Source)
> 	at org.apache.lucene.store.FSInputStream.(Unknown Source)
> 	at org.apache.lucene.store.FSDirectory.openFile(Unknown Source)
> 	at
> org.apache.lucene.index.SegmentReader.openNorms(Unknown Source)
> 	at org.apache.lucene.index.SegmentReader.(Unknown Source)
> 	at
> org.apache.lucene.index.IndexWriter.mergeSegments(Unknown Source)
> 	at org.apache.lucene.index.IndexWriter.optimize(Unknown Source)
> 	at
> org.apache.cocoon.components.search.SimpleLuceneCocoonIndexerI
> mpl.index(
> SimpleLuceneCocoonIndexerImpl.java:259)
> 	at
> org.apache.cocoon.www.samples.search.create_index_xsp.createIndex(/
> Users/jermq/Library/TomCat/work/Standalone/localhost/cocoon/cocoon-
> files/org/apache/cocoon/www/samples/search/create_index_xsp.java:106)
> 	at
> org.apache.cocoon.www.samples.search.create_index_xsp.generate
> (/Users/
> jermq/Library/TomCat/work/Standalone/localhost/cocoon/cocoon-f
> iles/org/
> apache/cocoon/www/samples/search/create_index_xsp.java:190)
> 	at
> org.apache.cocoon.generation.ServerPagesGenerator.generate(Ser
> verPagesGe
> nerator.java:269)
> 	at
> org.apache.cocoon.components.pipeline.AbstractProcessingPipeli
> ne.process
> XMLPipeline(AbstractProcessingPipeline.java:512)
> 	at
> org.apache.cocoon.components.pipeline.impl.AbstractCachingProc
> essingPipe
> line.processXMLPipeline(AbstractCachingProcessingPipeline.java:204)
> 	at
> org.apache.cocoon.components.pipeline.AbstractProcessingPipeli
> ne.process
> (AbstractProcessingPipeline.java:483)
> 	at
> org.apache.cocoon.components.treeprocessor.sitemap.SerializeNo
> de.invoke(
> SerializeNode.java:149)
> 	at
> org.apache.cocoon.components.treeprocessor.AbstractParentProce
> ssingNode.
> invokeNodes(AbstractParentProcessingNode.java:85)
> 	at
> org.apache.cocoon.components.treeprocessor.sitemap.PreparableM
> atchNode.i
> nvoke(PreparableMatchNode.java:166)
> 	at
> org.apache.cocoon.components.treeprocessor.AbstractParentProce
> ssingNode.
> invokeNodes(AbstractParentProcessingNode.java:109)
> 	at
> org.apache.cocoon.components.treeprocessor.sitemap.PipelineNod
> e.invoke(P
> ipelineNode.java:153)
> 	at
> org.apache.cocoon.components.treeprocessor.AbstractParentProce
> ssingNode.
> invokeNodes(AbstractParentProcessingNode.java:109)
> 	at
> org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNo
> de.invoke(
> PipelinesNode.java:143)
> 	at
> org.apache.cocoon.components.treeprocessor.TreeProcessor.proce
> ss(TreePro
> cessor.java:326)
> 	at
> org.apache.cocoon.components.treeprocessor.TreeProcessor.proce
> ss(TreePro
> cessor.java:308)
> 	at
> org.apache.cocoon.components.treeprocessor.sitemap.MountNode.i
> nvoke(Moun
> tNode.java:131)
> 	at
> org.apache.cocoon.components.treeprocessor.AbstractParentProce
> ssingNode.
> invokeNodes(AbstractParentProcessingNode.java:85)
> 	at
> org.apache.cocoon.components.treeprocessor.sitemap.PreparableM
> atchNode.i
> nvoke(PreparableMatchNode.java:166)
> 	at
> org.apache.cocoon.components.treeprocessor.AbstractParentProce
> ssingNode.
> invokeNodes(AbstractParentProcessingNode.java:109)
> 	at
> org.apache.cocoon.components.treeprocessor.sitemap.PipelineNod
> e.invoke(P
> ipelineNode.java:153)
> 	at
> org.apache.cocoon.components.treeprocessor.AbstractParentProce
> ssingNode.
> invokeNodes(AbstractParentProcessingNode.java:109)
> 	at
> org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNo
> de.invoke(
> PipelinesNode.java:143)
> 	at
> org.apache.cocoon.components.treeprocessor.TreeProcessor.proce
> ss(TreePro
> cessor.java:326)
> 	at
> org.apache.cocoon.components.treeprocessor.TreeProcessor.proce
> ss(TreePro
> cessor.java:308)
> 	at
> org.apache.cocoon.components.treeprocessor.sitemap.MountNode.i
> nvoke(Moun
> tNode.java:131)
> 	at
> org.apache.cocoon.components.treeprocessor.AbstractParentProce
> ssingNode.
> invokeNodes(AbstractParentProcessingNode.java:85)
> 	at
> org.apache.cocoon.components.treeprocessor.sitemap.PreparableM
> atchNode.i
> nvoke(PreparableMatchNode.java:166)
> 	at
> org.apache.cocoon.components.treeprocessor.AbstractParentProce
> ssingNode.
> invokeNodes(AbstractParentProcessingNode.java:109)
> 	at
> org.apache.cocoon.components.treeprocessor.sitemap.PipelineNod
> e.invoke(P
> ipelineNode.java:153)
> 	at
> org.apache.cocoon.components.treeprocessor.AbstractParentProce
> ssingNode.
> invokeNodes(AbstractParentProcessingNode.java:109)
> 	at
> org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNo
> de.invoke(
> PipelinesNode.java:143)
> 	at
> org.apache.cocoon.components.treeprocessor.TreeProcessor.proce
> ss(TreePro
> cessor.java:326)
> 	at
> org.apache.cocoon.components.treeprocessor.TreeProcessor.proce
> ss(TreePro
> cessor.java:308)
> 	at org.apache.cocoon.Cocoon.process(Cocoon.java:595)
> 	at
> org.apache.cocoon.servlet.CocoonServlet.service(CocoonServlet.
> java:1069)
> 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
> 	at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilt
> er(Applica
> tionFilterChain.java:247)
> 	at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(Appli
> cationFilt
> erChain.java:193)
> 	at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardW
> rapperValv
> e.java:260)
> 	at
> org.apache.catalina.core.StandardPipeline$StandardPipelineValv
> eContext.i
> nvokeNext(StandardPipeline.java:643)
> 	at
> org.apache.catalina.core.StandardPipeline.invoke(StandardPipel
> ine.java:4
> 80)
> 	at
> org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
> 	at
> org.apache.catalina.core.StandardContextValve.invoke(StandardC
> ontextValv
> e.java:191)
> 	at
> org.apache.catalina.core.StandardPipeline$StandardPipelineValv
> eContext.i
> nvokeNext(StandardPipeline.java:643)
> 	at
> org.apache.catalina.core.StandardPipeline.invoke(StandardPipel
> ine.java:4
> 80)
> 	at
> org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
> 	at
> org.apache.catalina.core.StandardContext.invoke(StandardContex
> t.java:239
> 6)
> 	at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHost
> Valve.java
> :180)
> 	at
> org.apache.catalina.core.StandardPipeline$StandardPipelineValv
> eContext.i
> nvokeNext(StandardPipeline.java:643)
> 	at
> org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDi
> spatcherVa
> lve.java:170)
> 	at
> org.apache.catalina.core.StandardPipeline$StandardPipelineValv
> eContext.i
> nvokeNext(StandardPipeline.java:641)
> 	at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReport
> Valve.java
> :172)
> 	at
> org.apache.catalina.core.StandardPipeline$StandardPipelineValv
> eContext.i
> nvokeNext(StandardPipeline.java:641)
> 	at
> org.apache.catalina.core.StandardPipeline.invoke(StandardPipel
> ine.java:4
> 80)
> 	at
> org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
> 	at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEn
> gineValve.
> java:174)
> 	at
> org.apache.catalina.core.StandardPipeline$StandardPipelineValv
> eContext.i
> nvokeNext(StandardPipeline.java:643)
> 	at
> org.apache.catalina.core.StandardPipeline.invoke(StandardPipel
> ine.java:4
> 80)
> 	at
> org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
> 	at
> org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.
> java:223)
> 	at
> org.apache.coyote.http11.Http11Processor.process(Http11Process
> or.java:40
> 5)
> 	at
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandle
> r.processC
> onnection(Http11Protocol.java:380)
> 	at
> org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoi
> nt.java:50
> 8)
> 	at
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(
> ThreadPool
> .java:533)
> 	at java.lang.Thread.run(Thread.java:491)
>
> request-uri
>
> /cocoon/samples/search/create
>
> path-info
>
> samples/search/create
>
>
>
> Thanks again
>
> regards Jeremy
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
> For additional commands, email: cocoon-dev-help@xml.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org


__________________________________________________
Do You Yahoo!?
Everything you'll ever need on one web page
from News and Sport to Email and Music Charts
http://uk.my.yahoo.com

---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org


Mime
View raw message