incubator-jspwiki-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Juan Pablo Santos Rodríguez <juanpablo.san...@gmail.com>
Subject Re: 3 out of 6 tests from SearchManagerTest failing if run all together
Date Thu, 21 Jun 2012 23:38:46 GMT
Hallo,

@Harry: I've been comparing bot setInfoStream outputs. They're basically
the same output, only a couple of differences
- when running all 6 tests I only get up to 15 IW, instead of 18
- the exceptions either when writing the segments file or when closing the
index (and therefore committing changes)
IW 8 [Fri Jun 22 00:36:22 CEST 2012; JSPWiki Lucene Indexer]: commit: wrote
segments file "segments_1"
IFD [Fri Jun 22 00:36:22 CEST 2012; JSPWiki Lucene Indexer]: now checkpoint
"segments_1" [0 segments ; isCommit = true]
IFD [Fri Jun 22 00:36:22 CEST 2012; JSPWiki Lucene Indexer]: delete pending
file _0.tis
IFD [Fri Jun 22 00:36:22 CEST 2012; JSPWiki Lucene Indexer]: delete "_0.tis"
IFD [Fri Jun 22 00:36:22 CEST 2012; JSPWiki Lucene Indexer]: unable to
remove file "_0.tis": java.io.IOException: Cannot delete
E:\tmp\testworkdir\lucene\_0.tis; Will re-try later.
IFD [Fri Jun 22 00:36:22 CEST 2012; JSPWiki Lucene Indexer]: delete pending
file _0.nrm
IFD [Fri Jun 22 00:36:22 CEST 2012; JSPWiki Lucene Indexer]: delete "_0.nrm"
IFD [Fri Jun 22 00:36:22 CEST 2012; JSPWiki Lucene Indexer]: unable to
remove file "_0.nrm": java.io.IOException: Cannot delete
E:\tmp\testworkdir\lucene\_0.nrm; Will re-try later.

My environment: Win 7 x64, with or without cygwin, with JDK 1.6.0_19 first
and 1.7.0_3 secondly, 64 bit edition of both JDKs. I can try to run the ant
tests at work (JDK 1.6, 32-bit) but most probably not until next Monday /
Tuesday. Also, I'll try to run with newer JDK6/7 versions but again, not
until next Monday/Tuesday.

@Florian: it seems to me that the problem is caused due to a concurrency
issue, as the index is updated in a separate thread, so it would be
difficult to serialize its execution. Also, being a concurrency issue makes
it hard to reproduce in a unit test.

The singleton approach would be feasible too and most probably would also
get rid off the (my) exceptions problem. I preferred the re-schedule
approach because if an exception occurs while doing the index operation, we
are now logging it and then acting as if nothing had happened, with the
page not being indexed; re-scheduling the index operation simply places the
WikiPage back in the index queue (as the indexing hasn't been performed) so
the index operation can be retried later on.


rgrds,
juan pablo


On Thu, Jun 21, 2012 at 10:28 PM, Florian Holeczek <florian@holeczek.de>wrote:

> Hi Juan Pablo,
>
> I'd rather not fix working code only because some tests are not working
> due to (test-)architectural issues.
> Not sure if I understood the problem completely though.
> Wouldn't the right way be serializing the tests or using a singleton for
> the Lucene index?
>
> Regards
>  Florian
>
>
> ----- Ursprüngliche Mail -----
> Von: "Juan Pablo Santos Rodríguez" <juanpablo.santos@gmail.com>
> An: jspwiki-dev@incubator.apache.org
> Gesendet: Donnerstag, 21. Juni 2012 18:55:35
> Betreff: Re: 3 out of 6 tests from SearchManagerTest failing if run all
> together
>
> Ok, so finally, I think I got it.
>
> Updates to lucene Index are done in a separate thread, via the
> LuceneSearchProvider.LuceneUpdater inner class, which is executed on a
> different thread. This may lead to situations where there are more than one
> IndexWriter instances working on the same directory, provoking the
> exception noted on the first email.
>
> The proposed fix is relatively easy, on
> LuceneSearchProvider.luceneIndexPage( WikiPage page, String text,
> IndexWriter writer ) method, if an update index operation fails, simply
> re-schedule the index operation:
>
>  protected synchronized void updateLuceneIndex( WikiPage page, String text
> )
>    {
> [...]
>        try
>        {
> [...]
>            luceneIndexPage( page, text, writer );
>        }
>        catch ( IOException e )
>        {
>            log.error("Unable to update page '" + page.getName() + "' from
> Lucene index", e);
>            reindexPage( page );
>        }
>        catch( Exception e )
>        {
>            log.error("Unexpected Lucene exception - please check
> configuration!",e);
>            reindexPage( page );
>        }
>        finally
> [...]
>
> This approach could cause some sort of infinite loop if there's something
> severe happening (i.e.: disk full, no permission to write on the work dir,
> etc.), but I think that those kind of situations would be noted throughout
> the whole application, so there would be little to do on those cases
> anyway.
>
> WDYT about the suggested change?
>
>
> br,
> juan pablo
>
>
> On Thu, Jun 21, 2012 at 1:00 AM, Juan Pablo Santos Rodríguez <
> juanpablo.santos@gmail.com> wrote:
>
> > Hello Harry,
> >
> > I've increased the time up to 100s and still the same tests failing /
> > exceptions. I think it has to be something related to index updating, as
> > the IOExceptions noted by enabling writer.setInfoStream( System.out );
> > occur every time a page is updated.
> >
> > Also, the exceptions are caught always at
> > LuceneSearchProvider.LuceneUpdater threads so maybe they are trying to
> open
> > an IndexWriter while the JUnit test has his own IndexWriter opened.
> Anyway,
> > I'll continue doing more tests to see if I can isolate the cause of the
> > exception.
> >
> >
> > regards,
> > juan pablo
> >
> >
> >
> > On Wed, Jun 20, 2012 at 6:21 AM, Harry Metske <harry.metske@gmail.com
> >wrote:
> >
> >> Wat happens of you increase the waittime specified in the
> >> SearchManagerTest?
> >>
> >> I think I had these ones when this time was too short.
> >>
> >> regards,
> >> Harry
> >> Op 20 jun. 2012 00:45 schreef "Juan Pablo Santos Rodríguez" <
> >> juanpablo.santos@gmail.com> het volgende:
> >>
> >> > Hi,
> >> >
> >> > I'm stuck running the tests from SearchManagerTest. If I run them
> >> > individually, then all is OK.
> >> >
> >> > However, if I run them all together (via Ant task or via Eclipse),
> then
> >> the
> >> > following tests fail: testSimplesearch3, testTitleSearch,
> >> testTitleSearch2,
> >> > at the assertEquals() regarding res.size()
> >> >
> >> > jspwikitests.log yields several exceptions like this one:
> >> >
> >> > java.io.IOException: Cannot overwrite:
> E:\tmp\testworkdir\lucene\_0.fdt
> >> >    at
> >> >
> org.apache.lucene.store.FSDirectory.ensureCanWrite(FSDirectory.java:316)
> >> >    at
> >> > org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:305)
> >> >    at
> org.apache.lucene.index.FieldsWriter.<init>(FieldsWriter.java:83)
> >> >    at
> >> >
> >> >
> >>
> org.apache.lucene.index.StoredFieldsWriter.initFieldsWriter(StoredFieldsWriter.java:64)
> >> >    at
> >> >
> >> >
> >>
> org.apache.lucene.index.StoredFieldsWriter.finishDocument(StoredFieldsWriter.java:107)
> >> >    at
> >> >
> >> >
> >>
> org.apache.lucene.index.StoredFieldsWriter$PerDoc.finish(StoredFieldsWriter.java:151)
> >> >    at
> >> >
> >> >
> >>
> org.apache.lucene.index.DocumentsWriter$WaitQueue.writeDocument(DocumentsWriter.java:1404)
> >> >    at
> >> >
> >> >
> >>
> org.apache.lucene.index.DocumentsWriter$WaitQueue.add(DocumentsWriter.java:1424)
> >> >    at
> >> >
> >> >
> >>
> org.apache.lucene.index.DocumentsWriter.finishDocument(DocumentsWriter.java:1043)
> >> >    at
> >> >
> >> >
> >>
> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:772)
> >> >    at
> >> > org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2060)
> >> >    at
> >> > org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2034)
> >> >    at
> >> >
> >> >
> >>
> org.apache.wiki.search.LuceneSearchProvider.luceneIndexPage(LuceneSearchProvider.java:485)
> >> >    at
> >> >
> >> >
> >>
> org.apache.wiki.search.LuceneSearchProvider.updateLuceneIndex(LuceneSearchProvider.java:377)
> >> > [...]
> >> >
> >> > Also, if I place writer.setInfoStream( System.out ); just after the
> >> writer
> >> > is instantiated, I get several messages similar to this one:
> >> > IFD [Wed Jun 20 00:20:07 CEST 2012; JSPWiki Lucene Indexer]: unable to
> >> > remove file "_0.fdt": java.io.IOException: Cannot delete
> >> > E:\tmp\testworkdir\lucene\_0.fdt; Will re-try later.
> >> >
> >> > Done a little googling and found that may be related to opening an
> >> > IndexWriter on the index while there's another one open [1] or
> removing
> >> > files while the index is open at that directory [2]. I'll continue
> >> looking
> >> > at this tomorrow, but I was wondering if anyone else is experiencing
> >> this
> >> > problem too.
> >> >
> >> >
> >> > br,
> >> > juan pablo
> >> >
> >> > [1]: http://www.gossamer-threads.com/lists/lucene/java-user/92262
> >> > [2]:
> http://www.gossamer-threads.com/lists/lucene/java-user/62201#62201
> >> >
> >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message