incubator-jspwiki-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Florian Holeczek <flor...@holeczek.de>
Subject Re: 3 out of 6 tests from SearchManagerTest failing if run all together
Date Thu, 21 Jun 2012 20:28:40 GMT
Hi Juan Pablo,

I'd rather not fix working code only because some tests are not working due to (test-)architectural
issues.
Not sure if I understood the problem completely though.
Wouldn't the right way be serializing the tests or using a singleton for the Lucene index?

Regards
 Florian


----- Ursprüngliche Mail -----
Von: "Juan Pablo Santos Rodríguez" <juanpablo.santos@gmail.com>
An: jspwiki-dev@incubator.apache.org
Gesendet: Donnerstag, 21. Juni 2012 18:55:35
Betreff: Re: 3 out of 6 tests from SearchManagerTest failing if run all together

Ok, so finally, I think I got it.

Updates to lucene Index are done in a separate thread, via the
LuceneSearchProvider.LuceneUpdater inner class, which is executed on a
different thread. This may lead to situations where there are more than one
IndexWriter instances working on the same directory, provoking the
exception noted on the first email.

The proposed fix is relatively easy, on
LuceneSearchProvider.luceneIndexPage( WikiPage page, String text,
IndexWriter writer ) method, if an update index operation fails, simply
re-schedule the index operation:

 protected synchronized void updateLuceneIndex( WikiPage page, String text )
    {
[...]
        try
        {
[...]
            luceneIndexPage( page, text, writer );
        }
        catch ( IOException e )
        {
            log.error("Unable to update page '" + page.getName() + "' from
Lucene index", e);
            reindexPage( page );
        }
        catch( Exception e )
        {
            log.error("Unexpected Lucene exception - please check
configuration!",e);
            reindexPage( page );
        }
        finally
[...]

This approach could cause some sort of infinite loop if there's something
severe happening (i.e.: disk full, no permission to write on the work dir,
etc.), but I think that those kind of situations would be noted throughout
the whole application, so there would be little to do on those cases anyway.

WDYT about the suggested change?


br,
juan pablo


On Thu, Jun 21, 2012 at 1:00 AM, Juan Pablo Santos Rodríguez <
juanpablo.santos@gmail.com> wrote:

> Hello Harry,
>
> I've increased the time up to 100s and still the same tests failing /
> exceptions. I think it has to be something related to index updating, as
> the IOExceptions noted by enabling writer.setInfoStream( System.out );
> occur every time a page is updated.
>
> Also, the exceptions are caught always at
> LuceneSearchProvider.LuceneUpdater threads so maybe they are trying to open
> an IndexWriter while the JUnit test has his own IndexWriter opened. Anyway,
> I'll continue doing more tests to see if I can isolate the cause of the
> exception.
>
>
> regards,
> juan pablo
>
>
>
> On Wed, Jun 20, 2012 at 6:21 AM, Harry Metske <harry.metske@gmail.com>wrote:
>
>> Wat happens of you increase the waittime specified in the
>> SearchManagerTest?
>>
>> I think I had these ones when this time was too short.
>>
>> regards,
>> Harry
>> Op 20 jun. 2012 00:45 schreef "Juan Pablo Santos Rodríguez" <
>> juanpablo.santos@gmail.com> het volgende:
>>
>> > Hi,
>> >
>> > I'm stuck running the tests from SearchManagerTest. If I run them
>> > individually, then all is OK.
>> >
>> > However, if I run them all together (via Ant task or via Eclipse), then
>> the
>> > following tests fail: testSimplesearch3, testTitleSearch,
>> testTitleSearch2,
>> > at the assertEquals() regarding res.size()
>> >
>> > jspwikitests.log yields several exceptions like this one:
>> >
>> > java.io.IOException: Cannot overwrite: E:\tmp\testworkdir\lucene\_0.fdt
>> >    at
>> > org.apache.lucene.store.FSDirectory.ensureCanWrite(FSDirectory.java:316)
>> >    at
>> > org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:305)
>> >    at org.apache.lucene.index.FieldsWriter.<init>(FieldsWriter.java:83)
>> >    at
>> >
>> >
>> org.apache.lucene.index.StoredFieldsWriter.initFieldsWriter(StoredFieldsWriter.java:64)
>> >    at
>> >
>> >
>> org.apache.lucene.index.StoredFieldsWriter.finishDocument(StoredFieldsWriter.java:107)
>> >    at
>> >
>> >
>> org.apache.lucene.index.StoredFieldsWriter$PerDoc.finish(StoredFieldsWriter.java:151)
>> >    at
>> >
>> >
>> org.apache.lucene.index.DocumentsWriter$WaitQueue.writeDocument(DocumentsWriter.java:1404)
>> >    at
>> >
>> >
>> org.apache.lucene.index.DocumentsWriter$WaitQueue.add(DocumentsWriter.java:1424)
>> >    at
>> >
>> >
>> org.apache.lucene.index.DocumentsWriter.finishDocument(DocumentsWriter.java:1043)
>> >    at
>> >
>> >
>> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:772)
>> >    at
>> > org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2060)
>> >    at
>> > org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2034)
>> >    at
>> >
>> >
>> org.apache.wiki.search.LuceneSearchProvider.luceneIndexPage(LuceneSearchProvider.java:485)
>> >    at
>> >
>> >
>> org.apache.wiki.search.LuceneSearchProvider.updateLuceneIndex(LuceneSearchProvider.java:377)
>> > [...]
>> >
>> > Also, if I place writer.setInfoStream( System.out ); just after the
>> writer
>> > is instantiated, I get several messages similar to this one:
>> > IFD [Wed Jun 20 00:20:07 CEST 2012; JSPWiki Lucene Indexer]: unable to
>> > remove file "_0.fdt": java.io.IOException: Cannot delete
>> > E:\tmp\testworkdir\lucene\_0.fdt; Will re-try later.
>> >
>> > Done a little googling and found that may be related to opening an
>> > IndexWriter on the index while there's another one open [1] or removing
>> > files while the index is open at that directory [2]. I'll continue
>> looking
>> > at this tomorrow, but I was wondering if anyone else is experiencing
>> this
>> > problem too.
>> >
>> >
>> > br,
>> > juan pablo
>> >
>> > [1]: http://www.gossamer-threads.com/lists/lucene/java-user/92262
>> > [2]: http://www.gossamer-threads.com/lists/lucene/java-user/62201#62201
>> >
>>
>
>

Mime
View raw message