lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler" <...@thetaphi.de>
Subject RE: (LUCENE-1844) Speed up junit tests
Date Sun, 21 Feb 2010 20:35:38 GMT
I fixed the backwards test and removed the assertion there. This was forgotten to be merged
back. The reason why this test fails:

 

TestRAMDir creates 10 threads that start to add files to a RAMDir and adds content to these
files. The RAMFile updates its own size and also updates the size of the enclosing RAMDir
(using AtomicLong). The problem is, that all threads do this in parallel and maybe another
thread added the contents to the file and have updated the parents AtomicLong but not yet
its own size. As update local size and in RAMDir is no longer atomic for the another thread,
the assertion fails. In previous versions of RamDir both updates were one atomic op, synced
to the directory. For speed reasons this was removed, so writing to RAMFiles no longer locks
the parent directory.

 

-----

Uwe Schindler

H.-H.-Meier-Allee 63, D-28213 Bremen

 <http://www.thetaphi.de/> http://www.thetaphi.de

eMail: uwe@thetaphi.de

 

From: Robert Muir [mailto:rcmuir@gmail.com] 
Sent: Sunday, February 21, 2010 8:44 PM
To: java-dev@lucene.apache.org
Subject: Re: (LUCENE-1844) Speed up junit tests

 

Mike removed this assertion in LUCENE-2095, so this only happens in the backwards tests.

On Sun, Feb 21, 2010 at 2:26 PM, Uwe Schindler <uwe@thetaphi.de> wrote:

Another test-bug that now shows as a real test failure (and not only in stderr as before,
thanks to LUCENE-2274). Happens quite often, will check logs on Hudson, how often this happens.

The test failure on my solaris box occurred in backwards branch of trunk.

 

    [junit] Testsuite: org.apache.lucene.store.TestRAMDirectory

    [junit] Tests run: 4, Failures: 1, Errors: 0, Time elapsed: 0.259 sec

    [junit] 

    [junit] ------------- Standard Error -----------------

    [junit] The following exceptions were thrown by threads:

    [junit] *** Thread: Thread-16978 ***

    [junit] junit.framework.AssertionFailedError: expected:<84992> but was:<86016>

    [junit]     at junit.framework.Assert.fail(Assert.java:47)

    [junit]     at junit.framework.Assert.failNotEquals(Assert.java:277)

    [junit]     at junit.framework.Assert.assertEquals(Assert.java:64)

    [junit]     at junit.framework.Assert.assertEquals(Assert.java:130)

    [junit]     at junit.framework.Assert.assertEquals(Assert.java:136)

    [junit]     at org.apache.lucene.store.TestRAMDirectory$1.run(TestRAMDirectory.java:129)

    [junit] ------------- ---------------- ---------------

    [junit] Testcase: testRAMDirectorySize(org.apache.lucene.store.TestRAMDirectory):   FAILED

    [junit] Some threads throwed uncaught exceptions!

    [junit] junit.framework.AssertionFailedError: Some threads throwed uncaught exceptions!

    [junit]     at org.apache.lucene.util.LuceneTestCase.tearDown(LuceneTestCase.java:142)

    [junit]     at org.apache.lucene.store.TestRAMDirectory.tearDown(TestRAMDirectory.java:160)

    [junit]     at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:250)

    [junit] 

    [junit] 

    [junit] TEST org.apache.lucene.store.TestRAMDirectory FAILED

 

-----

Uwe Schindler

H.-H.-Meier-Allee 63, D-28213 Bremen

http://www.thetaphi.de <http://www.thetaphi.de/> 

eMail: uwe@thetaphi.de

 

From: Robert Muir [mailto:rcmuir@gmail.com] 
Sent: Sunday, February 21, 2010 10:53 AM


To: java-dev@lucene.apache.org
Subject: Re: (LUCENE-1844) Speed up junit tests

 

here is what i was worried about, if we cannot fix, i can revert back to forking. This is
not reproduceable all the time:

    [junit] Testcase: testParallelMultiSort(org.apache.lucene.search.TestSort): Caused an
ERROR
    [junit] java.util.ConcurrentModificationException
    [junit] java.lang.RuntimeException: java.util.ConcurrentModificationException
    [junit]     at org.apache.lucene.search.ParallelMultiSearcher.foreach(ParallelMultiSearcher.java:216)
    [junit]     at org.apache.lucene.search.ParallelMultiSearcher.search(ParallelMultiSearcher.java:121)
    [junit]     at org.apache.lucene.search.Searcher.search(Searcher.java:49)
    [junit]     at org.apache.lucene.search.TestSort.assertMatches(TestSort.java:965)
    [junit]     at org.apache.lucene.search.TestSort.runMultiSorts(TestSort.java:891)
    [junit]     at org.apache.lucene.search.TestSort.testParallelMultiSort(TestSort.java:629)
    [junit]     at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:208)
    [junit] Caused by: java.util.ConcurrentModificationException
    [junit]     at java.util.WeakHashMap$HashIterator.nextEntry(WeakHashMap.java:762)
    [junit]     at java.util.WeakHashMap$KeyIterator.next(WeakHashMap.java:795)
    [junit]     at org.apache.lucene.search.FieldCacheImpl.getCacheEntries(FieldCacheImpl.java:75)
    [junit]     at org.apache.lucene.util.FieldCacheSanityChecker.checkSanity(FieldCacheSanityChecker.java:72)
    [junit]     at org.apache.lucene.search.FieldCacheImpl$Cache.printNewInsanity(FieldCacheImpl.java:205)
    [junit]     at org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:194)
    [junit]     at org.apache.lucene.search.FieldCacheImpl.getInts(FieldCacheImpl.java:357)
    [junit]     at org.apache.lucene.search.FieldCacheImpl$IntCache.createValue(FieldCacheImpl.java:373)
    [junit]     at org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:183)
    [junit]     at org.apache.lucene.search.FieldCacheImpl.getInts(FieldCacheImpl.java:357)
    [junit]     at org.apache.lucene.search.FieldComparator$IntComparator.setNextReader(FieldComparator.java:438)
    [junit]     at org.apache.lucene.search.TopFieldCollector$OneComparatorNonScoringCollector.setNextReader(TopFieldCollector.java:95)
    [junit]     at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:207)
    [junit]     at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:197)
    [junit]     at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:175)
    [junit]     at org.apache.lucene.search.MultiSearcher$MultiSearcherCallableWithSort.call(MultiSearcher.java:420)
    [junit]     at org.apache.lucene.search.MultiSearcher$MultiSearcherCallableWithSort.call(MultiSearcher.java:394)
    [junit]     at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    [junit]     at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    [junit]     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    [junit]     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    [junit]     at java.lang.Thread.run(Thread.java:619)
    [junit]
    [junit]
    [junit] TEST org.apache.lucene.search.TestSort FAILED

On Sun, Feb 21, 2010 at 2:10 AM, Robert Muir <rcmuir@gmail.com> wrote:

ok, can do. if it turns out there is a problem, we can revert until some work has been done
on the tests.

 

On Sat, Feb 20, 2010 at 5:45 PM, Michael McCandless <lucene@mikemccandless.com> wrote:

Currently the tests run 1 jvm per test suite (eg, TestIndexWriter has
its own jvm), I believe, and we haven't seen test failures... so I
think for the most part tests are not interfering with each other
(messing up global state).

It should be less likely that we see interactions across test suites
(but obviously still possible).

I think we should commit this and then if there are somehow problems
we can address them, then?

Mike


On Sun, Feb 14, 2010 at 6:27 AM, Robert Muir <rcmuir@gmail.com> wrote:
> its not just statics, I think we should really look at ensuring files are
> closed etc, or eventually there will be a problem!
>
> I guess in general the tradeoff is, it requires us to have better test code.
>
> On Sun, Feb 14, 2010 at 5:53 AM, Uwe Schindler <uwe@thetaphi.de> wrote:
>>
>> At least we should check all core tests to not set any static defaults
>> without try...finally! Are there any possibilities inside Eclipse/other-IDEs
>> to check this?
>>
>> Uwe
>>
>> -----
>> Uwe Schindler
>> H.-H.-Meier-Allee 63, D-28213 Bremen
>> http://www.thetaphi.de
>> eMail: uwe@thetaphi.de
>>
>> > -----Original Message-----
>> > From: Michael McCandless [mailto:lucene@mikemccandless.com]
>> > Sent: Sunday, February 14, 2010 11:43 AM
>> > To: java-dev@lucene.apache.org
>> > Subject: Re: (LUCENE-1844) Speed up junit tests
>> >
>> > Wow -- this is MUCH faster!  I think we should switch...
>> >
>> > It seems like we use a batchtest for all core tests, then for all
>> > back-compat tests, then once per contrib package?  Ie, so "ant
>> > test-core" uses one jvm?
>> >
>> > I think we should simply fix any badly behaved tests (that don't
>> > restore statics).  It's impressive we already have no test failures
>> > when we do this... I guess our tests are already cleaning things up
>> > (though also probably not often changing global state, or, changing it
>> > in a way that'd lead other tests to fail).
>> >
>> > Mike
>> >
>> > On Sat, Feb 13, 2010 at 5:23 PM, Robert Muir <rcmuir@gmail.com> wrote:
>> > > On Fri, Nov 27, 2009 at 1:27 PM, Michael McCandless
>> > > <lucene@mikemccandless.com> wrote:
>> > >>
>> > >> Also one thing I'd love to try is NOT forking the JVM for each test
>> > >> (fork="no" in the junit task).  I wonder how much time that'd buy...
>> > >>
>> > >
>> > > it shaves off a good deal of time on my machine.
>> > >
>> > > 'ant test-core': 4 minutes, 39 seconds -> 3 minutes, 3 seconds
>> > > 'ant test':  11 minutes, 8 seconds -> 7 minutes, 13 seconds
>> > >
>> > > however, it makes me a little nervous because i'm not sure all the
>> > tests
>> > > cleanup nicely if they change statics and stuff.
>> > > anyway, here's the trivial patch (you don't want fork=no, because it
>> > turns
>> > > off assertions)
>> > >
>> > > Index: common-build.xml
>> > > ===================================================================
>> > > --- common-build.xml    (revision 909395)
>> > > +++ common-build.xml    (working copy)
>> > > @@ -398,7 +398,7 @@
>> > >             </condition>
>> > >                 <mkdir dir="@{junit.output.dir}"/>
>> > >             <junit printsummary="off" haltonfailure="no"
>> > maxmemory="512M"
>> > > -             errorProperty="tests.failed"
>> > failureProperty="tests.failed">
>> > > +             errorProperty="tests.failed"
>> > failureProperty="tests.failed"
>> > > forkmode="perBatch">
>> > >               <classpath refid="@{junit.classpath}"/>
>> > >               <assertions>
>> > >                 <enable package="org.apache.lucene"/>
>> > >
>> > > --
>> > > Robert Muir
>> > > rcmuir@gmail.com
>> > >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> > For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>
>
>
> --
> Robert Muir
> rcmuir@gmail.com
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org





-- 

Robert Muir
rcmuir@gmail.com




-- 
Robert Muir
rcmuir@gmail.com




-- 
Robert Muir
rcmuir@gmail.com


Mime
View raw message