lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler" <...@thetaphi.de>
Subject RE: [JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.7.0_67) - Build # 4310 - Still Failing!
Date Sun, 14 Sep 2014 11:29:24 GMT
Hi,

I know why it fails consequently with the new Java 7 APIs:
FileOutputStream since version 1.0 of Java has a custom finalizer() method, so the file gets
closed on GC, kept until now for backwards compatibility. But if you the stream with Java
7's NIO.2, the created OutputStream instance (some non-public internal platform specific impl
class) no longer has a finalizer(), so the file handle is never closed when the OutputStream
is GCed. This is great (f*ck finalizers)!

So by using NIO.2 we have a very nice "detector" for unclosed file, if people do broken stuff
like "new Properties(new FileInputStream(...))" as seen in Solr quite often!

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de


> -----Original Message-----
> From: Uwe Schindler [mailto:uwe@thetaphi.de]
> Sent: Sunday, September 14, 2014 1:16 PM
> To: dev@lucene.apache.org
> Subject: RE: [JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.7.0_67) -
> Build # 4310 - Still Failing!
> 
> The problem here is incomplete cleanup:
> 
> This test parses the whole algorithm, it then creates an Algorithm instance
> containing all Tasks to execute. The "WriteLineDocsTask" is instantiated,
> which in fact opens the output stream for the line docs file in its ctor. But as
> the tasks are never executed, so the Task's close() is never called.
> 
> I fixed the whole thing by making Algorithm implement AutoCloseable (not
> Closeable, because it throws not only IOException). I also backported the
> whole stuff to Lucene 4, because this is unrelated to Robert's changes.
> Roberts changes just triggered other code execution paths, so GC did not
> clean up algorithm.
> 
> TODO: verify other uses of Algorithm and check that they are correctly
> closed. In fact I have seen this bug sometimes when running tests locally, so
> this was existent since long time, just in most cases GC was faster!
> 
> Uwe
> 
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
> 
> 
> > -----Original Message-----
> > From: Uwe Schindler [mailto:uwe@thetaphi.de]
> > Sent: Sunday, September 14, 2014 12:21 PM
> > To: dev@lucene.apache.org
> > Subject: RE: [JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.7.0_67) -
> > Build # 4310 - Still Failing!
> >
> > This fails reproducible.
> >
> > I will dig!
> >
> > -----
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, D-28213 Bremen
> > http://www.thetaphi.de
> > eMail: uwe@thetaphi.de
> >
> > > -----Original Message-----
> > > From: Policeman Jenkins Server [mailto:jenkins@thetaphi.de]
> > > Sent: Sunday, September 14, 2014 2:48 AM
> > > To: noble@apache.org; sarowe@gmail.com; rmuir@apache.org;
> > > uwe@thetaphi.de; mikemccand@apache.org; dev@lucene.apache.org
> > > Subject: [JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.7.0_67) -
> > > Build #
> > > 4310 - Still Failing!
> > >
> > > Build:
> > > http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/4310/
> > > Java: 64bit/jdk1.7.0_67 -XX:-UseCompressedOops -
> > > XX:+UseConcMarkSweepGC
> > >
> > > 1 tests failed.
> > > FAILED:
> > >
> >
> junit.framework.TestSuite.org.apache.lucene.benchmark.byTask.TestPerfT
> > > a
> > > sksParse
> > >
> > > Error Message:
> > > Could not remove the following files (in the order of attempts):
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt:
> > > java.nio.file.AccessDeniedException:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt:
> > > java.nio.file.AccessDeniedException:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001:
> > > java.nio.file.DirectoryNotEmptyException:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001
> > >
> > > Stack Trace:
> > > java.io.IOException: Could not remove the following files (in the
> > > order of
> > > attempts):
> > >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt:
> > > java.nio.file.AccessDeniedException:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt
> > >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt:
> > > java.nio.file.AccessDeniedException:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt
> > >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001:
> > > java.nio.file.DirectoryNotEmptyException:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001
> > >
> > > 	at __randomizedtesting.SeedInfo.seed([2D34950549928D3F]:0)
> > > 	at org.apache.lucene.util.IOUtils.rm(IOUtils.java:288)
> > > 	at
> > > org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(Tes
> > > tR
> > > ul
> > > eTemporaryFilesCleanup.java:126)
> > > 	at
> > > com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlwa
> > > ys
> > > (T
> > > estRuleAdapter.java:31)
> > > 	at
> > > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(S
> > > ta
> > > t
> > > ementAdapter.java:43)
> > > 	at
> > > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(S
> > > ta
> > > t
> > > ementAdapter.java:36)
> > > 	at
> > > org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRul
> > > eA
> > > ss
> > > ertionsRequired.java:43)
> > > 	at
> > > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFa
> > > il
> > > ure
> > > .java:48)
> > > 	at
> > > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(Tes
> > > tR
> > > ule
> > > IgnoreAfterMaxFailures.java:65)
> > > 	at
> > > org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleI
> > > gn
> > > ore
> > > TestSuites.java:55)
> > > 	at
> > > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(S
> > > ta
> > > t
> > > ementAdapter.java:36)
> > > 	at
> > >
> >
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.
> > > run(ThreadLeakControl.java:365)
> > > 	at java.lang.Thread.run(Thread.java:745)
> > >
> > >
> > >
> > >
> > > Build Log:
> > > [...truncated 7622 lines...]
> > >    [junit4] Suite:
> org.apache.lucene.benchmark.byTask.TestPerfTasksParse
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> Spatial Strategy:
> > > RecursivePrefixTreeStrategy(SPG:(QuadPrefixTree(maxLevels:26,ctx:Spa
> > > ti al Context{geo=true, calculator=Haversine,
> > > worldBounds=Rect(minX=-
> > > 180.0,maxX=180.0,minY=-
> > > 90.0,maxY=90.0)})),pointsOnly,pruneLeafyBranches)
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   2> NOTE: test params are: codec=Lucene410: {}, docValues:{},
> > > sim=DefaultSimilarity, locale=sq_AL, timezone=America/Yakutat
> > >    [junit4]   2> NOTE: Windows 7 6.1 amd64/Oracle Corporation 1.7.0_67
> (64-
> > > bit)/cpus=2,threads=1,free=120281832,total=140083200
> > >    [junit4]   2> NOTE: All tests run in this JVM: [DocMakerTest,
> > > WriteLineDocTaskTest, AddIndexesTaskTest, StreamUtilsTest,
> > > TrecContentSourceTest, TestQualityRun, PerfTaskTest,
> > > LineDocSourceTest, TestPerfTasksLogic, TestPerfTasksParse]
> > >    [junit4]   2> NOTE: reproduce with: ant test  -
> > Dtestcase=TestPerfTasksParse
> > > -Dtests.seed=2D34950549928D3F -Dtests.slow=true -Dtests.locale=sq_AL
> > > - Dtests.timezone=America/Yakutat -Dtests.file.encoding=ISO-8859-1
> > >    [junit4] ERROR   0.00s | TestPerfTasksParse (suite) <<<
> > >    [junit4]    > Throwable #1: java.io.IOException: Could not remove the
> > > following files (in the order of attempts):
> > >    [junit4]    >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt:
> > > java.nio.file.AccessDeniedException:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt
> > >    [junit4]    >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt:
> > > java.nio.file.AccessDeniedException:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt
> > >    [junit4]    >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001:
> > > java.nio.file.DirectoryNotEmptyException:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001
> > >    [junit4]    > 	at
> > > __randomizedtesting.SeedInfo.seed([2D34950549928D3F]:0)
> > >    [junit4]    > 	at org.apache.lucene.util.IOUtils.rm(IOUtils.java:288)
> > >    [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
> > >    [junit4] Completed in 3.00s, 3 tests, 1 error <<< FAILURES!
> > >
> > > [...truncated 33 lines...]
> > > BUILD FAILED
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > > Windows\build.xml:491: The following error occurred while executing
> > > this
> > > line:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > > Windows\build.xml:471: The following error occurred while executing
> > > this
> > > line:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > Windows\build.xml:60:
> > > The following error occurred while executing this line:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\extra-
> > > targets.xml:39: The following error occurred while executing this line:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > > Windows\lucene\build.xml:462: The following error occurred while
> > > executing this line:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > > Windows\lucene\common-build.xml:2140: The following error occurred
> > > while executing this line:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > > Windows\lucene\module-build.xml:58: The following error occurred
> > > while executing this line:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > > Windows\lucene\common-build.xml:1358: The following error occurred
> > > while executing this line:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > > Windows\lucene\common-build.xml:961: There were test failures: 18
> > > suites,
> > > 93 tests, 1 suite-level error
> > >
> > > Total time: 31 minutes 50 seconds
> > > Build step 'Invoke Ant' marked build as failure [description-setter]
> > > Description set: Java: 64bit/jdk1.7.0_67 -XX:-UseCompressedOops -
> > > XX:+UseConcMarkSweepGC Archiving artifacts Recording test results
> > > Email was triggered for: Failure - Any Sending email for trigger:
> > > Failure - Any
> > >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For
> > additional commands, e-mail: dev-help@lucene.apache.org
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional
> commands, e-mail: dev-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message