lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler" <...@thetaphi.de>
Subject RE: [JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_51) - Build # 9828 - Failure!
Date Wed, 26 Mar 2014 09:01:15 GMT
Hi Dawid,

 

Another idea (only as workaround for now):

On the windows builds, can I force the test runner to nuke the “J0, J1,…” folders although
tests failed – on non-developer computers I see no reason to keep them after failed tests?
Maybe with a sysprop passed to ANT? In that case the tests would produce many files, but the
contents are cleaned up after the ANT task finishes.

 

The whole problem always appears, if Windows tests failed, all Jx folders are kept alive and
then the next run fails. If the Windows test passed before, the Jx folders seem to be cleaned
up and so the following build succeeds.

 

Uwe

 

-----

Uwe Schindler

H.-H.-Meier-Allee 63, D-28213 Bremen

 <http://www.thetaphi.de/> http://www.thetaphi.de

eMail: uwe@thetaphi.de

 

From: dawid.weiss@gmail.com [mailto:dawid.weiss@gmail.com] On Behalf Of Dawid Weiss
Sent: Wednesday, March 26, 2014 9:46 AM
To: dev@lucene.apache.org
Subject: Re: [JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_51) - Build # 9828 - Failure!

 

 

I'll take a look, perhaps I messed something up at the framework level as well. Maybe we should
add a 'clean cwd after suite' rule... although it'd probably make a *lot* of tests fail...
:)


Dawid

 

On Wed, Mar 26, 2014 at 9:40 AM, Uwe Schindler <uwe@thetaphi.de> wrote:

Hi Dawid,
 

The framework correctly removes its own files on successful run, in the case of a failed build:
no (but that’s fine). 

 

The test files should be removed by the test itself (although they may be nuked later by the
test framework removing the “Jx” folders). But, if you look into the logs, you always
see the well-known message with something like: “!!!! WARNING: best effort to remove C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.TestShortCircuitedRequests-1395693845226
FAILED !!!!!”

 

Maybe the order of execution was shanged in test shutdown, so it is no longer possible to
delete the test files after shutdown of test (because still open). I have no idea how Solr
cleans up after its test, but there must definitely somebody have a look. The remaining files
in the folder are also of “successful” test, its just many offenders, keeping the files
(doesn’t matter if they succeed or not).

 

I can reproduce the same on my local machine. When running solr tests, I get horrible amounts
of data in the J0,J1,J2,… subdirectories. Each test creates a full Solr instance dir and
never cleans them up. Some of them do, so you see then disappearing while running test, but
the offenders like listed here in the screenshots stay alive. And those are huge.

 

What is also interesting to me (if looking at the screenshot): Some of the test directories
have older timestamps, but the parent directory was created when the test runs (later!). I
have no idea, *who* changes the last modified date of those folders (maybe happens when unzipping
something and lastmod dates are preserved), but the folders are definitely not created at
the time shown in Windows Explorer. The workspace was definitely correctly nuked before running
the tests – ant clean works!

 

As a quick fix, I can enlarge the virtual disk of this Windows VM, but I am limited because
of SSD capacity, so this won’t help – especially if we get more tests. And I don’t think
we should require something like 4 GiB of disk space to run Solr tests!

 

I will open a bug report to review the test setup and how the cleanup is done in afterClass
for Solr tests. Lucene always removes all files, there are never any relicts.

 

Uwe

 

-----

Uwe Schindler

H.-H.-Meier-Allee 63, D-28213 Bremen

 <http://www.thetaphi.de/> http://www.thetaphi.de

eMail: uwe@thetaphi.de

 

From: dawid.weiss@gmail.com [mailto:dawid.weiss@gmail.com] On Behalf Of Dawid Weiss
Sent: Wednesday, March 26, 2014 8:59 AM


To: dev@lucene.apache.org
Subject: Re: [JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_51) - Build # 9828 - Failure!

 

 

The framework itself should clean up its own directories and files but I don't think it removes
test files. It definitely doesn't remove anything

in case of an unsuccessful build -- did those builds pass?


Dawid

 

On Wed, Mar 26, 2014 at 1:47 AM, Uwe Schindler <uwe@thetaphi.de> wrote:

Hi,

 

the windows VM again ran out of disk space a minute ago.

 

The Windows VM had initially approx. 8 GB free space. After running both workspaces (4.x and
trunk), the Solr-core work folder used approx. 3 to 4 GiB each. Looking into the directory,
it looks like (maybe that is a windows problem), nothing is cleaned up in the JUnit runner
directory. I was expecting that tests clean up after running, which seems no longer be the
case.

 

Trunk:

 



 

 

4.x

 



-----

Uwe Schindler

H.-H.-Meier-Allee 63, D-28213 Bremen

http://www.thetaphi.de

eMail: uwe@thetaphi.de

 

 

> -----Original Message-----

> From: dawid.weiss@gmail.com [mailto:dawid.weiss@gmail.com] On Behalf

> Of Dawid Weiss

> Sent: Tuesday, March 25, 2014 9:47 AM

> To: dev@lucene.apache.org

> Subject: Re: [JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_51) - Build #

> 9828 - Failure!

> 

> These tests are loooong, so theoretically we could add a guard thread at the

> suite level that would watch for disk capacity thresholds...

> but it still seems like an overkill to me.

> 

> Dawid

> 

> On Tue, Mar 25, 2014 at 9:36 AM, Uwe Schindler < <mailto:uwe@thetaphi.de> uwe@thetaphi.de>
wrote:

> > Hi,

> >

> > I also analyzed this specific events file. The events file looks quite fine, no

> endless logging in this case. It was also only 95 MB (quite normal). So it looks

> like something else filled the disk space, so checking size of event file is not

> really useful (only for the last case). After reverting the virtual box to the

> clean snapshot it had 9 GB free. So something must fill the disk (e.g. indexes

> other data). If this happens again, I will do a complete analysis of whet is

> there to find in workspace. I only know, that all the 9 GB of space are inside

> the Jenkins Workspace folder, so nothing outside fills the disk (like windows

> itsself).

> >

> > I will think about it a bit more.

> >

> > -----

> > Uwe Schindler

> > H.-H.-Meier-Allee 63, D-28213 Bremen

> >  <http://www.thetaphi.de> http://www.thetaphi.de

> > eMail:  <mailto:uwe@thetaphi.de> uwe@thetaphi.de

> >

> >

> >> -----Original Message-----

> >> From:  <mailto:dawid.weiss@gmail.com> dawid.weiss@gmail.com [ <mailto:dawid.weiss@gmail.com>
mailto:dawid.weiss@gmail.com] On

> Behalf

> >> Of Dawid Weiss

> >> Sent: Tuesday, March 25, 2014 9:25 AM

> >> To:  <mailto:dev@lucene.apache.org> dev@lucene.apache.org

> >> Cc: Mark Miller

> >> Subject: Re: [JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_51) -

> >> Build #

> >> 9828 - Failure!

> >>

> >> > Would it be possible to catch those cases while running tests

> >> > (maybe before the disk is full) and fail the build? Maybe something

> >> > that the event file is not allowed to grow beyond a specific size.

> >> > If it grows, the test framework fails the whole build? We can have

> >> > something like maximum size of 1 GB (configureable).

> >>

> >> I honestly think this is trying to cater for an insane specific

> >> scenario of a faulty test. Think of it: a single test that logs gigs

> >> to disk... Guarding against it may be next to impossible at the test

> >> framework level. We can put a condition in ant that checks for remaining

> temp space and fails if it's less than 5gb...

> >>

> >> Dawid

> >>

> >> ---------------------------------------------------------------------

> >> To unsubscribe, e-mail:  <mailto:dev-unsubscribe@lucene.apache.org> dev-unsubscribe@lucene.apache.org
For

> >> additional commands, e-mail:  <mailto:dev-help@lucene.apache.org> dev-help@lucene.apache.org

> >

> >

> > ---------------------------------------------------------------------

> > To unsubscribe, e-mail:  <mailto:dev-unsubscribe@lucene.apache.org> dev-unsubscribe@lucene.apache.org
For

> > additional commands, e-mail:  <mailto:dev-help@lucene.apache.org> dev-help@lucene.apache.org

> >

> 

> ---------------------------------------------------------------------

> To unsubscribe, e-mail:  <mailto:dev-unsubscribe@lucene.apache.org> dev-unsubscribe@lucene.apache.org
For additional

> commands, e-mail:  <mailto:dev-help@lucene.apache.org> dev-help@lucene.apache.org

 

 


Mime
View raw message