crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gabriel Reid <gabriel.r...@gmail.com>
Subject Re: [VOTE] Release Apache Crunch 0.8.0 RC0
Date Thu, 07 Nov 2013 16:23:29 GMT
On Thu, Nov 7, 2013 at 4:13 PM, Chao Shi <stepinto@live.com> wrote:
> I think this failure may depend on the execution order of tests, because:
>
> 1) I can reproduce this failure on some source tree, but cannot on another
> 2) I can only reproduce it with "mvn install", but "mvn test
> -Dtest=org.apache.crunch.CheckpointIT" cannot
> 3) I was trying to add some printf in the case, after saving the file (I
> modified that CheckpointIT.java), I cannot reproduce it now.
>
> Does anyone have some clue?

I have a feeling that this isn't due to order of execution. If it is
down to that, then as far as I can see it would mean that there is
something wrong with the TemporaryPath class, as that seems to be the
only thing that is shared between tests in that class.

I'm actually wondering if this could be down to a timing issue in the
checkpointing code. The FileTargetImpl class checks if the
lastModified timestamp of the checkpointed file is greater than the
lastModified timestamp on the input source. Based on a check I just
did locally here on Mac OS X and on a Linux machine with ext4, the
time granularity of the lastModified timestamp on files is only
second-based, not millisecond-based. I'm thinking that this could be
an issue of the checkpointed file being created less than a second
after the startup of the test and creation of the source file
(although that seems really quick), and so the checkpointing wouldn't
think that the source file is older than the checkpointed file, so it
would run the full pipeline again (therefore resulting in two
execution stages instead of one.

Does that sound plausible? If so, the only thing I can see that would
keep the code correct and prevent the test from failing would be to
add a Thread.sleep() into the test, but I hate having stuff like that
in tests.

- Gabriel

Mime
View raw message