crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Wills <jwi...@cloudera.com>
Subject Re: [VOTE] Release Apache Crunch 0.8.0 RC0
Date Thu, 07 Nov 2013 16:27:38 GMT
That does sound plausible-- I actually just ran into that issue on another
project I'm working on (i.e., like yesterday.)


On Thu, Nov 7, 2013 at 8:23 AM, Gabriel Reid <gabriel.reid@gmail.com> wrote:

> On Thu, Nov 7, 2013 at 4:13 PM, Chao Shi <stepinto@live.com> wrote:
> > I think this failure may depend on the execution order of tests, because:
> >
> > 1) I can reproduce this failure on some source tree, but cannot on
> another
> > 2) I can only reproduce it with "mvn install", but "mvn test
> > -Dtest=org.apache.crunch.CheckpointIT" cannot
> > 3) I was trying to add some printf in the case, after saving the file (I
> > modified that CheckpointIT.java), I cannot reproduce it now.
> >
> > Does anyone have some clue?
>
> I have a feeling that this isn't due to order of execution. If it is
> down to that, then as far as I can see it would mean that there is
> something wrong with the TemporaryPath class, as that seems to be the
> only thing that is shared between tests in that class.
>
> I'm actually wondering if this could be down to a timing issue in the
> checkpointing code. The FileTargetImpl class checks if the
> lastModified timestamp of the checkpointed file is greater than the
> lastModified timestamp on the input source. Based on a check I just
> did locally here on Mac OS X and on a Linux machine with ext4, the
> time granularity of the lastModified timestamp on files is only
> second-based, not millisecond-based. I'm thinking that this could be
> an issue of the checkpointed file being created less than a second
> after the startup of the test and creation of the source file
> (although that seems really quick), and so the checkpointing wouldn't
> think that the source file is older than the checkpointed file, so it
> would run the full pipeline again (therefore resulting in two
> execution stages instead of one.
>
> Does that sound plausible? If so, the only thing I can see that would
> keep the code correct and prevent the test from failing would be to
> add a Thread.sleep() into the test, but I hate having stuff like that
> in tests.
>
> - Gabriel
>



-- 
Director of Data Science
Cloudera <http://www.cloudera.com>
Twitter: @josh_wills <http://twitter.com/josh_wills>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message