crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Wills <josh.wi...@gmail.com>
Subject Re: Exceptions in integration test cases
Date Tue, 02 Apr 2013 19:39:13 GMT
In the context of that test (PCollectionGetSizeIT), the EOFException is
expected-- there isn't any data to read b/c the file is empty. We could
change up the impl to throw a CrunchRuntimeException instead.

For the cogroup test, that sounds suspiciously like a deep copy issue.
Gabriel is the most knowledgable person to ask, he always fixes my deep
copy errors. :)

Josh


On Tue, Apr 2, 2013 at 10:33 AM, Matthias Friedrich <matt@mafr.de> wrote:

> Hi,
>
> I'm a bit worried about the IO exceptions we get in our integration
> test (see [1], grep for EOFException). Since we don't validate the
> results, there may be data loss, at least in local mode.
>
> I have stumbled upon a test case [2] where Crunch seems to reliably
> lose some data when using Writables, while the same code works fine
> with Avros. This time, I was able to reproduce this on two machines
> (with slightly different error messages, ie. [3]).
>
> Has anyone seen this before? Is anyone able to reproduce the error
> or is it just me? :)
>
> Thanks,
>   Matthias
>
> [1]
> https://builds.apache.org/job/Crunch-master-integration/244/consoleFull
> [2] http://users.mafr.de/~matthias/crunch/CogroupIT.patch
> [3] http://users.mafr.de/~matthias/crunch/CogroupIT-error.txt
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message