crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthias Friedrich <m...@mafr.de>
Subject Re: Exceptions in integration test cases
Date Wed, 03 Apr 2013 06:38:13 GMT
Fantastic, Gabriel. Thanks a lot!

Regards,
  Matthias

On Tuesday, 2013-04-02, Gabriel Reid wrote:
> 
> On 02 Apr 2013, at 21:39, Josh Wills <josh.wills@gmail.com> wrote:
> 
> > In the context of that test (PCollectionGetSizeIT), the EOFException is
> > expected-- there isn't any data to read b/c the file is empty. We could
> > change up the impl to throw a CrunchRuntimeException instead.
> > 
> > For the cogroup test, that sounds suspiciously like a deep copy issue.
> > Gabriel is the most knowledgable person to ask, he always fixes my deep
> > copy errors. :)
> 
> Well that's a bizarre coincidence -- I just encountered this object reuse bug in CoGroup
earlier today at work, and was going over mails from today while running the integration tests
after having written a patch for it :-)
> 
> In any case, yes, there is a deep copy issue with CoGroup -- I'm actually very surprised
that this is still in there, as I was sure that I had gone over CoGroup in the past to make
sure that object reuse wasn't an issue.
> 
> In any case, I'll commit the fix shortly.
> 
> - Gabriel
> 
> 
> > On Tue, Apr 2, 2013 at 10:33 AM, Matthias Friedrich <matt@mafr.de> wrote:
> > 
> >> Hi,
> >> 
> >> I'm a bit worried about the IO exceptions we get in our integration
> >> test (see [1], grep for EOFException). Since we don't validate the
> >> results, there may be data loss, at least in local mode.
> >> 
> >> I have stumbled upon a test case [2] where Crunch seems to reliably
> >> lose some data when using Writables, while the same code works fine
> >> with Avros. This time, I was able to reproduce this on two machines
> >> (with slightly different error messages, ie. [3]).
> >> 
> >> Has anyone seen this before? Is anyone able to reproduce the error
> >> or is it just me? :)
> >> 
> >> Thanks,
> >>  Matthias
> >> 
> >> [1]
> >> https://builds.apache.org/job/Crunch-master-integration/244/consoleFull
> >> [2] http://users.mafr.de/~matthias/crunch/CogroupIT.patch
> >> [3] http://users.mafr.de/~matthias/crunch/CogroupIT-error.txt
> >> 
> 

Mime
View raw message