crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gabriel Reid (JIRA)" <>
Subject [jira] [Commented] (CRUNCH-301) Cogrouping tables where RHS has a Scala tuple value type causes duplicated and missing RHS values
Date Thu, 21 Nov 2013 15:09:36 GMT


Gabriel Reid commented on CRUNCH-301:

I was actually just looking at this, and thought it might be better to just have setContext
call setConfiguration with its configuration, and then remove the null checking in getConfiguration().
However, for some reason a lot of the integration tests start throwing OOMEs with this change
in place, which totally baffles me.

In any case, I agree with Micah's point that an explicitly-set Configuration probably makes
more sense -- that being said, the setting of the Configuration happens in different context
than the setting of the context, so I can't think of any situation where it would actually
become an issue.

> Cogrouping tables where RHS has a Scala tuple value type causes duplicated and missing
RHS values
> -------------------------------------------------------------------------------------------------
>                 Key: CRUNCH-301
>                 URL:
>             Project: Crunch
>          Issue Type: Bug
>          Components: Scrunch
>    Affects Versions: 0.8.0
>         Environment: Hadoop 2
>            Reporter: David Whiting
>         Attachments: CRUNCH-301.patch, IsolatedBug.scala
> Suppose you have three record types, Rec1, Rec2 and Rec3.
> Rec1 references Rec2 via key1, and Rec2 references Rec3 (one-to-many) by key2. If you
innerJoin Rec2 and Rec3 to make a PCollection[(Rec2,Rec3)] and they cogroup it against Rec1,
then instead of surfacing n different (Rec2,Rec3) tuples applicable to the Rec1, it surfaces
just one of the (Rec2, Rec3) tuples multiple times.
> This only happens when running with MRPipeline, and not with MemPipeline.
> Attached is the simplest complete program I could come up with which will produce this
unexpected result:
> The result that is produced is:
> Rec1(1,tjena)	Rec1(1,hello)	(Rec2(1,a,0.5),Rec3(a,4))	(Rec2(1,a,0.5),Rec3(a,4))	(Rec2(1,a,0.5),Rec3(a,4))
> Rec1(2,goodbye)	(Rec2(2,c,9.9),Rec3(c,6))
> As you can see, there's a single (Rec2, Rec3) tuple repeated many times, instead of showing
all the distinct ones. This does not happen if you join against Rec2 on its own.

This message was sent by Atlassian JIRA

View raw message