crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tycho Lamerigts (JIRA)" <>
Subject [jira] [Commented] (CRUNCH-485) groupByKey on Spark incorrect if key is Avro record with defined sort order
Date Thu, 08 Jan 2015 08:42:35 GMT


Tycho Lamerigts commented on CRUNCH-485:

It looks like it could work to me. I didn't run it. And it's not that ugly either, you can't
help it that org.apache.avro.Schema is not already Serializable. :-) I don't know this project's
process - do you need anything more from me, or do I now just wait until it makes it into
0.12 or so?

> groupByKey on Spark incorrect if key is Avro record with defined sort order
> ---------------------------------------------------------------------------
>                 Key: CRUNCH-485
>                 URL:
>             Project: Crunch
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.11.0
>            Reporter: Tycho Lamerigts
>            Assignee: Josh Wills
>         Attachments: CRUNCH-485.patch
> GroupByKey on Spark is incorrect if the key type is an Avro record with defined sort
order (
> Instead, it serializes the entire avro record to a binary blob (byte array) and groups
identical blobs. This is wrong. By contrast, groupByKey on MapReduce works as expected, so
it does take Avro's sort order into account.
> The culprit is probably the following code from org.apache.crunch.impl.spark.collect.PGroupedTableImpl#getJavaRDDLikeInternal
> {code}
> groupedRDD = PairMapFunction(ptype.getOutputMapFn(), runtime.getRuntimeContext()))
>           .mapToPair(new MapOutputFunction(keySerde, valueSerde))
>           .groupByKey(numPartitions);
> {code}
> where MapOutputFunction simply converts the entire key object to a binary blob, without
taking sort order into account.

This message was sent by Atlassian JIRA

View raw message