crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gabriel Reid (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CRUNCH-455) Sort.sort doesn't work with ReverseAvroComparator in MemPipeline
Date Mon, 11 Aug 2014 18:38:12 GMT

    [ https://issues.apache.org/jira/browse/CRUNCH-455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093121#comment-14093121
] 

Gabriel Reid commented on CRUNCH-455:
-------------------------------------

The removal of the re-use of the single AvroKey instance AvroKeyConverter worries me a (really
tiny) little bit, but that's just me fearing the unknown. I'd love to try some benchmarking
to see if this makes a real performance difference at all -- I'm pretty convinced it doesn't,
but the fact that this stuff is done everywhere in MR code makes me wonder.

I was looking at Shuffler to see if there's any other way to do it, but the only thing I see
is having a special case for handling Avro where we make a copy of the AvroKey object before
putting it in the map, which wouldn't be too cool either.

> Sort.sort doesn't work with ReverseAvroComparator in MemPipeline
> ----------------------------------------------------------------
>
>                 Key: CRUNCH-455
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-455
>             Project: Crunch
>          Issue Type: Bug
>          Components: Core
>            Reporter: David Whiting
>            Assignee: Josh Wills
>            Priority: Minor
>         Attachments: CRUNCH-455.patch
>
>
> The mem Shuffler class discards the config that arrives with the GroupingOptions and
only uses the unmodified Conifguration from the pipeline object, which means that "crunch.schema"
is not set and causes a NullPointerException when you try and execute it.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message