crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gabriel Reid (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CRUNCH-455) Sort.sort doesn't work with ReverseAvroComparator in MemPipeline
Date Mon, 11 Aug 2014 20:16:12 GMT

    [ https://issues.apache.org/jira/browse/CRUNCH-455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093231#comment-14093231
] 

Gabriel Reid commented on CRUNCH-455:
-------------------------------------

I just ran a little benchmark with the before and after version of this patch on a 5-node
cluster with a word count on 2 billion records, and there was no time difference between the
two. Actually, thinking about it a bit more, this makes more sense than I thought it would,
as the objects being created and discarded (AvroKeys) are super lightweight and just a wrapper
around the real value.

Anyhow, I'm now officially not worried about object reuse and +1 on this patch.

> Sort.sort doesn't work with ReverseAvroComparator in MemPipeline
> ----------------------------------------------------------------
>
>                 Key: CRUNCH-455
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-455
>             Project: Crunch
>          Issue Type: Bug
>          Components: Core
>            Reporter: David Whiting
>            Assignee: Josh Wills
>            Priority: Minor
>         Attachments: CRUNCH-455.patch
>
>
> The mem Shuffler class discards the config that arrives with the GroupingOptions and
only uses the unmodified Conifguration from the pipeline object, which means that "crunch.schema"
is not set and causes a NullPointerException when you try and execute it.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message