crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Wills (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CRUNCH-231) Support legacy Mappers and Reducers in Crunch pipelines
Date Mon, 01 Jul 2013 12:47:21 GMT

    [ https://issues.apache.org/jira/browse/CRUNCH-231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13696780#comment-13696780
] 

Josh Wills commented on CRUNCH-231:
-----------------------------------

I understand your initial reaction-- this isn't the sort of thing that I would ever want to
put into the top-level APIs. But I think that lots of developers get started with Hadoop by
writing against the low-level APIs and end up tweaking and adding things until they end up
with a bit of a mess. As a developer, I empathize with their desire to get themselves out
of it via incremental refactoring as opposed to rewriting the entire thing from scratch, and
I think that using their existing mappers and reducers in conjunction w/Crunch's various test
support utilities can help them do just that. 
                
> Support legacy Mappers and Reducers in Crunch pipelines
> -------------------------------------------------------
>
>                 Key: CRUNCH-231
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-231
>             Project: Crunch
>          Issue Type: Improvement
>          Components: Core
>            Reporter: Josh Wills
>            Assignee: Josh Wills
>         Attachments: mapred.patch
>
>
> I've had a few requests for Crunch to support existing Mappers and Reducers using the
underlying Java APIs as part of regular pipelines, so that users could evolve existing MapReduce
jobs into Crunch pipelines gradually, instead of being forced to rewrite everything all at
once in order to map it onto Crunch's model.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message