crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dave Beech (JIRA)" <>
Subject [jira] [Commented] (CRUNCH-231) Support legacy Mappers and Reducers in Crunch pipelines
Date Sun, 30 Jun 2013 20:43:20 GMT


Dave Beech commented on CRUNCH-231:

My first instinct is not to like this! For me, the higher-level "pipeline" model is much cleaner
and nicer to work with than standard mapreduce. Also, mixing different levels of abstraction
in one codebase feels like a bad thing to be doing. 
Having said all that, I can see the appeal of being able to gradually migrate old code across
in terms of lowering the barrier to entry and therefore growing the userbase of Crunch. 

>From a quick glance at the patch it looks like just writables are supported. I guess the
levels of crazy hacking would need to be stepped up to support Avro -- especially Avro mapred
rather than mapreduce!

I'll try and give the patch a go early this week and come back with some more comments.
> Support legacy Mappers and Reducers in Crunch pipelines
> -------------------------------------------------------
>                 Key: CRUNCH-231
>                 URL:
>             Project: Crunch
>          Issue Type: Improvement
>          Components: Core
>            Reporter: Josh Wills
>            Assignee: Josh Wills
>         Attachments: mapred.patch
> I've had a few requests for Crunch to support existing Mappers and Reducers using the
underlying Java APIs as part of regular pipelines, so that users could evolve existing MapReduce
jobs into Crunch pipelines gradually, instead of being forced to rewrite everything all at
once in order to map it onto Crunch's model.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message