hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: How to make different mappers execute different processing on a same data ?
Date Tue, 11 Sep 2012 03:50:07 GMT
Hey Jason,

While I am not sure on whats the best way to automatically "evaluate"
during the execution of a job, the MultipleInputs class offers a way
to run different map implementations within a single job for different
input paths. You could perhaps leverage that with duplicated (or
symlinked?) input paths.

Otherwise, perhaps do all the N types of computation in a single map()
call, and judge the time inside it at the end of all, before emitting?

On Tue, Sep 11, 2012 at 9:03 AM, Jason Yang <lin.yang.jason@gmail.com> wrote:
> Hi, all
>
> I've got a question about how to make different mappers execute different
> processing on a same data?
>
> Here is my scenario:
> I got to process a data, however, there multiple choices to process this
> data and I have no idea which one is better, so I was thinking that maybe I
> could execute multiple mappers, in which different processing solution is
> applied, and eventually the best one is chosen according to some evaluation
> functions.
>
> But I'm not sure whether this could be done in MapReduce.
>
> Any help would be appreciated.
>
> --
> YANG, Lin
>



-- 
Harsh J

Mime
View raw message