hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Haibo Chen <haiboc...@cloudera.com>
Subject Re: What is the class that launches the reducers?
Date Thu, 25 Aug 2016 23:36:19 GMT
Mappers and reducers that are all scheduled when the job starts running.
Don't think there is a knob to suspend the job once all mappers finish.

On Thu, Aug 25, 2016 at 4:10 PM, xeon Mailinglist <xeonmailinglist@gmail.com
> wrote:

> But then I need to set identity maps to run the reducers. If I suspend a
> job after the maps finish, I don't need to set identity maps up. I want to
> suspend a job so that I don't run identity maps and get better performance.
>
> On Aug 25, 2016 10:12 PM, "Haibo Chen" <haibochen@cloudera.com> wrote:
>
> One thing you can try is to write a map-only job first and then verify the
> map out.
>
> On Thu, Aug 25, 2016 at 1:18 PM, xeon Mailinglist <
> xeonmailinglist@gmail.com> wrote:
>
>> I am using Mapreduce v2.
>>
>> On Aug 25, 2016 8:18 PM, "xeon Mailinglist" <xeonmailinglist@gmail.com>
>> wrote:
>>
>> > I am trying to implement a mechanism in MapReduce v2 that allows to
>> > suspend and resume a job. I must suspend a job when all the mappers
>> finish,
>> > and resume the job from that point after some time. I do this, because I
>> > want to verify the integrity of the map output before executing the
>> > reducers.
>> >
>> > I am looking for the class that tells when the Reduce tasks should
>> start.
>> > Does anyone know where is this?
>> >
>>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message