hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vinod Kumar Vavilapalli <vino...@apache.org>
Subject Re: map execute twice
Date Thu, 24 Apr 2014 17:02:55 GMT
This can happen when maps are marked as failed *after* they have
successfully completed the map operation. One common reason when this can
happen is reducers faiingl to fetch the map-outputs due to the node that
ran the mapper going down, the machine freezing up etc.

+Vinod
Hortonworks Inc.
http://hortonworks.com/


On Tue, Apr 22, 2014 at 7:28 PM, EdwardKing <zhangsc@neusoft.com> wrote:

>  I use Hadoop 2.2.0, I know hadoop will execute map first,when map is
> 100%, it then execute reduce, after reduce is 100%,job will end. I execute
> a job,the map is from 0% to 100% and map is from 0% to 100% again, why map
> execute twice?  Thanks.
>
> Hadoop job information for Stage-1: number of mappers: 1; number of
> reducers: 1
> 2014-04-22 19:08:49,118 Stage-1 map = 0%,  reduce = 0%
> 2014-04-22 19:11:46,722 Stage-1 map = 100%,  reduce = 0%
> 2014-04-22 19:12:27,633 Stage-1 map = 0%,  reduce = 0%
> 2014-04-22 19:14:37,655 Stage-1 map = 15%,  reduce = 0%, Cumulative CPU
> 1.5 sec
> 2014-04-22 19:15:39,248 Stage-1 map = 33%,  reduce = 0%, Cumulative CPU
> 3.34 sec
> 2014-04-22 19:15:59,395 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU
> 3.34 sec
> 2014-04-22 19:16:40,988 Stage-1 map = 0%,  reduce = 0%
> 2014-04-22 19:18:56,845 Stage-1 map = 11%,  reduce = 0%, Cumulative CPU
> 2.57 sec
> 2014-04-22 19:19:46,574 Stage-1 map = 15%,  reduce = 0%, Cumulative CPU
> 2.73 sec
> 2014-04-22 19:20:30,718 Stage-1 map = 22%,  reduce = 0%, Cumulative CPU
> 2.82 sec
> 2014-04-22 19:20:35,007 Stage-1 map = 41%,  reduce = 0%, Cumulative CPU
> 3.57 sec
> 2014-04-22 19:20:55,280 Stage-1 map = 45%,  reduce = 0%, Cumulative CPU
> 3.76 sec
> 2014-04-22 19:21:27,247 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU
> 4.41 sec
> 2014-04-22 19:22:28,362 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU
> 4.41 sec
> 2014-04-22 19:22:49,170 Stage-1 map = 100%,  reduce = 67%, Cumulative CPU
> 4.41 sec
> 2014-04-22 19:22:52,995 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU
> 5.69 sec
> MapReduce Total cumulative CPU time: 5 seconds 690 msec
> Ended Job = job_1398218615130_0001
>
>
>
> ---------------------------------------------------------------------------------------------------
> Confidentiality Notice: The information contained in this e-mail and any
> accompanying attachment(s)
> is intended only for the use of the intended recipient and may be
> confidential and/or privileged of
> Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader
> of this communication is
> not the intended recipient, unauthorized use, forwarding, printing,
> storing, disclosure or copying
> is strictly prohibited, and may be unlawful.If you have received this
> communication in error,please
> immediately notify the sender by return e-mail, and delete the original
> message and all copies from
> your system. Thank you.
>
> ---------------------------------------------------------------------------------------------------
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
View raw message