Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of vinodkv@hortonworks.com
 designates 209.85.214.182 as permitted sender)
MIME-Version: 1.0
Sender: vinodkv@hortonworks.com
In-Reply-To: <BEF105A1161A4517AABE021EFBA28055@P200304006390>
References: <BEF105A1161A4517AABE021EFBA28055@P200304006390>
Date: Thu, 24 Apr 2014 10:02:55 -0700
Message-ID: 
 <CAMEEY-+vgNemDfMyGxY8yt4p2c8G5kQrR90ncD2f+FYhzZgWNQ@mail.gmail.com>
Subject: Re: map execute twice
From: Vinod Kumar Vavilapalli <vinodkv@apache.org>
To: "mapreduce-user@hadoop.apache.org" <user@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=089e0158ad3803748304f7ccd252

--089e0158ad3803748304f7ccd252
Content-Type: text/plain; charset=US-ASCII

This can happen when maps are marked as failed *after* they have
successfully completed the map operation. One common reason when this can
happen is reducers faiingl to fetch the map-outputs due to the node that
ran the mapper going down, the machine freezing up etc.

+Vinod
Hortonworks Inc.
http://hortonworks.com/


On Tue, Apr 22, 2014 at 7:28 PM, EdwardKing <zhangsc@neusoft.com> wrote:

>  I use Hadoop 2.2.0, I know hadoop will execute map first,when map is
> 100%, it then execute reduce, after reduce is 100%,job will end. I execute
> a job,the map is from 0% to 100% and map is from 0% to 100% again, why map
> execute twice?  Thanks.
>
> Hadoop job information for Stage-1: number of mappers: 1; number of
> reducers: 1
> 2014-04-22 19:08:49,118 Stage-1 map = 0%,  reduce = 0%
> 2014-04-22 19:11:46,722 Stage-1 map = 100%,  reduce = 0%
> 2014-04-22 19:12:27,633 Stage-1 map = 0%,  reduce = 0%
> 2014-04-22 19:14:37,655 Stage-1 map = 15%,  reduce = 0%, Cumulative CPU
> 1.5 sec
> 2014-04-22 19:15:39,248 Stage-1 map = 33%,  reduce = 0%, Cumulative CPU
> 3.34 sec
> 2014-04-22 19:15:59,395 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU
> 3.34 sec
> 2014-04-22 19:16:40,988 Stage-1 map = 0%,  reduce = 0%
> 2014-04-22 19:18:56,845 Stage-1 map = 11%,  reduce = 0%, Cumulative CPU
> 2.57 sec
> 2014-04-22 19:19:46,574 Stage-1 map = 15%,  reduce = 0%, Cumulative CPU
> 2.73 sec
> 2014-04-22 19:20:30,718 Stage-1 map = 22%,  reduce = 0%, Cumulative CPU
> 2.82 sec
> 2014-04-22 19:20:35,007 Stage-1 map = 41%,  reduce = 0%, Cumulative CPU
> 3.57 sec
> 2014-04-22 19:20:55,280 Stage-1 map = 45%,  reduce = 0%, Cumulative CPU
> 3.76 sec
> 2014-04-22 19:21:27,247 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU
> 4.41 sec
> 2014-04-22 19:22:28,362 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU
> 4.41 sec
> 2014-04-22 19:22:49,170 Stage-1 map = 100%,  reduce = 67%, Cumulative CPU
> 4.41 sec
> 2014-04-22 19:22:52,995 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU
> 5.69 sec
> MapReduce Total cumulative CPU time: 5 seconds 690 msec
> Ended Job = job_1398218615130_0001
>
>
>
> ---------------------------------------------------------------------------------------------------
> Confidentiality Notice: The information contained in this e-mail and any
> accompanying attachment(s)
> is intended only for the use of the intended recipient and may be
> confidential and/or privileged of
> Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader
> of this communication is
> not the intended recipient, unauthorized use, forwarding, printing,
> storing, disclosure or copying
> is strictly prohibited, and may be unlawful.If you have received this
> communication in error,please
> immediately notify the sender by return e-mail, and delete the original
> message and all copies from
> your system. Thank you.
>
> ---------------------------------------------------------------------------------------------------
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

--089e0158ad3803748304f7ccd252
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">This can happen when maps are marked as failed *after* the=
y have successfully completed the map operation. One common reason when thi=
s can happen is reducers faiingl to fetch the map-outputs due to the node t=
hat ran the mapper going down, the machine freezing up etc.<br>
<div class=3D"gmail_extra"><br clear=3D"all"><div><div dir=3D"ltr">+Vinod<b=
r>Hortonworks Inc.<br><a href=3D"http://hortonworks.com/" target=3D"_blank"=
>http://hortonworks.com/</a></div></div>
<br><br><div class=3D"gmail_quote">On Tue, Apr 22, 2014 at 7:28 PM, EdwardK=
ing <span dir=3D"ltr">&lt;<a href=3D"mailto:zhangsc@neusoft.com" target=3D"=
_blank">zhangsc@neusoft.com</a>&gt;</span> wrote:<br><blockquote class=3D"g=
mail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-l=
eft:1ex">
<u></u>


<div bgcolor=3D"#ffffff">
<div><font face=3D"Arial">I use Hadoop 2.2.0, I know hadoop will execute ma=
p=20
first,when map is 100%, it=A0then execute reduce,=A0after=A0reduce is=20
100%,job will end.=A0I execute a job,the map is from 0% to 100% and map is=
=20
from=A00% to 100% again, why map execute twice?=A0 Thanks.</font></div>
<div><font face=3D"Arial"></font>=A0</div><font color=3D"#0000ff" face=3D"A=
rial">Hadoop job information for Stage-1: number of mappers: 1; number of=
=20
reducers: 1<br>2014-04-22 19:08:49,118 Stage-1 map =3D 0%,=A0 reduce =3D=20
0%<br>2014-04-22 19:11:46,722 Stage-1 map =3D 100%,=A0 reduce =3D=20
0%<br>2014-04-22 19:12:27,633 Stage-1 map =3D 0%,=A0 reduce =3D 0%<br>2014-=
04-22=20
19:14:37,655 Stage-1 map =3D 15%,=A0 reduce =3D 0%, Cumulative CPU 1.5=20
sec<br>2014-04-22 19:15:39,248 Stage-1 map =3D 33%,=A0 reduce =3D 0%, Cumul=
ative=20
CPU 3.34 sec<br>2014-04-22 19:15:59,395 Stage-1 map =3D 100%,=A0 reduce =3D=
 0%,=20
Cumulative CPU 3.34 sec<br>2014-04-22 19:16:40,988 Stage-1 map =3D 0%,=A0=
=20
reduce =3D 0%<br>2014-04-22 19:18:56,845 Stage-1 map =3D 11%,=A0 reduce =3D=
 0%,=20
Cumulative CPU 2.57 sec<br>2014-04-22 19:19:46,574 Stage-1 map =3D 15%,=A0=
=20
reduce =3D 0%, Cumulative CPU 2.73 sec<br>2014-04-22 19:20:30,718 Stage-1 m=
ap =3D=20
22%,=A0 reduce =3D 0%, Cumulative CPU 2.82 sec<br>2014-04-22 19:20:35,007=
=20
Stage-1 map =3D 41%,=A0 reduce =3D 0%, Cumulative CPU 3.57 sec<br>2014-04-2=
2=20
19:20:55,280 Stage-1 map =3D 45%,=A0 reduce =3D 0%, Cumulative CPU 3.76=20
sec<br>2014-04-22 19:21:27,247 Stage-1 map =3D 100%,=A0 reduce =3D 0%, Cumu=
lative=20
CPU 4.41 sec<br>2014-04-22 19:22:28,362 Stage-1 map =3D 100%,=A0 reduce =3D=
 0%,=20
Cumulative CPU 4.41 sec<br>2014-04-22 19:22:49,170 Stage-1 map =3D 100%,=A0=
=20
reduce =3D 67%, Cumulative CPU 4.41 sec<br>2014-04-22 19:22:52,995 Stage-1 =
map =3D=20
100%,=A0 reduce =3D 100%, Cumulative CPU 5.69 sec<br>MapReduce Total cumula=
tive=20
CPU time: 5 seconds 690 msec<br>Ended Job =3D job_1398218615130_0001</font>
<div><font face=3D"Arial">=A0</font></div><p>------------------------------=
---------------------------------------------------------------------<br>
Confidentiality Notice: The information contained in this e-mail and any ac=
companying attachment(s) <br>
is intended only for the use of the intended recipient and may be confident=
ial and/or privileged of <br>
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader =
of this communication is <br>
not the intended recipient, unauthorized use, forwarding, printing,=A0 stor=
ing, disclosure or copying <br>
is strictly prohibited, and may be unlawful.If you have received this commu=
nication in error,please <br>
immediately notify the sender by return e-mail, and delete the original mes=
sage and all copies from <br>
your system. Thank you. <br>
---------------------------------------------------------------------------=
------------------------</p></div>
</blockquote></div><br></div></div>

<br>
<span style=3D"color:rgb(128,128,128);font-family:Arial,sans-serif;font-siz=
e:10px">CONFIDENTIALITY NOTICE</span><br style=3D"color:rgb(128,128,128);fo=
nt-family:Arial,sans-serif;font-size:10px"><span style=3D"color:rgb(128,128=
,128);font-family:Arial,sans-serif;font-size:10px">NOTICE: This message is =
intended for the use of the individual or entity to which it is addressed a=
nd may contain information that is confidential, privileged and exempt from=
 disclosure under applicable law. If the reader of this message is not the =
intended recipient, you are hereby notified that any printing, copying, dis=
semination, distribution, disclosure or forwarding of this communication is=
 strictly prohibited. If you have received this communication in error, ple=
ase contact the sender immediately and delete it from your system. Thank Yo=
u.</span>
--089e0158ad3803748304f7ccd252--