Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of vinodkv@hortonworks.com
 designates 209.85.210.48 as permitted sender)
From: Vinod Kumar Vavilapalli <vinodkv@hortonworks.com>
Mime-Version: 1.0 (Apple Message framework v1278)
Content-Type: multipart/signed;
 boundary="Apple-Mail=_F5C2228F-011A-488B-A6B4-18D116401754";
 protocol="application/pgp-signature"; micalg=pgp-sha1
Subject: Re: "attempt*" directories in user logs
Date: Mon, 10 Dec 2012 10:49:49 -0800
In-Reply-To: <1DD6A29F-062E-4E40-BAF4-964BFDDC2E71@gmail.com>
To: user@hadoop.apache.org
References: 
 <CAMt-yjC+63Y91BF534TOVuzXSvYYeLgZ=t_iDJ0Mxg0rGi5OpA@mail.gmail.com>
 <CAOcnVr272=dGdRqau5jcqcTPsy5kTdk=Ba-aLbVqBhSWdynMsg@mail.gmail.com>
 <CAO3ErG_xDELEVKzkSVaAds+8Qpqwr2Aji-E8LPNpT+k4dUyWdg@mail.gmail.com>
 <343176F5-0EF6-4C64-A4EB-7C0B5ADF4670@gmail.com>
 <CAO6JcpjxXkhys4_QpwiB5O+PJpm_P-aix2WPtSVAuC2fyHYiPg@mail.gmail.com>
 <1DD6A29F-062E-4E40-BAF4-964BFDDC2E71@gmail.com>
Message-Id: <244C4BD5-E0C4-421C-8B19-9D32CB4CDB85@apache.org>


--Apple-Mail=_F5C2228F-011A-488B-A6B4-18D116401754
Content-Type: multipart/alternative;
	boundary="Apple-Mail=_74798E79-97B0-41A8-9CD0-467200232402"


--Apple-Mail=_74798E79-97B0-41A8-9CD0-467200232402
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=iso-8859-1


MR launches multiple attempts for single Task in case of TaskAttempt =
failures or when speculative execution is turned on. In either case, a =
given Task will only ever have one successful TaskAttempt whose output =
will be accepted (committed).

Number of reduces is set to 1 by default in mapred-default.xml - you =
should explicitly set it to zero if you don't want reducers.

By master, I suppose you mean JobTracker. JobTracker doesn't show all =
the attempts for a given Task, you should navigate to per-task page to =
see that.


Thanks,
+Vinod Kumar Vavilapalli
Hortonworks Inc.
http://hortonworks.com/

On Dec 9, 2012, at 6:53 AM, Oleg Zhurakousky wrote:

> I studying user logs on the two node cluster that I have setup and I =
was wondering if anyone can shed some light on these "attempt*' =
directories
>> $ ls
> attempt_201212051224_0021_m_000000_0  =
attempt_201212051224_0021_m_000003_0  job-acls.xml
> attempt_201212051224_0021_m_000002_0  =
attempt_201212051224_0021_r_000000_0
>=20
> I mean its obvious that its talking about 3 attempts for Map task and =
1 attempt for reduce task. However my current MR job only results in =
some output written to "attempt_201212051224_0021_m_000000_0". Nothing =
is the reduce part (understandably since I don't even have a reducer, so =
my question is:
>=20
> 1. The two more M attempts. . . what are they?
> 2. Why was there an attempt to do a Reduce when no reducer was =
provided.implemented
> 3. Why my master node only had 1 attempt for M task but the slave had =
all that's displayed and questioned above (the 'ls' output above is from =
the slave node)
>=20
> Thanks
> Oleg


--Apple-Mail=_74798E79-97B0-41A8-9CD0-467200232402
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=iso-8859-1

<html><head></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space; =
"><div><br></div><div>MR launches multiple attempts for single Task in =
case of TaskAttempt failures or when speculative execution is turned on. =
In either case, a given Task will only ever have one successful =
TaskAttempt whose output will be accepted =
(committed).</div><div><br></div><div>Number of reduces is set to 1 by =
default in mapred-default.xml - you should explicitly set it to zero if =
you don't want reducers.</div><div><br></div><div>By master, I suppose =
you mean JobTracker. JobTracker doesn't show all the attempts for a =
given Task, you should navigate to per-task page to see =
that.</div><div><br></div><br><div>
<span class=3D"Apple-style-span" style=3D"border-collapse: separate; =
color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; =
font-variant: normal; font-weight: normal; letter-spacing: normal; =
line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: =
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: =
0px; -webkit-border-horizontal-spacing: 0px; =
-webkit-border-vertical-spacing: 0px; =
-webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: =
auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div><div =
class=3D"">Thanks,</div><div class=3D"">+Vinod Kumar =
Vavilapalli</div><div class=3D"">Hortonworks Inc.<br class=3D""><a =
href=3D"http://hortonworks.com/" =
class=3D"">http://hortonworks.com/</a></div></div></span>
</div>

<br><div><div>On Dec 9, 2012, at 6:53 AM, Oleg Zhurakousky =
wrote:</div><br class=3D"Apple-interchange-newline"><blockquote =
type=3D"cite"><div>I studying user logs on the two node cluster that I =
have setup and I was wondering if anyone can shed some light on these =
"attempt*' directories<br><blockquote type=3D"cite">$ =
ls<br></blockquote>attempt_201212051224_0021_m_000000_0 =
&nbsp;attempt_201212051224_0021_m_000003_0 =
&nbsp;job-acls.xml<br>attempt_201212051224_0021_m_000002_0 =
&nbsp;attempt_201212051224_0021_r_000000_0<br><br>I mean its obvious =
that its talking about 3 attempts for Map task and 1 attempt for reduce =
task. However my current MR job only results in some output written to =
"attempt_201212051224_0021_m_000000_0". Nothing is the reduce part =
(understandably since I don't even have a reducer, so my question =
is:<br><br>1. The two more M attempts. . . what are they?<br>2. Why was =
there an attempt to do a Reduce when no reducer was =
provided.implemented<br>3. Why my master node only had 1 attempt for M =
task but the slave had all that's displayed and questioned above (the =
'ls' output above is from the slave =
node)<br><br>Thanks<br>Oleg<br></div></blockquote></div><br></body></html>=

--Apple-Mail=_74798E79-97B0-41A8-9CD0-467200232402--

--Apple-Mail=_F5C2228F-011A-488B-A6B4-18D116401754
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
	filename=signature.asc
Content-Type: application/pgp-signature;
	name=signature.asc
Content-Description: Message signed with OpenPGP using GPGMail

-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)

iQEcBAEBAgAGBQJQxi7NAAoJECVa31bDbF8PmyYH/3zpwxZaSkUfuU3T/D7FiiT+
nIYt2mskm+AP3MvM1ECNWMP2GEnpvbnRXVxa1E5ZPWVN4BppT+GSB72qoE9Xgsf5
S+Mj9YHBG3/qo19Ltk3q//iydF6LLI2z/90npLaOu7Ya2td7N1Y3E+0SMdQ7HlRs
MX4Afiuz9V5T25dfPuUGAsx2y5fuZFOBxdTsPr96rAogmCv+bs4MJB5jlQwmFHrb
OrnwKWy8w2qtEYN/M5Co4FXu5zPPEQm0mO54jPq/5THIIHs+X2L0GjJpeNg/CNtg
muqUdSbN5LCeepOPq8pmOG98HpVR2teq5WoBZPS2PcBxZtYgQhKiObtnXP7rJ4c=
=0Y7R
-----END PGP SIGNATURE-----

--Apple-Mail=_F5C2228F-011A-488B-A6B4-18D116401754--