hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vinod Kumar Vavilapalli <vino...@hortonworks.com>
Subject Re: "attempt*" directories in user logs
Date Mon, 10 Dec 2012 18:49:49 GMT

MR launches multiple attempts for single Task in case of TaskAttempt failures or when speculative
execution is turned on. In either case, a given Task will only ever have one successful TaskAttempt
whose output will be accepted (committed).

Number of reduces is set to 1 by default in mapred-default.xml - you should explicitly set
it to zero if you don't want reducers.

By master, I suppose you mean JobTracker. JobTracker doesn't show all the attempts for a given
Task, you should navigate to per-task page to see that.


Thanks,
+Vinod Kumar Vavilapalli
Hortonworks Inc.
http://hortonworks.com/

On Dec 9, 2012, at 6:53 AM, Oleg Zhurakousky wrote:

> I studying user logs on the two node cluster that I have setup and I was wondering if
anyone can shed some light on these "attempt*' directories
>> $ ls
> attempt_201212051224_0021_m_000000_0  attempt_201212051224_0021_m_000003_0  job-acls.xml
> attempt_201212051224_0021_m_000002_0  attempt_201212051224_0021_r_000000_0
> 
> I mean its obvious that its talking about 3 attempts for Map task and 1 attempt for reduce
task. However my current MR job only results in some output written to "attempt_201212051224_0021_m_000000_0".
Nothing is the reduce part (understandably since I don't even have a reducer, so my question
is:
> 
> 1. The two more M attempts. . . what are they?
> 2. Why was there an attempt to do a Reduce when no reducer was provided.implemented
> 3. Why my master node only had 1 attempt for M task but the slave had all that's displayed
and questioned above (the 'ls' output above is from the slave node)
> 
> Thanks
> Oleg


Mime
View raw message