hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "hailong.yang1115" <hailong.yang1...@gmail.com>
Subject Re: Re: Re: measure the time taken to complete map and reduce phase
Date Fri, 08 Jul 2011 05:34:43 GMT

I think the TOTAL_MAPS has the same meaning with the FINISHED_MAPS, which represents the total
number of map tasks successfully executed. Although there is another metric named Launched_map_tasks,
which is the sum of  FINISHED_MAPS, FAILED_MAPS and KILLED_MAPS.

And it is the same for Reduce tasks.


Cheer!

Hailong

2011-07-08 



***********************************************
* Hailong Yang, PhD. Candidate 
* Sino-German Joint Software Institute, 
* School of Computer Science&Engineering, Beihang University
* Phone: (86-010)82315908
* Email: hailong.yang1115@gmail.com
* Address: G413, New Main Building in Beihang University, 
*              No.37 XueYuan Road,HaiDian District, 
*              Beijing,P.R.China,100191
***********************************************



发件人: sangroya 
发送时间: 2011-07-07  23:50:45 
收件人: hadoop-user 
抄送: 
主题: Re: Re: measure the time taken to complete map and reduce phase 
 
Hi,
Thanks for the response!
I have the following queries regarding the Job History file.
I want to know if what the TOTAL_MAPS in the Job history represents.
Also, if FINISHED_MAPS represents the TOTAL_MAPS or the (TOTAL_MAPS -
FAILED_MAPS).
Does FINISHED_MAPS represents successfully executed maps.
I have the same question for REDUCE tasks.
Thanks,
Amit
On Thu, Jul 7, 2011 at 10:58 AM, Hailong [via Lucene]
<ml-node+3147566-1425016666-409464@n3.nabble.com> wrote:
> Hi sangroya,
>
> I think you may be interested in reading the following piece of code from
> JobHistory.java in Hadoop.
>
>     /**
>      * Generates the job history filename for a new job
>      */
>     private static String getNewJobHistoryFileName(JobConf jobConf, JobID
> id) {
>       return JOBTRACKER_UNIQUE_STRING
>              + id.toString() + "_" + getUserName(jobConf) + "_"
>              + trimJobName(getJobName(jobConf));
>     }
>
>     /**
>      * Trims the job-name if required
>      */
>     private static String trimJobName(String jobName) {
>       if (jobName.length() > JOB_NAME_TRIM_LENGTH) {
>         jobName = jobName.substring(0, JOB_NAME_TRIM_LENGTH);
>       }
>       return jobName;
>     }
>
> Roughly speaking, the history file name is composed in the following way:
>
> hostname of JT + "_" + start time of JT + "_" + job id + "_" + user name +
> "_" + trimed job name
>
> Cheers!
>
> Hailong
>
> 2011-07-07
>
>
>
> ***********************************************
> * Hailong Yang, PhD. Candidate
> * Sino-German Joint Software Institute,
> * School of Computer Science&Engineering, Beihang University
> * Phone: (86-010)82315908
> * Email: [hidden email]
> * Address: G413, New Main Building in Beihang University,
> *              No.37 XueYuan Road,HaiDian District,
> *              Beijing,P.R.China,100191
> ***********************************************
>
>
>
> 发件人: sangroya
> 发送时间: 2011-07-07  15:49:58
> 收件人: hadoop-user
> 抄送:
> 主题: Re: measure the time taken to complete map and reduce phase
>
> Hi,
> Thanks!
> I am able to parse the Job History Logs(JHL). But, I need to know how
> hadoop assign a name to a file in Job History Logs(JHL).
> I can see that files are named on my local single node cluster as this:
> localhost_1309975809398_job_201107062010_0759_sangroya_word+count.
> But, I am just wondering, what is the exact pattern to name every file
> like this.
> Best Regards,
> Amit
> On Tue, Jul 5, 2011 at 6:53 AM, Hailong [via Lucene]
> <[hidden email]> wrote:
>> Hi sangroya,
>>
>> You can look at the job administration portal at port of 50030 on your
>> JobTracker such as '<a
>> href="<a href="http://localhost:50030'">http://localhost:50030'"><a
>> href="http://localhost:50030'">http://localhost:50030'. At the bottom of the
>> web page there is an item named 'Job Tracker History', click into it and
>> find you job with the job id. There goes the information you want.
>>
>>
>> Cheers!
>>
>> Hailong
>>
>> 2011-07-05
>>
>>
>>
>> ***********************************************
>> * Hailong Yang, PhD. Candidate
>> * Sino-German Joint Software Institute,
>> * School of Computer Science&Engineering, Beihang University
>> * Phone: (86-010)82315908
>> * Email: [hidden email]
>> * Address: G413, New Main Building in Beihang University,
>> *              No.37 XueYuan Road,HaiDian District,
>> *              Beijing,P.R.China,100191
>> ***********************************************
>>
>>
>>
>> 发件人: sangroya
>> 发送时间: 2011-07-05  10:56:38
>> 收件人: hadoop-user
>> 抄送:
>> 主题: measure the time taken to complete map and reduce phase
>>
>> Hi,
>> I am trying to monitor the time to complete a map phase and reduce
>> phase in hadoop. Is there any way to measure the time taken to
>> complete map and reduce phase in a cluster.
>> Thanks,
>> Amit
>> --
>> View this message in context:
>>
>> http://lucene.472066.n3.nabble.com/measure-the-time-taken-to-complete-map-and-reduce-phase-tp3136991p3136991.html
>> Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
>>
>>
>> ________________________________
>> If you reply to this email, your message will be added to the discussion
>> below:
>>
>> http://lucene.472066.n3.nabble.com/measure-the-time-taken-to-complete-map-and-reduce-phase-tp3136991p3139665.html
>> To unsubscribe from measure the time taken to complete map and reduce
>> phase,
>> click here.
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/measure-the-time-taken-to-complete-map-and-reduce-phase-tp3136991p3147426.html
> Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
>
>
> ________________________________
> If you reply to this email, your message will be added to the discussion
> below:
> http://lucene.472066.n3.nabble.com/measure-the-time-taken-to-complete-map-and-reduce-phase-tp3136991p3147566.html
> To unsubscribe from measure the time taken to complete map and reduce phase,
> click here.
--
View this message in context: http://lucene.472066.n3.nabble.com/measure-the-time-taken-to-complete-map-and-reduce-phase-tp3136991p3148620.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message