hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "hailong.yang1115" <hailong.yang1...@gmail.com>
Subject Re: Re: measure the time taken to complete map and reduce phase
Date Thu, 07 Jul 2011 08:57:55 GMT
Hi sangroya,

I think you may be interested in reading the following piece of code from JobHistory.java
in Hadoop.

    /**
     * Generates the job history filename for a new job
     */
    private static String getNewJobHistoryFileName(JobConf jobConf, JobID id) {
      return JOBTRACKER_UNIQUE_STRING
             + id.toString() + "_" + getUserName(jobConf) + "_" 
             + trimJobName(getJobName(jobConf));
    }
    
    /**
     * Trims the job-name if required
     */
    private static String trimJobName(String jobName) {
      if (jobName.length() > JOB_NAME_TRIM_LENGTH) {
        jobName = jobName.substring(0, JOB_NAME_TRIM_LENGTH);
      }
      return jobName;
    }
    
Roughly speaking, the history file name is composed in the following way:

hostname of JT + "_" + start time of JT + "_" + job id + "_" + user name + "_" + trimed job
name

Cheers!

Hailong

2011-07-07 



***********************************************
* Hailong Yang, PhD. Candidate 
* Sino-German Joint Software Institute, 
* School of Computer Science&Engineering, Beihang University
* Phone: (86-010)82315908
* Email: hailong.yang1115@gmail.com
* Address: G413, New Main Building in Beihang University, 
*              No.37 XueYuan Road,HaiDian District, 
*              Beijing,P.R.China,100191
***********************************************



发件人: sangroya 
发送时间: 2011-07-07  15:49:58 
收件人: hadoop-user 
抄送: 
主题: Re: measure the time taken to complete map and reduce phase 
 
Hi,
Thanks!
I am able to parse the Job History Logs(JHL). But, I need to know how
hadoop assign a name to a file in Job History Logs(JHL).
I can see that files are named on my local single node cluster as this:
localhost_1309975809398_job_201107062010_0759_sangroya_word+count.
But, I am just wondering, what is the exact pattern to name every file
like this.
Best Regards,
Amit
On Tue, Jul 5, 2011 at 6:53 AM, Hailong [via Lucene]
<ml-node+3139665-1716333714-409464@n3.nabble.com> wrote:
> Hi sangroya,
>
> You can look at the job administration portal at port of 50030 on your
> JobTracker such as '<a
> href="http://localhost:50030'">http://localhost:50030'. At the bottom of the
> web page there is an item named 'Job Tracker History', click into it and
> find you job with the job id. There goes the information you want.
>
>
> Cheers!
>
> Hailong
>
> 2011-07-05
>
>
>
> ***********************************************
> * Hailong Yang, PhD. Candidate
> * Sino-German Joint Software Institute,
> * School of Computer Science&Engineering, Beihang University
> * Phone: (86-010)82315908
> * Email: [hidden email]
> * Address: G413, New Main Building in Beihang University,
> *              No.37 XueYuan Road,HaiDian District,
> *              Beijing,P.R.China,100191
> ***********************************************
>
>
>
> 发件人: sangroya
> 发送时间: 2011-07-05  10:56:38
> 收件人: hadoop-user
> 抄送:
> 主题: measure the time taken to complete map and reduce phase
>
> Hi,
> I am trying to monitor the time to complete a map phase and reduce
> phase in hadoop. Is there any way to measure the time taken to
> complete map and reduce phase in a cluster.
> Thanks,
> Amit
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/measure-the-time-taken-to-complete-map-and-reduce-phase-tp3136991p3136991.html
> Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
>
>
> ________________________________
> If you reply to this email, your message will be added to the discussion
> below:
> http://lucene.472066.n3.nabble.com/measure-the-time-taken-to-complete-map-and-reduce-phase-tp3136991p3139665.html
> To unsubscribe from measure the time taken to complete map and reduce phase,
> click here.
--
View this message in context: http://lucene.472066.n3.nabble.com/measure-the-time-taken-to-complete-map-and-reduce-phase-tp3136991p3147426.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message