hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amareshwari Sri Ramadasu (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-2178) Job history on HDFS
Date Fri, 11 Jan 2008 15:44:34 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558010#action_12558010

Amareshwari Sri Ramadasu commented on HADOOP-2178:

To address use cases 1 and 2 suggested by Eric,  I propose the following approach.

If the job tracker is static, we will store history logs in a location specified by hadoop.job.history.location,
by default it is local file system.
If the job tracker is not static (like HOD JT) we will store log files in user specified location,
by default it is job output directory.

We will not have index file any more, because appending becomes an issue in DFS. And we dont
need one in case of non-static JT.
For static JT, We can do listing of files in the log directory to show the first page.

> Job history on HDFS
> -------------------
>                 Key: HADOOP-2178
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2178
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Amareshwari Sri Ramadasu
>            Assignee: Amareshwari Sri Ramadasu
>             Fix For: 0.16.0
> This issue addresses the following items :
> 1.  Check for accuracy of job tracker history logs.
> 2.  After completion of the job, copy the JobHistory.log(Master index file) and the job
history files to the DFS.
> 3. User can load the history with commands
> bin/hadoop job -history <directory> 
> or
> bin/hadoop job -history <jobid>
> This will start a stand-alone jetty and load jsps

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message