hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sharad Agarwal (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-864) Enhance JobClient API implementations to look at history files to get information about jobs that are not in memory
Date Thu, 03 Sep 2009 10:23:33 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12750958#action_12750958
] 

Sharad Agarwal commented on MAPREDUCE-864:
------------------------------------------

An additional state COMPLETED can be added to Job#JobState {DEFINE, RUNNING} to signify that
job is completed and history file is there in HDFS. In COMPLETED state, Job client will read
the history file from HDFS location and construct the job information using JobHistory parser
(provided as part of MAPREDUCE-157). While calling api's like Job#getCounters, Job clients
would be transparent to the fact that information is being served from Jobtracker or from
parsed history data.

> Enhance JobClient API implementations to look at history files to get information about
jobs that are not in memory
> -------------------------------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-864
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-864
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>          Components: jobtracker
>            Reporter: Devaraj Das
>            Assignee: Sharad Agarwal
>             Fix For: 0.21.0
>
>
> MAPREDUCE-817 added an API to get the JobHistory URL from the JobTracker. This is useful
in two ways:
> 1) Users can use this API to get the URL, copy the history files to their local disk,
and, do processing on them
> 2) APIs like JobSubmissionProtocol.getJobCounters, can read a part of the history file,
and then return the information to the caller (if the job is not there in JT memory). This
would  mimic most of the CompletedJobsStatusStore functionality.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message