hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sanjay Dahiya (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-239) job tracker WI drops jobs after 24 hours
Date Mon, 07 Aug 2006 19:16:16 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-239?page=comments#action_12426298 ] 
            
Sanjay Dahiya commented on HADOOP-239:
--------------------------------------

I meant on JobTracker we log the job transition states in a structured ( XML?) persistent
file. This is different from the standard log file and can be parsed and displayed by a JSP
page (something like JobHistory.jsp) . It includes only completed and failed jobs. So from
the user interface perspective running jobs are displayed as they are currently but failed
and completed jobs are displayed from this log.

We can add extra information in this log than what we have currently as a postmortem analysis
like time spent by different phases or by different hosts (?). 

This will require unique job ids across jobtracker restarts otherwise it will be difficult
to track jobs in history with same id. 
This log file can contain a configurable numbers of days of history, which can be browsed
by time. Optionally this history can itself reside in HDFS. Does this make sense ?

> job tracker WI drops jobs after 24 hours
> ----------------------------------------
>
>                 Key: HADOOP-239
>                 URL: http://issues.apache.org/jira/browse/HADOOP-239
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Yoram Arnon
>         Assigned To: Sanjay Dahiya
>            Priority: Minor
>
> The jobtracker's WI, keeps track of jobs executed in the past 24 hours.
> if the cluster was idle for a day (say Sunday) it drops all its history.
> Monday morning, the page is empty.
> Better would be to store a fixed number of jobs (say 10 each of succeeded and failed
jobs).
> Also, if the job tracker is restarted, it loses all its history.
> The history should be persistent, withstanding restarts and upgrades.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message