hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Krishna Ramachandran (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1699) JobHistory shouldn't be disabled for any reason
Date Thu, 24 Jun 2010 20:43:52 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882320#action_12882320

Krishna Ramachandran commented on MAPREDUCE-1699:

yes - this is good. I have added a map and logSubmitted will initialize in case of an error
for that job and use that for subsequent logging (failed, killed etc.,)

Will update patch once testing is completed

Am not able to entirely take out disableHistory because of this code (incorrect javadoc -
there is no param - jobID)

     * Logs history meta-info to the history file. This needs to be called once
     * per history file. 
     * @param jobId job id, assigned by jobtracker. 
    static void logMetaInfo(ArrayList<PrintWriter> writers){
      if (!disableHistory){
        if (null != writers){
          JobHistory.log(writers, RecordTypes.Meta,
              new Keys[] {Keys.VERSION},
              new String[] {String.valueOf(VERSION)});

also there are couple of (public) methods that use this flag.  though I do not see where they
are called

   * Returns history disable status. by default history is enabled so this
   * method returns false. 
   * @return true if history logging is disabled, false otherwise. 
  public static boolean isDisableHistory() {
    return disableHistory;

   * Enable/disable history logging. Default value is false, so history 
   * is enabled by default. 
   * @param disableHistory true if history should be disabled, false otherwise. 
  public static void setDisableHistory(boolean disableHistory) {
    JobHistory.disableHistory = disableHistory;

> JobHistory shouldn't be disabled for any reason
> -----------------------------------------------
>                 Key: MAPREDUCE-1699
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: jobtracker
>    Affects Versions: 0.20.2
>            Reporter: Arun C Murthy
>            Assignee: Krishna Ramachandran
>             Fix For: 0.20.3
>         Attachments: mapred-1699-1.patch, mapred-1699-2.patch, mapred-1699-3.patch, mapred-1699.patch
> Recently we have had issues with JobTracker silently disabling job-history and starting
to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never
do this.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message