hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Devaraj Das (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1304) MAX_TASK_FAILURES should be configurable
Date Tue, 01 May 2007 06:03:15 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12492838
] 

Devaraj Das commented on HADOOP-1304:
-------------------------------------

Pasting Nigel's comments:
-----------
Some comments on the JobConf methods:
1) why no setter methods?
2) why are getters getNumMax... instead of getMaxNum...?
3) can this parameters be set per job? per instantiation of the JobTracker?
4) the @return javadoc comment should not start with the return type
5) the javadoc should document the default value (IMO)
6) the javadoc should document the property this method relates to
7) remove the word "we" from the javadoc comments -- such as "Get the configured number of
maximum attempts that will be made to run a map task, as specified by the <code>mapred.map.max.attempts</code>
property. If this property is not already set, the default is 4 attempts."
-----------

I think these config items should be per jobtracker instantiation to control the potential
abuse. In that case, it means that job submitters cannot tweak these, and it need not be part
of the jobconf at all. Do others agree with me on this?




> MAX_TASK_FAILURES should be configurable
> ----------------------------------------
>
>                 Key: HADOOP-1304
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1304
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>    Affects Versions: 0.12.3
>            Reporter: Christian Kunz
>         Assigned To: Devaraj Das
>         Attachments: 1304.patch, 1304.patch, 1304.patch
>
>
> After a couple of weeks of failed attempts I was able to finish a large job only after
I changed MAX_TASK_FAILURES to a higher value. In light of HADOOP-1144 (allowing a certain
amount of task failures without failing the job) it would be even better if this value could
be configured separately for mappers and reducers, because often a success of a job requires
the success of all reducers but not of all mappers.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message