hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1144) Hadoop should allow a configurable percentage of failed map tasks before declaring a job failed.
Date Wed, 02 May 2007 05:29:15 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12493011
] 

Owen O'Malley commented on HADOOP-1144:
---------------------------------------

I guess I'm ok with the mapred.max.{map,reduce}.failures.percent, although if we are trying
to make the names somewhat hierarchical, it should be more like: mapred.task.{map,reduce}.percent-failures.max
or some such.

Using counters to count failed tips would make sense, since we already have the infrastructure
to get it. Does the JobClient let you get the counters for individual tips?

I believe there is already a bug to have the framework skip bad records. That would be a better
solution, in my opinion since it handles input and processing.

> Hadoop should allow a configurable percentage of failed map tasks before declaring a
job failed.
> ------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-1144
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1144
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>    Affects Versions: 0.12.0
>            Reporter: Christian Kunz
>         Assigned To: Arun C Murthy
>             Fix For: 0.13.0
>
>
> In our environment it can occur that some map tasks will fail repeatedly because of corrupt
input data, which sometimes is non-critical as long as the amount is limited. In this case
it is annoying that the whole Hadoop job fails and cannot be restarted till the corrupt data
are identified and eliminated from the input. It would be extremely helpful if the job configuration
would allow to indicate how many map tasks are allowed to fail.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message