hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Enis Soztutar (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-153) skip records that throw exceptions
Date Tue, 24 Jun 2008 12:06:45 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Enis Soztutar updated HADOOP-153:

    Attachment: skipRecords_wip1.patch

I am attaching the patch I have developed several weeks ago for this issue. It is just an
illustrative example, but I guess the final patch might rely heavily on this. 
It captures several of the above items : 
1. Tasks have an associated badrecords list, which are kept in the JT's TaskInProgress objects.
I do not think we should skip "ranges" of possibly valid records in favor of skipping bad
2. bad records are pushed / pulled via Reporter / UmblicalProtocol / TaskStatus
3. configurable max number of skippable records per map and reduce. Percentages do not apply
here, since we do not know the absolute number of records to compute the percentage. 
4. No support for pipes / streaming. 
For the pipes case, I think we should first run the tasks normally, until an error occurs.
Upon the subsequent run of the same task, then the pipes  application should run in "debug"
mode. In debug mode, the application should send back the key(or recordID) before it is passed
to the mapper/reducer. 

For streaming, again the application should run in debug mode on second trial. In this mode,
the framework should send the key/value pairs one-by-one, meaning that it will not send another
pair over the pipe, before the previous are consumed by the application. But as far as I know,
there is no mechanism to detect this. So we might opt for delaying this to another issue.

> skip records that throw exceptions
> ----------------------------------
>                 Key: HADOOP-153
>                 URL: https://issues.apache.org/jira/browse/HADOOP-153
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.2.0
>            Reporter: Doug Cutting
>            Assignee: Sharad Agarwal
>         Attachments: skipRecords_wip1.patch
> MapReduce should skip records that throw exceptions.
> If the exception is thrown under RecordReader.next() then RecordReader implementations
should automatically skip to the start of a subsequent record.
> Exceptions in map and reduce implementations can simply be logged, unless they happen
under RecordWriter.write().  Cancelling partial output could be hard.  So such output errors
will still result in task failure.
> This behaviour should be optional, but enabled by default.  A count of errors per task
and job should be maintained and displayed in the web ui.  Perhaps if some percentage of records
(>50%?) result in exceptions then the task should fail.  This would stop jobs early that
are misconfigured or have buggy code.
> Thoughts?

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message