hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sharad Agarwal (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-3829) Narrown down skipped records based on user acceptable value
Date Thu, 04 Sep 2008 06:42:44 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Sharad Agarwal updated HADOOP-3829:

    Attachment: 3829_v2.patch

fairly tested patch. few additional things:
- moved counters from Counters.java to SkipBadRecords.java as these are specific to skip feature.
- fixed hasNext in SortedRanges.SkipRangeIterator.
- skipped records are not written to HDFS, if it is a test attempt (test attempt is to figure
out whether a range is good or bad. In this only records in test range are passed to the mapper/reducer,
others are skipped).
- recordreader.next not called beyond the test range during test attempt.
- renamed failedRanges to skipRanges in Task.java
- added config params to hadoop-default

> Narrown down skipped records based on user acceptable value
> -----------------------------------------------------------
>                 Key: HADOOP-3829
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3829
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: 3829_v1.153_7.patch, 3829_v1.patch, 3829_v2.patch
> This is an incremental step over HADOOP-153.
> If the number of skipped records in the neighborhood of a bad record are not acceptable
to the user, then narrow down the skipped range to the user acceptable value.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message