hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rushabh S Shah (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-9867) org.apache.hadoop.mapred.LineRecordReader does not handle multibyte record delimiters well
Date Thu, 26 Jun 2014 14:16:25 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-9867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044681#comment-14044681

Rushabh S Shah commented on HADOOP-9867:

Hey Vinayakaumar,
Thanks for checking out the patch and providing valuable feedback.
I did ran into this test  case while solving this jira.
I am going to file another jira for this specific test case (and a couple of more which I
came across) since the test case you mentioned is not in the scope of this jira.

> org.apache.hadoop.mapred.LineRecordReader does not handle multibyte record delimiters
> ------------------------------------------------------------------------------------------
>                 Key: HADOOP-9867
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9867
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.2, 0.23.9, 2.2.0
>         Environment: CDH3U2 Redhat linux 5.7
>            Reporter: Kris Geusebroek
>            Assignee: Vinayakumar B
>            Priority: Critical
>         Attachments: HADOOP-9867.patch, HADOOP-9867.patch, HADOOP-9867.patch, HADOOP-9867.patch
> Having defined a recorddelimiter of multiple bytes in a new InputFileFormat sometimes
has the effect of skipping records from the input.
> This happens when the input splits are split off just after a recordseparator. Starting
point for the next split would be non zero and skipFirstLine would be true. A seek into the
file is done to start - 1 and the text until the first recorddelimiter is ignored (due to
the presumption that this record is already handled by the previous maptask). Since the re
ord delimiter is multibyte the seek only got the last byte of the delimiter into scope and
its not recognized as a full delimiter. So the text is skipped until the next delimiter (ignoring
a full record!!)

This message was sent by Atlassian JIRA

View raw message