hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Abdul Qadeer (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4010) Chaging LineRecordReader algo so that it does not need to skip backwards in the stream
Date Thu, 04 Sep 2008 01:23:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12628224#action_12628224

Abdul Qadeer commented on HADOOP-4010:

bq. Due to new LineRecordReader algorithm, the first split will process one more line as compared
to other mappers

bq. That's probably not going to be acceptable to users of NLineInputFormat. Users employing
N formatted lines to initialize and run a mapper may find their jobs no longer work if the
input is offset or if a map receives N+1 lines. If this is necessary for the new algorithm,
rewriting or somehow accommodating this case may be required.

I have changed NLineInputFormat to work it with the new LineRecordReader algorithm.
The diff of the file is in the following.  After this change I don't need to make any change
in the TestLineInputFormat test case.

--- src/mapred/org/apache/hadoop/mapred/lib/NLineInputFormat.java       (revisio
n 687954)
+++ src/mapred/org/apache/hadoop/mapred/lib/NLineInputFormat.java       (working
@@ -93,10 +93,19 @@
         long begin = 0;
         long length = 0;
         int num = -1;
-        while ((num = lr.readLine(line)) > 0) {
+        while ((num = lr.readLine(line)) > 0) {
           length += num;
           if (numLines == N) {
+            //NLineInputFormat uses LineRecordReader, which
+            //always reads at least one character out of its
+            //upper split boundary.  So to use LineRecordReader
+            // such that there are N lines in each split, we move
+            //back the upper split limits of each split by one
+            //character.
+            if(begin == 0){
+              length--;
+            }
             splits.add(new FileSplit(fileName, begin, length, new String[]{}));

             begin += length;
             length = 0;

> Chaging LineRecordReader algo so that it does not need to skip backwards in the stream
> --------------------------------------------------------------------------------------
>                 Key: HADOOP-4010
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4010
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>    Affects Versions: 0.19.0
>            Reporter: Abdul Qadeer
>            Assignee: Abdul Qadeer
>             Fix For: 0.19.0
>         Attachments: Hadoop-4010.patch, Hadoop-4010_version2.patch
> The current algorithm of the LineRecordReader needs to move backwards in the stream (in
its constructor) to correctly position itself in the stream.  So it moves back one byte from
the start of its split and try to read a record (i.e. a line) and throws that away.  This
is so because it is sure that, this line would be taken care of by some other mapper.  This
algorithm is difficult and in-efficient if used for compressed stream where data is coming
to the LineRecordReader via some codecs. (Although in the current implementation, Hadoop does
not split a compressed file and only makes one split from the start to the end of the file
and so only one mapper handles it.  We are currently working on BZip2 codecs where splitting
is possible to work with Hadoop.  So this proposed change will make it possible to uniformly
handle plain as well as compressed stream.)
> In the new algorithm, each mapper always skips its first line because it is sure that,
that line would have been read by some other mapper.  So now each mapper must finish its reading
at a record boundary which is always beyond its upper split limit.  Due to this change, LineRecordReader
does not need to move backwards in the stream.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message