hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jothi Padmanabhan (JIRA)" <j...@apache.org>
Subject [jira] Issue Comment Edited: (HADOOP-3514) Reduce seeks during shuffle, by inline crcs
Date Mon, 08 Sep 2008 13:18:47 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12629136#action_12629136
] 

jothipn edited comment on HADOOP-3514 at 9/8/08 6:17 AM:
-------------------------------------------------------------------

The above patch fixes the problem observed when running with the native lzo library for map
output compression. 
The problem was with the IFileInputStream read method which is required to return the byte
read as an integer. A simple assignment of
{code}
int result = byte
{code}
 does not work as the byte is interpreted as a signed byte and so the assigned integer has
a wrong value. Instead the result has to be assigned as 
{code}
int result = (byte & 0xFF)
{code}
correctly assigns the byte to the integer

The following are the performance improvements observed with this patch when running the loadgen
program with
60 reduce copiers, 100 http threads, 450 task trackers, with the following command line (Thanks
Runping for the identifying the problem configuration that best illustrates the utility of
this patch)
{noformat}
bin/hadoop jar hadoop-0.19.0-dev-test.jar loadgen \
-D test.randomtextwrite.bytes_per_map=$((240*1024)) \
-D test.randomtextwrite.total_bytes=$((200*1024*100000)) \
-D mapred.compress.map.output=false \
-r 2200 \
-outKey org.apache.hadoop.io.Text \
-outValue org.apache.hadoop.io.Text \
-outFormat org.apache.hadoop.mapred.lib.NullOutputFormat \
-outdir fakeout
{noformat}

The patch showed an overall improvement of about 5% with about 10% improvement in shuffle.
                            Trunk              Patch
Map Time             6:10                 6:04
Reduce Time       17:11              16:22
Overall                23:21              22:26


      was (Author: jothipn):
    The above patch fixes the problem observed when running with the native lzo library for
map output compression. 
The problem was with the IFileInputStream read method which is required to return the byte
read as an integer. A simple assignment of
{code}
int result = byte
{code}
 does not work as the byte is interpreted as a signed byte and so the assigned integer has
a wrong value. Instead the result has to be assigned as 
{code}
int result = (byte & 0xFF)
{code}
correctly assigns the byte to the integer

The following are the performance improvements observed with this patch when running the loadgen
program with
60 reduce copiers, 100 http threads, 450 task trackers, with the following command line
<pre>
bin/hadoop jar hadoop-0.19.0-dev-test.jar loadgen \
-D test.randomtextwrite.bytes_per_map=$((240*1024)) \
-D test.randomtextwrite.total_bytes=$((200*1024*100000)) \
-D mapred.compress.map.output=false \
-r 2200 \
-outKey org.apache.hadoop.io.Text \
-outValue org.apache.hadoop.io.Text \
-outFormat org.apache.hadoop.mapred.lib.NullOutputFormat \
-outdir fakeout
</pre>

The patch showed an overall improvement of about 5% with about 10% improvement in shuffle.
                            Trunk              Patch
Map Time             6:10                 6:04
Reduce Time       17:11              16:22
Overall                23:21              22:26

  
> Reduce seeks during shuffle, by inline crcs
> -------------------------------------------
>
>                 Key: HADOOP-3514
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3514
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>    Affects Versions: 0.18.0
>            Reporter: Devaraj Das
>            Assignee: Jothi Padmanabhan
>             Fix For: 0.19.0
>
>         Attachments: hadoop-3514-v1.patch, hadoop-3514-v10.patch, hadoop-3514-v11.patch,
hadoop-3514-v12.patch, hadoop-3514-v2.patch, hadoop-3514-v3.patch, hadoop-3514-v4.patch, hadoop-3514-v5.patch,
hadoop-3514-v6.patch, hadoop-3514-v7.patch, hadoop-3514-v8.patch, hadoop-3514-v9.patch, hadoop-3514.patch
>
>
> The number of seeks can be reduced by half in the iFile if we move the crc into the iFile
rather than having a separate file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message