hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nikola Vujic (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5791) Shuffle phase is slow in Windows - FadviseFileRegion::transferTo does not read disks efficiently
Date Fri, 04 Apr 2014 13:17:16 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13959934#comment-13959934

Nikola Vujic commented on MAPREDUCE-5791:

In order to check the performance impact of this patch, I have tested the patch on a 80 node
hadoop cluster on Windows. Here are the results:

Terasort 5 TB
1540 map tasks
770 reduce tasks

                                 Elapsed      Avg Map Time     Avg Reduce time      Avg Shuffle
Time     Avg Merge Time
default                      3194.67  	  402.33	              248.67	              1471.33
optimized shuffle	2411.00	  392.00	              689.67	              674.67	            
default/optmized      1.33            1.03                     0.36                      
    2.18                          0.72

* optimized shuffle is configured to use 512K buffer size for the buffer copy shuffle.
** presented numbers are avg of at least 3 runs.
Optimized shuffle version is 1.33x faster than the default version. Gain in the shuffle phase
alone is 2.18x.

> Shuffle phase is slow in Windows - FadviseFileRegion::transferTo does not read disks
> ------------------------------------------------------------------------------------------------
>                 Key: MAPREDUCE-5791
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5791
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: client
>    Affects Versions: 3.0.0, 2.3.0
>            Reporter: Nikola Vujic
>            Assignee: Nikola Vujic
>             Fix For: 3.0.0, 2.4.0
>         Attachments: MAPREDUCE-5791.patch, MAPREDUCE-5791.patch, MAPREDUCE-5791.patch
> transferTo method in org.apache.hadoop.mapred.FadvisedFileRegion is using transferTo
method from a FileChannel to transfer data from a disk to socket. This is performing slow
in Windows, slower than in Linux. The reason is that transferTo method for the java.nio is
issuing 32K IO requests all the time. In Windows, these 32K transfers are not optimal and
we don't get the best performance form the underlying IO subsystem. In order to achieve better
performance when reading from the drives, we need to read data in bigger chunks, 512K for

This message was sent by Atlassian JIRA

View raw message