hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-5791) Shuffle phase is slow in Windows - FadviseFileRegion::transferTo does not read disks efficiently
Date Mon, 24 Mar 2014 19:12:48 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-5791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Chris Nauroth updated MAPREDUCE-5791:
-------------------------------------

       Resolution: Fixed
    Fix Version/s: 2.4.0
                   3.0.0
           Status: Resolved  (was: Patch Available)

I committed this to trunk, branch-2 and branch-2.4.  Nikola, thank you for reporting the issue
and contributing a patch.

> Shuffle phase is slow in Windows - FadviseFileRegion::transferTo does not read disks
efficiently
> ------------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-5791
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5791
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: client
>    Affects Versions: 3.0.0, 2.3.0
>            Reporter: Nikola Vujic
>            Assignee: Nikola Vujic
>             Fix For: 3.0.0, 2.4.0
>
>         Attachments: MAPREDUCE-5791.patch, MAPREDUCE-5791.patch, MAPREDUCE-5791.patch
>
>
> transferTo method in org.apache.hadoop.mapred.FadvisedFileRegion is using transferTo
method from a FileChannel to transfer data from a disk to socket. This is performing slow
in Windows, slower than in Linux. The reason is that transferTo method for the java.nio is
issuing 32K IO requests all the time. In Windows, these 32K transfers are not optimal and
we don't get the best performance form the underlying IO subsystem. In order to achieve better
performance when reading from the drives, we need to read data in bigger chunks, 512K for
example.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message