hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5873) Shuffle bandwidth computation includes time spent waiting for maps
Date Thu, 16 Oct 2014 11:40:38 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173645#comment-14173645

Hudson commented on MAPREDUCE-5873:

FAILURE: Integrated in Hadoop-Yarn-trunk #713 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/713/])
MAPREDUCE-5873. Shuffle bandwidth computation includes time spent waiting for maps. Contributed
by Siqi Li (jlowe: rev b9edad64034a9c8a121ec2b37792c190ba561e26)
* hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/ShuffleSchedulerImpl.java
* hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java
* hadoop-mapreduce-project/CHANGES.txt
* hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/LocalFetcher.java
* hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestShuffleScheduler.java

> Shuffle bandwidth computation includes time spent waiting for maps
> ------------------------------------------------------------------
>                 Key: MAPREDUCE-5873
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5873
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 2.3.0
>            Reporter: Siqi Li
>            Assignee: Siqi Li
>             Fix For: 2.6.0
>         Attachments: MAPREDUCE-5873.v1.patch, MAPREDUCE-5873.v2.patch, MAPREDUCE-5873.v3.patch,
MAPREDUCE-5873.v4.patch, MAPREDUCE-5873.v5.patch, MAPREDUCE-5873.v6.patch, MAPREDUCE-5873.v9.patch
> Currently ShuffleScheduler in ReduceTask JVM status displays bandwidth. Its definition
however is confusing because it captures the time where there is no copying because there
is a pause between when new wave of map outputs is available.
> current bw is definded as (bytes copied so far) / (total time in the copy phase so far)
> It would be more useful 
> 1) to measure bandwidth of a single copy call.
> 2) display aggregated bw as long as there is at least one fetcher is in the copy call.

This message was sent by Atlassian JIRA

View raw message