hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6958) Shuffle audit logger should log size of shuffle transfer
Date Tue, 19 Sep 2017 14:40:00 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16171800#comment-16171800

Hudson commented on MAPREDUCE-6958:

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12912 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/12912/])
MAPREDUCE-6958. Shuffle audit logger should log size of shuffle (jlowe: rev 3a20debddeac69596ceb5b36f8413529ea8570e6)
* (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java

> Shuffle audit logger should log size of shuffle transfer
> --------------------------------------------------------
>                 Key: MAPREDUCE-6958
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6958
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>            Reporter: Jason Lowe
>            Assignee: Jason Lowe
>            Priority: Minor
>             Fix For: 2.9.0, 3.0.0-beta1
>         Attachments: MAPREDUCE-6958.001.patch, MAPREDUCE-6958.002.patch, MAPREDUCE-6958.003.patch,
MAPREDUCE-6958-branch-2.002.patch, MAPREDUCE-6958-branch-2.8.002.patch
> The shuffle audit logger currently logs the job ID and reducer ID but nothing about the
size of the requested transfer.  It calculates this as part of the HTTP response headers,
so it would be trivial to log the response size.  This would be very valuable for debugging
network traffic storms from the shuffle handler.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org

View raw message