Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Fri, 15 Sep 2017 21:51:00 +0000 (UTC)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: mapreduce-issues@hadoop.apache.org
Message-ID: <JIRA.13102398.1505425986000.127392.1505512260084@Atlassian.JIRA>
In-Reply-To: <JIRA.13102398.1505425986000@Atlassian.JIRA>
References: <JIRA.13102398.1505425986000@Atlassian.JIRA> <JIRA.13102398.1505425986447@jira-lw-us.apache.org>
Subject: [jira] [Commented] (MAPREDUCE-6958) Shuffle audit logger should log
 size of shuffle transfer
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Fri, 15 Sep 2017 21:51:03 -0000


    [ https://issues.apache.org/jira/browse/MAPREDUCE-6958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16168577#comment-16168577 ] 

Chris Douglas commented on MAPREDUCE-6958:
------------------------------------------

Sorry to ask for revs on this kind of patch, but this changes the format of the audit log in a way that might break downstream consumers. The mapIds are printed after the reducer in the revised version. Could this keep the format as-is, with the length appended?

The shuffle sizes used to be available in the clienttrace log. Was that removed from the ShuffleHandler at some point?

> Shuffle audit logger should log size of shuffle transfer
> --------------------------------------------------------
>
>                 Key: MAPREDUCE-6958
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6958
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>            Reporter: Jason Lowe
>            Assignee: Jason Lowe
>            Priority: Minor
>         Attachments: MAPREDUCE-6958.001.patch, MAPREDUCE-6958.002.patch
>
>
> The shuffle audit logger currently logs the job ID and reducer ID but nothing about the size of the requested transfer.  It calculates this as part of the HTTP response headers, so it would be trivial to log the response size.  This would be very valuable for debugging network traffic storms from the shuffle handler.


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org