hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1270) Randomize the fetch of map outputs
Date Tue, 08 May 2007 11:24:16 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12494252
] 

Hadoop QA commented on HADOOP-1270:
-----------------------------------

Integrated in Hadoop-Nightly #82 (See http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/82/)

> Randomize the fetch of map outputs
> ----------------------------------
>
>                 Key: HADOOP-1270
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1270
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>    Affects Versions: 0.12.3
>            Reporter: Arun C Murthy
>         Assigned To: Arun C Murthy
>             Fix For: 0.13.0
>
>         Attachments: HADOOP-1270_20070425_1.patch, HADOOP-1270_20070504_2.patch, HADOOP-1270_20070505_3.patch,
post-H-1270.png, pre-H-1270.png
>
>
> HADOOP-248 did away with random probing of maps for locating map outputs and instead
we now rely on TaskCompletionEvents for the same. 
> However we lost out on the benefit that the randomization in probing resulted in an added
benefit where the map's jetty isn't overloaded with requests for the outputs. We have now
a situation where a map completes, the JT is notified, *all* the reduces get the TaskCompletionEvent
and pretty much swamp the poor map's jetty and this repeats for each map.
> I propose we make a minor change where we collect a set of TaskCompletionEvents and randomize
the list before firing the fetches. Should help fix this mass-hysteria at the map's jetty.
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message