hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mahadev konar (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3297) The way in which ReduceTask/TaskTracker gets completion events during shuffle can be improved
Date Wed, 07 May 2008 07:25:55 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12594808#action_12594808

Mahadev konar commented on HADOOP-3297:

the patch looks good... the only concern I have is that if we want to check and see that it
does not degrade perofrmace and lead to more problems at the jobtracker or otherwise.
We could try running sort with this patch or some map reduce job where we have a huge number
of mappers say 100,000 and 500 reducers or something. 

This is just to verify that the task tracker being agressive in fetching the mapoutputs does
not degrade performance. The maps are short lived, so its not hard to imagine a situation
that all the reduces start bombarding the jobtracker with requests for maps at the same time
asking for more. We should check to see if the jobtracker can handle the load and the performance
in such a situation does not degrade. 

> The way in which ReduceTask/TaskTracker gets completion events during shuffle can be
> ---------------------------------------------------------------------------------------------
>                 Key: HADOOP-3297
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3297
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.18.0
>         Attachments: 3297.patch, 3297.patch
> Certain things like poll frequency, number of events fetched in one go, etc. can probably
be improved to improve the shuffle performance. This would affect the task-->tasktracker
and the tasktracker-->jobtracker shuffle related RPCs.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message