hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1719) Improve the utilization of shuffle copier threads
Date Sun, 30 Dec 2007 08:54:43 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12554984
] 

Hadoop QA commented on HADOOP-1719:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
http://issues.apache.org/jira/secure/attachment/12372152/HADOOP-1719.patch
against trunk revision r607330.

    @author +1.  The patch does not contain any @author tags.

    javadoc +1.  The javadoc tool did not generate any warning messages.

    javac +1.  The applied patch does not generate any new compiler warnings.

    findbugs +1.  The patch does not introduce any new Findbugs warnings.

    core tests +1.  The patch passed core unit tests.

    contrib tests -1.  The patch failed contrib unit tests.

Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1431/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1431/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1431/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1431/console

This message is automatically generated.

> Improve the utilization of shuffle copier threads
> -------------------------------------------------
>
>                 Key: HADOOP-1719
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1719
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Devaraj Das
>            Assignee: Amar Kamat
>         Attachments: 1719.1.patch, 1719.patch, HADOOP-1719.patch, HADOOP-1719.patch
>
>
> In the current design, the scheduling of copies is done and the scheduler (the main loop
in fetchOutputs) won't schedule anything until it hears back from at least one of the copier
threads. Due to this, the main loop won't query the TaskTracker asking for new map locations
and may not be using all the copiers effectively. This may not be an issue for small-sized
map outputs, where at steady state, the frequency of such notifications is frequent.
> Ideally, we should schedule all what we can, and, depending on how busy we currently
are, query the tasktracker for more map locations.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message