Mailing-List: contact hadoop-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hadoop-dev@lucene.apache.org
Message-ID: <19456748.1187453790886.JavaMail.jira@brutus>
Date: Sat, 18 Aug 2007 09:16:30 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: hadoop-dev@lucene.apache.org
Subject: [jira] Commented: (HADOOP-1719) Improve the utilization of shuffle
 copier threads
In-Reply-To: <32443721.1187243491698.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HADOOP-1719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12520862 ] 

Hadoop QA commented on HADOOP-1719:
-----------------------------------

-1, new javadoc warnings

The javadoc tool appears to have generated warning messages when testing the latest attachment http://issues.apache.org/jira/secure/attachment/12364036/1719.1.patch against trunk revision r566886.

Test results:   http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/566/testReport/
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/566/console

Please note that this message is automatically generated and may represent a problem with the automation system and not the patch.

> Improve the utilization of shuffle copier threads
> -------------------------------------------------
>
>                 Key: HADOOP-1719
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1719
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.15.0
>
>         Attachments: 1719.1.patch, 1719.patch
>
>
> In the current design, the scheduling of copies is done and the scheduler (the main loop in fetchOutputs) won't schedule anything until it hears back from at least one of the copier threads. Due to this, the main loop won't query the TaskTracker asking for new map locations and may not be using all the copiers effectively. This may not be an issue for small-sized map outputs, where at steady state, the frequency of such notifications is frequent.
> Ideally, we should schedule all what we can, and, depending on how busy we currently are, query the tasktracker for more map locations.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.