hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Devaraj Das (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-1339) Shuffle should be refactored to a separate task by itself
Date Tue, 08 May 2007 06:25:15 GMT
Shuffle should be refactored to a separate task by itself
---------------------------------------------------------

                 Key: HADOOP-1339
                 URL: https://issues.apache.org/jira/browse/HADOOP-1339
             Project: Hadoop
          Issue Type: New Feature
          Components: mapred
            Reporter: Devaraj Das
             Fix For: 0.14.0


Currently, shuffle phase is part of the reduce task. The idea here is to move out the shuffle
as a first-class task. This will improve the usage of the network since we will then be able
to schedule shuffle tasks independently, and later on pin reduce tasks to those nodes. This
will make most sense for apps where there are multiple waves of reduces (the second wave of
reduces can directly start off doing the "reducer" phase).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message