hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun C Murthy (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-1721) Rack-aware Shuffle
Date Thu, 16 Aug 2007 10:59:31 GMT
Rack-aware Shuffle

                 Key: HADOOP-1721
                 URL: https://issues.apache.org/jira/browse/HADOOP-1721
             Project: Hadoop
          Issue Type: Improvement
          Components: mapred
            Reporter: Arun C Murthy
            Assignee: Arun C Murthy
             Fix For: 0.15.0

We could try and experiment with *rack-aware* scheduling of fetches per-reducer. Given the
disparities between in-rack and off-rack bandwidth it could be a improvement to do something
along these lines:

if (no. of known map-output locations > than no. of copier threads) {
  try to schedule 75% of copies off-rack
  try schedule 25% of copies in-rack

This could lead to better utilization of both in-rack & switch b/w...

Clearly we want to schedule more cross-switch than in-rack since off-rack copies will take
significantly more time; hence the 75-25 split.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message