hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun C Murthy (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-5870) Implement a memory-to-memory sort in the map task
Date Tue, 19 May 2009 07:18:45 GMT
Implement a memory-to-memory sort in the map task

                 Key: HADOOP-5870
                 URL: https://issues.apache.org/jira/browse/HADOOP-5870
             Project: Hadoop Core
          Issue Type: Improvement
          Components: mapred
            Reporter: Arun C Murthy

The motivation is similar to HADOOP-5831...

Currently we collect map-outputs in the sort buffer (io.sort.mb) which we eventually sort
and spill to disk. For latency-sensitive applications with sufficient memory, e.g. terasort,
we could do better by doing a memory-to-memory sort followed by a final memory-to-disk merge.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message