hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joydeep Sen Sarma (JIRA)" <j...@apache.org>
Subject [jira] Created: (MAPREDUCE-1639) Grouping using hashing instead of sorting
Date Sat, 27 Mar 2010 13:58:27 GMT
Grouping using hashing instead of sorting

                 Key: MAPREDUCE-1639
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1639
             Project: Hadoop Map/Reduce
          Issue Type: New Feature
            Reporter: Joydeep Sen Sarma

most applications of map-reduce care about grouping and not sorting. Sorting is a (relatively
expensive) way to achieve grouping. In order to achieve just grouping - one can:

- replace the sort on the Mappers with a HashTable - and maintain lists of key-values against
each hash-bucket.
- key-value tuples inside each hash bucket are sorted - before spilling or sending to Reducer.
Anytime this is done - Combiner can be invoked.
- HashTable is serialized by hash-bucketid. So merges (of either spills or Map Outputs) works
similar to today (at least there's no change in overall compute complexity of merge)

Of course this hashtable has nothing to do with partitioning. it's just a replacement for
map-side sort.


this is (pretty much) straight from the MARS project paper: http://www.cse.ust.hk/catalac/papers/mars_pact08.pdf.
They report a 45% speedup in inverted index calculation using hashing instead of sorting (reference
implementation is NOT against Hadoop though).

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message