hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Iván de Prado (JIRA) <j...@apache.org>
Subject [jira] [Created] (MAPREDUCE-4876) Adopt a Tuple MapReduce API instead of classic MapReduce one
Date Thu, 13 Dec 2012 17:02:14 GMT
Iván de Prado created MAPREDUCE-4876:

             Summary: Adopt a Tuple MapReduce API instead of classic MapReduce one
                 Key: MAPREDUCE-4876
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4876
             Project: Hadoop Map/Reduce
          Issue Type: Wish
            Reporter: Iván de Prado
            Priority: Minor

After using MapReduce for many years, we have notice that it lacks some important features:
compound records, easy intra-reduce sorting and join capabilities. We have elaborated slightly
modified MapReduce foundation to overcome these problems: Tuple MapReduce. You can see a full
paper published at the ICDM 2012 that describes it at http://pangool.net/TupleMapReduce.pdf

The good news are:
1) That it is not needed any architectural change on Hadoop to embrace Tuple MapReduce
2) Indeed, we have proven that it is possible to implement it on top of Hadoop. See the Pangool
Open Source project ( http://pangool.net/ ). 
3) It performs very efficiently ( http://pangool.net/benchmark.html )
4) It is compatible with all Hadoop stack: Writables, Serializers, Input/OutputFormats, etc.

We believe Hadoop community could benefits from it in different ways:
1) By getting ideas for future API redesign
2) By adopting Pangool inside Hadoop. Of course, we would be helping and contributing with
anything needed, including by doing any adaptation changes needed (too few, because as I told,
everything is compatible with existing MapReduce)

Obviously, we prefer the second. But at least, we believe some good ideas can be obtained
by looking at Tuple MapReduce and Pangool.  

There are also other improvements in Pangool that would improve Hadoop API:
1) Configuration by instance: passing parameters by constructor. For example, Pangool Input/OutputFormats
can be configured by providing values to the constructor
2) Stateful serialization. What is requested in  https://issues.apache.org/jira/browse/MAPREDUCE-1462
is supported by Pangool
3) First-class multipleinput/multipleoutput

Well, we are open to the discussion and to contribute.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message