hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aa...@buffalo.edu
Subject Hadoop: Divide and Conquer Algorithms
Date Sun, 28 Feb 2010 20:15:02 GMT
Hello Everybody,
                I have a small question. I want to know how would one implement
divide and conquer algorithms in Hadoop. For example suppose I want to implement
merge sort 100 lines in hadoop. There will be 10 mapper each sorting 10 lines.
Now comes the tough part

In the traditional version of merge sort each piece of 10 lines is combined to
form 5 pieces of 20 lines. The each piece of 20 lines is combined to form 3
pieces of 40 lines and so on. I am unable to understand how to implement this
functionality in the reducer. 

Any help would be welcome

PS Although the example I have given here is of Merge Sort, my actual problem is
some thing else so I cannot used the algorithm of external merge sort. 

Best Regards from Buffalo

Abhishek Agrawal

SUNY- Buffalo

View raw message