hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley" <omal...@apache.org>
Subject Re: Global Sorting and Multiple Reducers ?
Date Fri, 12 Nov 2010 18:15:53 GMT
At those data sizes, I'd make sure that you have very few maps and reduces.
Look at the input format that blends multiple files together.

If you have multiple reduces, you need the total order partitioner. It
samples the data to see where to cut the key space to make even cuts between
the reducers. You are guaranteed that all keys less than K0 go to reduce 0,
less than K1 go to reduce 1, and so on.

-- Owen

View raw message