hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Rosenstrauch <dar...@darose.net>
Subject Re: Partitioners - How to know if they are working
Date Fri, 17 Feb 2012 04:15:58 GMT
On 02/16/2012 12:49 PM, ext-fabio.almeida@nokia.com wrote:
> Hello All,
> I wrote my own partitioner and I would like to see if it's working.
> By printing the return of method getPartition I could see that the partitions were different,
but were they really working? To answer that I got the keys that every reducer task processed
and that was what I expected. It seems my partitioner is working properly. But not easy to
discover though.
> Does anyone know if there is an easier way to see if your customized partitioner is working?
For instance, a counter that shows how many partitioners a map generated or a reducer received?
> Thanks in advance,
> Fabio Almeida

At my last job we wrote a custom partitioner, and we tested it out 
completely outside of Hadoop using standard JUnit unit tests.



View raw message