hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hemanth Yamijala <yhema...@gmail.com>
Subject Re: Partitioned Datasets Map/Reduce
Date Tue, 06 Jul 2010 04:40:49 GMT

> I have written my custom partitioner for partitioning datasets. I want  to
> partition two datasets using the same partitioner and then in the  next
> mapreduce job, I want each mapper to handle the same partition from  the two
> sources and perform some function such as joining etc. How I  can I ensure that
> one mapper gets the split that corresponds to same  partition from both the
> sources?

Not really an answer to your specific question, but have you taken a
look at Pig (http://hadoop.apache.org/pig) which is suitable for
operations like Joining data sets ?

View raw message