hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bhaskar Ghosh <bjgin...@yahoo.co.in>
Subject How to create a composite value object for output from Map method
Date Sun, 19 Sep 2010 10:22:07 GMT
Hi All,

What would be the right approach to solve this problem:
	1. I need to output an object as the value from my map method. The object's 
class should have two mambers: an ArrayList<String> and another, an integer.

I used following two ways, but they are not working:
	* I wrote a class MyCompositeValueWritable that implements Writable interface.  
Inside the overridden readFields and write methods, I try to read/write using 
the ObjectWritable class. [see attached file MyWordCount_ObjVal1_2.java]

	* The custom class is a plain class 'MyCompositeValue' not implementing or 
inheriting anything. The Map and Reduce methods try to output the <key, 
value=<object of MyCompositeValue> > using the ObjectWritable class. [see 
attached file Case2.txt] 

	* Am I going wrong somewhere? Appreciate any help.

	2. I have another problem, in which I need two types of mappers and reducer, 
and I want to execute them in this order:
	* Mapper1 -> Reducer1 -> Mapper2 -> Reducer2
	* Is it possible through ChainMapper and/or ChainReducer classes? It yes, then 
how? Can anybody provide some starting working example, or point me to some good 
url for the same?
	* Currently, I am doing it as a work-around:
	* The first set of Mapper-Reducer write to HDFS. Then the second set of 
Mapper-Reducer pick up that output file from HDFS and writes further processed 
output to another HDFS directory.
	* An example would be really really  helpful.
Thanks
Bhaskar Ghosh

"Ignorance is Bliss... Knowledge never brings Peace!!!"


Mime
View raw message