hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saliya Ekanayake <esal...@gmail.com>
Subject Re: Writing Bytes from Map and Reduce functions
Date Wed, 31 Mar 2010 13:04:32 GMT
Hi Jiamin,

Thank you for the quick reply. Actually I have done it like this. Also I
have set the output format as SequenceFileOuputFormat. Everything works well
and I get reduced outputs (part files).

Now I want to read them from a separate java program to display the content
as in a GUI. But I couldn't find a way to read these part files. Is this
possible?

Thank you,
Saliya

On Wed, Mar 31, 2010 at 3:15 AM, welman Lu <welmanwenzi@gmail.com> wrote:

> Hi, Saliya,
>
> The data transformation in MapReduce is:
>
> *map*    (k1,v1)        -> list(k2,list(v2))
> *reduce* (k2, list(v2)) -> (k3, list(v3))
>
> The output from map will be sent to reducer as input directly. In your
> recude function, you can only get k2, v2 as input type. So, in your case,
> the type of the data should be:
> k1 = Text | v1 = Text
> k2 = Text | v2 = BytesWritable
> k3 = Text | v3 = BytesWritable
>
> Hence for your code, I think you can write:
> In job configuration:
> JobConf conf = new JobConf(YourClass.class);
> conf.setOutputKeyClass(k3.class);
> conf.setOutputValueClass(v3.class);
>
> then in map class, set the map class as:
> class YourMapClass extends MapReduceBase
>     implements Mapper<k1, v1, k2, v2> {
> ....
> }
>
> If your v3 is different from the v2, then you can in the job configuration
> set
> conf.setMapOutputKeyClass(k2.class);
> conf.setMapOutputValueClass(v2.class);
>
> Hope this can help you!
>
>
> Best Regards
> Jiamin Lu
>
>


-- 
Saliya Ekanayake
http://www.esaliya.blogspot.com
http://www.esaliya.wordpress.com

Mime
View raw message