hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Datta, Saurav" <sda...@paypal.com>
Subject Re: Passing instance of a class to Mapper
Date Tue, 13 Oct 2015 20:04:56 GMT
Thanks very much Chris!
I will try it out. Do you have any examples showing this  ?

--


From: Chris Nauroth
Reply-To: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>"
Date: Tuesday, October 13, 2015 at 12:07 PM
To: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>"
Subject: Re: Passing instance of a class to Mapper

Hello Saurav,

You are correct that it generally is not possible to pass an instance of a class directly
to a mapper (or reducer).  This is because the mapper tasks execute on arbitrary nodes in
the Hadoop cluster, running in different JVM processes from the JVM running the client that
submits the job.

A typical solution is for the client to populate the Configuration object with relevant primitive
data type values.

http://hadoop.apache.org/docs/r2.7.1/api/org/apache/hadoop/conf/Configuration.html

This configuration propagates to all map and reduce tasks of the job.  The Mapper can override
the setup function to do one-time initialization at the start of the task.

http://hadoop.apache.org/docs/r2.7.1/api/org/apache/hadoop/mapreduce/Mapper.html#setup(org.apache.hadoop.mapreduce.Mapper.Context)

As part of this one-time initialization, you can read the values back out of the Configuration.
 As I said earlier, these will be only primitive types like String or int.  If it's helpful,
your setup method can use the primitive values read from configuration to reconstruct an instance
of any class that you want.

I hope this helps.

--Chris Nauroth

From: <Datta>, Saurav <sdatta@paypal.com<mailto:sdatta@paypal.com>>
Reply-To: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>" <user@hadoop.apache.org<mailto:user@hadoop.apache.org>>
Date: Monday, October 12, 2015 at 11:14 PM
To: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>" <user@hadoop.apache.org<mailto:user@hadoop.apache.org>>
Subject: Passing instance of a class to Mapper

Hello,

I am trying to pass an instance of a class to a Mapper. However, I understand Hadoop does
not allow this.
Any workaround to make this happen ?

Regards,
Saurav Datta

Data Engineer| Desk - (408)967-7360| Cell - (408)666-1722
Mime
View raw message