Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1895810016 for ; Thu, 15 Aug 2013 04:14:41 +0000 (UTC) Received: (qmail 46522 invoked by uid 500); 15 Aug 2013 04:14:35 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 46144 invoked by uid 500); 15 Aug 2013 04:14:34 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 46137 invoked by uid 99); 15 Aug 2013 04:14:33 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Aug 2013 04:14:33 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,MSGID_FROM_MTA_HEADER,RCVD_IN_DNSWL_NONE X-Spam-Check-By: apache.org Received-SPF: error (nike.apache.org: local policy) Received: from [65.55.111.94] (HELO blu0-omc2-s19.blu0.hotmail.com) (65.55.111.94) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Aug 2013 04:14:25 +0000 Received: from BLU0-SMTP97 ([65.55.111.73]) by blu0-omc2-s19.blu0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675); Wed, 14 Aug 2013 21:13:43 -0700 X-TMN: [coLQzjI0RtpMcDu0nUIpoVDvLzAuSnWM] X-Originating-Email: [francis.hu@reachjunction.com] Message-ID: Received: from testPC ([116.23.186.141]) by BLU0-SMTP97.phx.gbl over TLS secured channel with Microsoft SMTPSVC(6.0.3790.4675); Wed, 14 Aug 2013 21:13:34 -0700 From: Francis.Hu To: References: In-Reply-To: Subject: =?gb2312?B?tPC4tDogUGFzc2luZyBhbiBvYmplY3QgaW4gbWFwcGVy?= Date: Thu, 15 Aug 2013 12:13:09 +0800 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_0192_01CE99B0.D2C32020" X-Mailer: Microsoft Office Outlook 12.0 Thread-Index: Ac6ZZzA4Xhx4cLcHRoOQnKQzCzOXaAABNYQA Content-Language: en-us x-cr-hashedpuzzle: CJ0V Cr0H C+Dq D6gP EDb6 EWbN E7ph E9rH GIgE GIzk GfhJ G19E HcQL Hj8E H659 LSF4;1;dQBzAGUAcgBAAGgAYQBkAG8AbwBwAC4AYQBwAGEAYwBoAGUALgBvAHIAZwA=;Sosha1_v1;7;{C9657A86-122B-435D-AA4B-6AC6F4ED437B};ZgByAGEAbgBjAGkAcwAuAGgAdQBAAHIAZQBhAGMAaABqAHUAbgBjAHQAaQBvAG4ALgBjAG8AbQA=;Thu, 15 Aug 2013 04:13:07 GMT;VHsNWToAIABQAGEAcwBzAGkAbgBnACAAYQBuACAAbwBiAGoAZQBjAHQAIABpAG4AIABtAGEAcABwAGUAcgA= x-cr-puzzleid: {C9657A86-122B-435D-AA4B-6AC6F4ED437B} X-OriginalArrivalTime: 15 Aug 2013 04:13:36.0561 (UTC) FILETIME=[D04E7210:01CE996D] X-Virus-Checked: Checked by ClamAV on apache.org ------=_NextPart_000_0192_01CE99B0.D2C32020 Content-Type: text/plain; charset="gb2312" Content-Transfer-Encoding: quoted-printable hi,=20 =20 what I did when data need to be saved into DB : =20 //configure DB connection info. DBConfiguration.configureDB(config, HadoopJobConfigParser.getProperty(DB_DRIVER), HadoopJobConfigParser.getProperty(DB_URL), HadoopJobConfigParser.getProperty(DB_USER), HadoopJobConfigParser.getProperty(DB_PWD)); =20 //write a OutputKeyClass like below: public class DisplayRequestRecord implements DBWritable, WritableComparable{ //TODO implement write() and readFiels() methods } =20 //set the job's output job.setOutputKeyClass(DisplayRequestRecord.class); job.setOutputValueClass(DisplayRequestRecord.class); DBOutputFormat.setOutput(job, tableName(),Fields()); =20 //define reducer class like below: public class DBDisplayRequestReducerNew extends Reducer{ public void reduce(Text key, Iterable values, Context context) throws IOException, InterruptedException { DisplayRequestRecord dr =3D new DisplayRequestRecord(); context.write(dr,dr); } } =20 =20 Hope it helps! =20 Thanks, Francis.Hu =20 =B7=A2=BC=FE=C8=CB: Sivaram RL [mailto:sivaramrls@gmail.com]=20 =B7=A2=CB=CD=CA=B1=BC=E4: Thursday, August 15, 2013 11:26 =CA=D5=BC=FE=C8=CB: user@hadoop.apache.org =D6=F7=CC=E2: Re: Passing an object in mapper =20 Hi, =20 In the Configuration object in your driver class you can set the = properties as key value pair. This configuration object will be set in the Job = Object. The same properties can be accessed in the mapper/reducer using the = Context Object -> getConfiguration() -> get(propertyName). =20 Hope this helps. =20 Regards, Sivaram R L =20 On Thu, Aug 15, 2013 at 8:39 AM, jamal sasha = wrote: Hi, I am initializing an object in driver code. For sake of argument let say I want to save data to some database.. say: Connection con =3D new Connection(host, db); Now, in reducer I want to do something like con.write(key,value)=20 =20 So, how do i pass this object from driver to mapper / reducer/? Any clue? suggestions? Thanks =20 ------=_NextPart_000_0192_01CE99B0.D2C32020 Content-Type: text/html; charset="gb2312" Content-Transfer-Encoding: quoted-printable

hi,

 

what I did when data need to be saved into DB = :

 

//configure DB connection info.

DBConfigurati= on.configureDB(config,

  &= nbsp;                 = HadoopJobConfigParser.getProperty(DB_DRIVER),

  &= nbsp;                 = HadoopJobConfigParser.getProperty(DB_URL),

  &= nbsp;                 = HadoopJobConfigParser.getProperty(DB_USER),

        =             = HadoopJobConfigParser.getProperty(DB_PWD));

 

//write a OutputKeyClass like = below:

public class = DisplayRequestRecord implements DBWritable, = WritableComparable<DisplayRequestRecord>{

  &= nbsp; //TODO implement write() and readFiels() methods

}<= /span>

 

//set the job's output

job.setOutput= KeyClass(DisplayRequestRecord.class);

job.setOutputValueClass(DisplayRequestRecord.class);

DBOutputForma= t.setOutput(job, tableName(),Fields());

 

//define reducer class like below:

public class DBDisplayRequestReducerNew extends Reducer<Text, DoubleWritable, DisplayRequestRecord, DisplayRequestRecord>{

  &= nbsp; public = void= reduce(Text key, Iterable<DoubleWritable> values,

  &= nbsp;         Context context) throws IOException, InterruptedException = {

  &= nbsp;     DisplayRequestRecord dr =3D new DisplayRequestRecord();

  &= nbsp;     = context.write(dr,dr);

  &= nbsp; }

}<= /span>

 

 

Hope it helps!

 

Thanks,

Francis.Hu

 

=B7=A2=BC=FE=C8=CB: Sivaram RL [mailto:sivaramrls@gmail.com]
=B7=A2=CB=CD=CA=B1=BC= =E4: = Thursday, August 15, 2013 11:26
=CA=D5=BC=FE=C8=CB: = user@hadoop.apache.org
=D6=F7=CC=E2: Re: = Passing an object in mapper

 

Hi,

 

In the Configuration object in = your driver class you can set the properties as key value pair. This configuration = object will be set in the Job Object. The same properties can be accessed in = the mapper/reducer using the Context Object -> getConfiguration() -> get(propertyName).

 

Hope this = helps.

 

Regards,

Sivaram R = L

 

On Thu, Aug 15, 2013 at 8:39 AM, = jamal sasha <jamalshasha@gmail.com> wrote:

Hi,

   I am initializing = an object in driver code.

For sake of argument let say I = want to save data to some database..

say:

Connection con =3D new = Connection(host, db);

Now, in reducer I want to do = something like

con.write(key,value) 

 

So, how do i pass this object = from driver to mapper / reducer/?
Any clue? suggestions?

Thanks

 

------=_NextPart_000_0192_01CE99B0.D2C32020--