Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 53C1B100A0 for ; Thu, 15 Aug 2013 04:42:19 +0000 (UTC) Received: (qmail 88897 invoked by uid 500); 15 Aug 2013 04:42:13 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 88750 invoked by uid 500); 15 Aug 2013 04:42:10 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 88738 invoked by uid 99); 15 Aug 2013 04:42:08 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Aug 2013 04:42:08 +0000 X-ASF-Spam-Status: No, hits=3.2 required=5.0 tests=FREEMAIL_REPLY,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of jamalshasha@gmail.com designates 209.85.192.177 as permitted sender) Received: from [209.85.192.177] (HELO mail-pd0-f177.google.com) (209.85.192.177) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Aug 2013 04:42:04 +0000 Received: by mail-pd0-f177.google.com with SMTP id y10so360706pdj.36 for ; Wed, 14 Aug 2013 21:41:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=URdjhR0FEhOPvQcE6tKGPPAlJl/xUsrPUPv8elJZYAY=; b=HkqyCLHwtt+TtFRb5BeW/yuSWBO6JRqu5WF3vhjiADRPC7CbhKEmrnZkM7pIj9nDF3 rEb1v9GQmhkCVPuzK6ZdFkuWWm6iKIpo8J9S4xnU7VHtpFCQXZ5MS5iHpUWJVC+2suPa KOFIvDCaho35ja0BW6s9UQQDj+4grYFLDoKUGu30g3ZhnRO450AQZuxKZ1tX0e/KbRaE PlpcwO0gTWd/5+L7X3olOo/jp2yRWU3qW72dyh+xNsC8mML5Ukjq0QG8CmdvKy4AxDWu 1u9twUSAiCwdlT5GEyZwGNcnlThk7xlACc9yD0rSN5k8TIow/Yh9vrVn7bBBT6VnkL3z an7Q== MIME-Version: 1.0 X-Received: by 10.68.104.101 with SMTP id gd5mr13582606pbb.132.1376541704299; Wed, 14 Aug 2013 21:41:44 -0700 (PDT) Received: by 10.70.125.137 with HTTP; Wed, 14 Aug 2013 21:41:44 -0700 (PDT) In-Reply-To: References: Date: Wed, 14 Aug 2013 21:41:44 -0700 Message-ID: Subject: =?GB2312?Q?Re=3A_=B4=F0=B8=B4=3A_Passing_an_object_in_mapper?= From: jamal sasha To: "user@hadoop.apache.org" Content-Type: multipart/alternative; boundary=047d7b675f504da87604e3f51717 X-Virus-Checked: Checked by ClamAV on apache.org --047d7b675f504da87604e3f51717 Content-Type: text/plain; charset=GB2312 Content-Transfer-Encoding: quoted-printable Thanks guys :) On Wed, Aug 14, 2013 at 9:13 PM, Francis.Hu w= rote: > hi, **** > > ** ** > > what I did when data need to be saved into DB :**** > > ** ** > > //configure DB connection info.**** > > DBConfiguration.*configureDB*(config,**** > > HadoopJobConfigParser.*getProperty*(*DB_DRIVER*),**** > > HadoopJobConfigParser.*getProperty*(*DB_URL*),**** > > HadoopJobConfigParser.*getProperty*(*DB_USER*),**** > > HadoopJobConfigParser.*getProperty*(*DB_PWD*));**** > > ** ** > > //write a OutputKeyClass like below:**** > > public class DisplayRequestRecord implements DBWritable, > WritableComparable{**** > > //TODO implement write() and readFiels() methods**** > > }**** > > ** ** > > //set the job's output**** > > job.setOutputKeyClass(DisplayRequestRecord.class);**** > > job.setOutputValueClass(DisplayRequestRecord.class);**** > > *DBOutputFormat.setOutput(job, tableName(),Fields());* > > * * > > //define reducer class like below:**** > > public class DBDisplayRequestReducerNew extends Reducer DoubleWritable, DisplayRequestRecord, DisplayRequestRecord>{**** > > *public* *void* reduce(Text key, Iterable values,**** > > Context context) *throws* IOException, InterruptedException {= * > *** > > DisplayRequestRecord dr =3D new DisplayRequestRecord();**** > > context.write(dr,dr);**** > > }**** > > }**** > > ** ** > > ** ** > > Hope it helps!**** > > ** ** > > Thanks,**** > > Francis.Hu**** > > ** ** > > *=B7=A2=BC=FE=C8=CB:* Sivaram RL [mailto:sivaramrls@gmail.com] > *=B7=A2=CB=CD=CA=B1=BC=E4:* Thursday, August 15, 2013 11:26 > *=CA=D5=BC=FE=C8=CB:* user@hadoop.apache.org > *=D6=F7=CC=E2:* Re: Passing an object in mapper**** > > ** ** > > Hi,**** > > ** ** > > In the Configuration object in your driver class you can set the > properties as key value pair. This configuration object will be set in th= e > Job Object. The same properties can be accessed in the mapper/reducer usi= ng > the Context Object -> getConfiguration() -> get(propertyName).**** > > ** ** > > Hope this helps.**** > > ** ** > > Regards,**** > > Sivaram R L**** > > ** ** > > On Thu, Aug 15, 2013 at 8:39 AM, jamal sasha > wrote:**** > > Hi,**** > > I am initializing an object in driver code.**** > > For sake of argument let say I want to save data to some database..**** > > say:**** > > Connection con =3D new Connection(host, db);**** > > Now, in reducer I want to do something like**** > > con.write(key,value) **** > > ** ** > > So, how do i pass this object from driver to mapper / reducer/? > Any clue? suggestions?**** > > Thanks**** > > ** ** > --047d7b675f504da87604e3f51717 Content-Type: text/html; charset=GB2312 Content-Transfer-Encoding: quoted-printable
Thanks guys :)


On Wed, Aug 14, 2013 at 9:13 PM, Francis.Hu <francis.hu@reachjunction.com> wrote:

hi,

&nb= sp;

what I did= when data need to be saved into DB :

&nb= sp;

//configur= e DB connection info.

DBConfiguration.configure= DB(config,

    &nbs= p;               HadoopJobConfigParser.getProp= erty(DB_DRIVER),

    &nbs= p;               HadoopJobConfigParser.getProp= erty(DB_URL),

    &nbs= p;               HadoopJobConfigParser.getProp= erty(DB_USER),

          &nb= sp;         HadoopJobConfigParser.get= Property(DB_PWD));

 

//write a = OutputKeyClass like below:

public class DisplayRequestR= ecord implements DBWritable, WritableComparable<DisplayRequestRecord>{

    //TODO implement write() and readFiels() methods

}

 <= /p>

//set the = job's output

job.setOutputKeyClass(Displa= yRequestRecord.class);

job.setOutputValueClass(DisplayRequestRecord.class);

DBOutputFormat.setOutput(= job, tableName(),Fields());

 

//define reducer class like below:

public class DBDisplayRequestReducerNew extends Reducer<Text, DoubleWritable, DisplayRequestRecord, DisplayRequestRecord>{

    public void reduce(Text key, Iterable<DoubleWritable> values,

    &nbs= p;       Context context) throws IOException, InterruptedException {<= u>

    &nbs= p;   DisplayRequestRecord dr =3D new DisplayRequestRecord();

    &nbs= p;   context.write(dr,dr);

    }<= u>

}

&nb= sp;

&nb= sp;

Hope it he= lps!

&nb= sp;

Thanks,=

Francis.Hu=

&nb= sp;

=B7=A2=BC=FE=C8=CB: Sivaram= RL [mailto:sivar= amrls@gmail.com]
=B7=A2= =CB=CD=CA=B1=BC=E4: Thursday, August 1= 5, 2013 11:26
=CA=D5= =BC=FE=C8=CB: user@hadoop.apache.org
=D6=F7= =CC=E2: Re: Passing an object in mapper

 

Hi,

 

In the Configuration object in = your driver class you can set the properties as key value pair. This configuration obje= ct will be set in the Job Object. The same properties can be accessed in the mapper/reducer using the Context Object -> getConfiguration() -> get(propertyName).

 

Hope this helps.<= /span>

 

Regards,

Sivaram R L

=  

On Thu, Aug 15, 2013 at 8:39 AM= , jamal sasha <jamals= hasha@gmail.com> wrote:

Hi,

   I am initializing = an object in driver code.

For sake of argument let say I = want to save data to some database..

say:

Connection con =3D new Connecti= on(host, db);

Now, in reducer I want to do so= mething like

con.write(key,value) 

 

So, how do i pass this object f= rom driver to mapper / reducer/?
Any clue? suggestions?

Thanks

 


--047d7b675f504da87604e3f51717--