hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amandeep Khurana <ama...@gmail.com>
Subject Re: Import the results into SimpleDB
Date Wed, 12 May 2010 02:18:52 GMT
Mark,

You can do it either ways. Create the connection object for the database in
the configure() or setup() method of the mapper (depending on which api you
are using) and insert the record from the mapper function. You dont have to
have a reducer. If you create an output format, the mapper can directly
write to it. In essence you'll be doing the same thing. Its easier to create
an output format if you'll be writing more of such code.

-Amandeep


Amandeep Khurana
Computer Science Graduate Student
University of California, Santa Cruz


On Tue, May 11, 2010 at 7:15 PM, Mark Kerzner <markkerzner@gmail.com> wrote:

> Hi, Nick,
>
> should I then Provide the RecordWriter implementation in the OutputFormat,
> which will connect to the database and write a record to it, instead of to
> HDFS?
>
> Thank you,
> Mark
>
> On Tue, May 11, 2010 at 9:08 PM, Jones, Nick <nick.jones@amd.com> wrote:
>
> > Hi Mark,
> > It would be better to create an outputformat instead of directly
> connecting
> > from the mapper. The outputformat would be called regardless of the
> > existence of the reducers.
> >
> > Make sure and set the job setNumReduceTasks(0). (I'm not sure setting the
> > class to null would work.)
> >
> > Nick
> > Sent by radiation.
> >
> > ----- Original Message -----
> > From: Mark Kerzner <markkerzner@gmail.com>
> > To: core-user@hadoop.apache.org <core-user@hadoop.apache.org>
> > Sent: Tue May 11 21:02:05 2010
> > Subject: Import the results into SimpleDB
> >
> > Hi,
> >
> > I want a Hadoop job that will simply take each line of the input text
> file
> > and store it (after parsing) in a database, like SimpleDB.
> >
> > Can I put this code into Mapper, make no call to "collect" in it, and
> have
> > no reducers at all? Do I set the reduce class to
> > null, conf.setReducerClass(null)? or not set it at all?
> >
> > Thank you,
> > Mark
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message