hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Venner <jason.had...@gmail.com>
Subject Re: Writing different output types from map.
Date Tue, 22 Dec 2009 14:43:32 GMT
Your choice is as Edward says, write a wrapper class to hold all of the
objects you wish to write, or to write multiple files by manually opening
additional output files and writing the particular objects to their own
file.

On Thu, Dec 10, 2009 at 11:31 AM, Edward Capriolo <edlinuxguru@gmail.com>wrote:

> On Thu, Dec 10, 2009 at 4:07 AM, Mehul Sutariya
> <mehulgsutariya@gmail.com> wrote:
> > Hello everyone,
> >
> > I am trying to write different output types from mapper. For example, say
> I
> > have 3 classes A,B and C such that:
> >
> > - C extends A and
> > - B extends A
> >
> > and my mapper is:
> > Mapper<K1, V1, K2, A>
> > Reducer<K2, A, K3, V3>
> >
> > Now, in my map function, when I try to write a record, where the value is
> an
> > instance of B or C, Hadoop framework throws an exception because the
> > framework does some sort of interface checking. I understand the problem
> > with doing this is that there would be no way for the reducer to know
> what
> > specialized type of instance it is, when it is reading the records that
> map
> > stage wrote using the readFields() method and hence I get a Type mismatch
> > error.
> >
> > Has anyone felt the need of doing that, or is there a workaround for such
> > type of operations? I would like to know possible alternatives if any.
> >
> > Right now, as a workaround, I am writing converting my records and
> writing
> > it as text as the output of map phase and then parsing the record again
> in
> > reduce phase to generate the objects and then finally write the
> appropriate
> > objects.
> >
> > Thanks,
> > Mehul.
> >
>
> You may want to take a look at how Nutch works. Nutch wrapps different
> objects with a NutchWritable this way the Object Class that gets
> collected is always the same.
>



-- 
Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=jewlerymall
www.prohadoopbook.com a community for Hadoop Professionals

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message