hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Zhang <zjf...@gmail.com>
Subject Re: Saving Intermediate Results from the Mapper
Date Mon, 23 Nov 2009 02:38:41 GMT
Hi Jason,

which option is for setting disable the removal of intermediate data ?

Thank you

Jeff Zhang


On Mon, Nov 23, 2009 at 10:27 AM, Jason Venner <jason.hadoop@gmail.com>wrote:

> You can manually write the map output to a new file, there are a number of
> examples of opening a sequence file and writing to it on the web or in the
> example code for various hadoop books.
>
> You can also disable the removal of intermediate data, which will result in
> potentially large amounts of data being left in the mapred.local.dir.
>
>
> On Sun, Nov 22, 2009 at 3:56 PM, Gordon Linoff <glinoff@gmail.com> wrote:
>
> > I am starting to learn Hadoop, using the Yahoo virtual machine with
> version
> > 0.18.
> >
> > My question is rather simple.  I would like to execute a map/reduce job.
> >  In
> > addition to getting the results from the reduce, I would also like to
> save
> > the intermediate results from the map in another HDFS file.  Is this
> > possible?
> >
> > --gordon
> >
>
>
>
> --
> Pro Hadoop, a book to guide you from beginner to hadoop mastery,
> http://www.amazon.com/dp/1430219424?tag=jewlerymall
> www.prohadoopbook.com a community for Hadoop Professionals
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message