Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:in-reply-to:accept-language:content-language:
	x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:content-type:mime-version;
	b=ZQ4IZWqq79N0qHC7TGulySBlJ1k6PDSiHYMLdyjb+7kajr4tGEbkCaNQA/5pV/5J
From: Amogh Vasekar <amogh@yahoo-inc.com>
To: "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>
Date: Mon, 23 Nov 2009 13:16:24 +0530
Subject: Re: Saving Intermediate Results from the Mapper
Thread-Topic: Saving Intermediate Results from the Mapper
Thread-Index: Acpr5iyjz5UyZX50QJmHr1aWFWsIhgAKuFx1
Message-ID: <C7303BA8.42B3%amogh@yahoo-inc.com>
In-Reply-To: <8211a1320911221838n6c023d58ye7aa402beec99c26@mail.gmail.com>
Accept-Language: en-US
Content-Language: en
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C7303BA842B3amoghyahooinccom_"
MIME-Version: 1.0

--_000_C7303BA842B3amoghyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Hi,
keep.tasks.files.pattern is what you need, as the name suggests its a patte=
rn match on intermediate outputs generated.

Wrt to copying map data to hdfs, your mappers close() method should help yo=
u achieve this, but might slow up your tasks.

Amogh


On 11/23/09 8:08 AM, "Jeff Zhang" <zjffdu@gmail.com> wrote:

Hi Jason,

which option is for setting disable the removal of intermediate data ?

Thank you

Jeff Zhang


On Mon, Nov 23, 2009 at 10:27 AM, Jason Venner <jason.hadoop@gmail.com>wrot=
e:

> You can manually write the map output to a new file, there are a number o=
f
> examples of opening a sequence file and writing to it on the web or in th=
e
> example code for various hadoop books.
>
> You can also disable the removal of intermediate data, which will result =
in
> potentially large amounts of data being left in the mapred.local.dir.
>
>
> On Sun, Nov 22, 2009 at 3:56 PM, Gordon Linoff <glinoff@gmail.com> wrote:
>
> > I am starting to learn Hadoop, using the Yahoo virtual machine with
> version
> > 0.18.
> >
> > My question is rather simple.  I would like to execute a map/reduce job=
.
> >  In
> > addition to getting the results from the reduce, I would also like to
> save
> > the intermediate results from the map in another HDFS file.  Is this
> > possible?
> >
> > --gordon
> >
>
>
>
> --
> Pro Hadoop, a book to guide you from beginner to hadoop mastery,
> http://www.amazon.com/dp/1430219424?tag=3Djewlerymall
> www.prohadoopbook.com a community for Hadoop Professionals
>


--_000_C7303BA842B3amoghyahooinccom_--