Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: <a5accba30909271939i7760bbe2wa05528ffc1daf9@mail.gmail.com>
References: <a5accba30909252334h782536e5v228e4abed81929a4@mail.gmail.com>
	<BCF6C174-3420-4B3C-8385-85E671D48B56@cloudfactory.org>
 <a5accba30909271939i7760bbe2wa05528ffc1daf9@mail.gmail.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Mon, 28 Sep 2009 10:36:25 -0700
Message-ID: <45f85f70909281036o58e86e05v80b9677b37a3751@mail.gmail.com>
Subject: Re: Where are temp files stored?
To: common-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485f7c820d5e81c0474a6bdee

--001485f7c820d5e81c0474a6bdee
Content-Type: text/plain; charset=ISO-8859-1

On Sun, Sep 27, 2009 at 7:39 PM, Starry SHI <starrysl@gmail.com> wrote:

> Hi Dave.
>
> Thank you for your reply!
>
> I have checked {dfs.data.dir}/tmp, the tmp files are there while the job is
> running. However, it seems that the tmp files on each node are the same.
> That is to say, the whole HDFS is sharing the same tmp files. This looks
> strange, because each node shou
>
ld process its own part of data. Do you have
> some ideas on this point?
>

The mapreduce intermediate data is stored in mapred.local.dir. The default
value for this is hadoop.tmp.dir/mapred/local. Note that it is cleaned up
after jobs finish executing.

-Todd


> /* Tomorrow is another day. So is today. */
>
>
> On Sat, Sep 26, 2009 at 15:07, dave bayer <daveb@cloudfactory.org> wrote:
>
> >
> > On Sep 25, 2009, at 11:34 PM, Starry SHI wrote:
> >
> >  Hi.
> >>
> >> I am wondering where the temp files (intermediate files) are stored.
> They
> >> should be located in the hadoop.tmp.dir by default, right? why I cannot
> >> find
> >> them in either the local file system and hdfs?
> >>
> >
> > You might look under ${dfs.data.dir}/tmp. Granted, I've not consulted the
> > code to verify that is how the path is built, but that is where I've seen
> > them on my cluster...
> >
> >  Another question is about the replication of the intermediate files. By
> >> default, will the intermediate (tmp) files be written to HDFS?
> >>
> >
> > No, they live on the node that processed the map task. You wouldn't
> > want to spend the cycles/time to do multiple replication of this data out
> > to other nodes (and then cleanup it up) when you can rerun the task if
> > the node holding the data happens to go down (unlikely).
> >
> > dave bayer
> >
>

--001485f7c820d5e81c0474a6bdee--