hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: Where are temp files stored?
Date Mon, 28 Sep 2009 17:36:25 GMT
On Sun, Sep 27, 2009 at 7:39 PM, Starry SHI <starrysl@gmail.com> wrote:

> Hi Dave.
>
> Thank you for your reply!
>
> I have checked {dfs.data.dir}/tmp, the tmp files are there while the job is
> running. However, it seems that the tmp files on each node are the same.
> That is to say, the whole HDFS is sharing the same tmp files. This looks
> strange, because each node shou
>
ld process its own part of data. Do you have
> some ideas on this point?
>

The mapreduce intermediate data is stored in mapred.local.dir. The default
value for this is hadoop.tmp.dir/mapred/local. Note that it is cleaned up
after jobs finish executing.

-Todd


> /* Tomorrow is another day. So is today. */
>
>
> On Sat, Sep 26, 2009 at 15:07, dave bayer <daveb@cloudfactory.org> wrote:
>
> >
> > On Sep 25, 2009, at 11:34 PM, Starry SHI wrote:
> >
> >  Hi.
> >>
> >> I am wondering where the temp files (intermediate files) are stored.
> They
> >> should be located in the hadoop.tmp.dir by default, right? why I cannot
> >> find
> >> them in either the local file system and hdfs?
> >>
> >
> > You might look under ${dfs.data.dir}/tmp. Granted, I've not consulted the
> > code to verify that is how the path is built, but that is where I've seen
> > them on my cluster...
> >
> >  Another question is about the replication of the intermediate files. By
> >> default, will the intermediate (tmp) files be written to HDFS?
> >>
> >
> > No, they live on the node that processed the map task. You wouldn't
> > want to spend the cycles/time to do multiple replication of this data out
> > to other nodes (and then cleanup it up) when you can rerun the task if
> > the node holding the data happens to go down (unlikely).
> >
> > dave bayer
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message