Return-Path: Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: (qmail 40537 invoked from network); 9 Dec 2010 01:25:37 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 9 Dec 2010 01:25:37 -0000 Received: (qmail 12045 invoked by uid 500); 9 Dec 2010 01:25:37 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 11993 invoked by uid 500); 9 Dec 2010 01:25:37 -0000 Mailing-List: contact hdfs-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-user@hadoop.apache.org Delivered-To: mailing list hdfs-user@hadoop.apache.org Received: (qmail 11985 invoked by uid 99); 9 Dec 2010 01:25:36 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Dec 2010 01:25:36 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [209.85.214.182] (HELO mail-iw0-f182.google.com) (209.85.214.182) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Dec 2010 01:25:32 +0000 Received: by iwn39 with SMTP id 39so2758443iwn.41 for ; Wed, 08 Dec 2010 17:25:11 -0800 (PST) Received: by 10.231.14.136 with SMTP id g8mr26541iba.114.1291857911439; Wed, 08 Dec 2010 17:25:11 -0800 (PST) MIME-Version: 1.0 Received: by 10.231.182.70 with HTTP; Wed, 8 Dec 2010 17:24:51 -0800 (PST) In-Reply-To: References: <7CA0D5FE7FA83048893E9230C1E9C0280B4A662B@DNVREMSA01.jsq.bsg.ad.adp.com> From: Philip Zeyliger Date: Wed, 8 Dec 2010 17:24:51 -0800 Message-ID: Subject: Re: odd temp dir created in hdfs from hdfs-site.xml file setting? To: hdfs-user@hadoop.apache.org Content-Type: multipart/alternative; boundary=0003255745bee962120496f01b69 --0003255745bee962120496f01b69 Content-Type: text/plain; charset=ISO-8859-1 Hadoop's default configurations use hadoop.tmp.dir both for the local file system and HDFS. Specifically, mapred.system.dir is, I believe, a directory on HDFS, and defaults to be a subdirectory of hadoop.tmp.dir. -- Philip [130]air::~(16603)$grep -C 3 hadoop.tmp.dir /home/philip/hadoop/hadoop-0.20.2+737/src/mapred/mapred-default.xml mapred.local.dir ${hadoop.tmp.dir}/mapred/local The local directory where MapReduce stores intermediate data files. May be a comma-separated list of directories on different devices in order to spread disk i/o. -- mapred.system.dir ${hadoop.tmp.dir}/mapred/system The directory where MapReduce stores control files. mapreduce.jobtracker.staging.root.dir ${hadoop.tmp.dir}/mapred/staging The root of the staging area for users' job files In practice, this should be the directory where users' home directories are located (usually /user) -- mapred.temp.dir ${hadoop.tmp.dir}/mapred/temp A shared directory for temporary files. On Wed, Dec 8, 2010 at 5:03 PM, Aaron Eng wrote: > hadoop.tmp.dir is supposed a location the local file system, not a > directory within HDFS. > > That value should also be in core-site.xml instead of hdfs-site.xml. That > might be the cause of your issue. > > > On Wed, Dec 8, 2010 at 4:47 PM, Hiller, Dean (Contractor) < > dean.hiller@broadridge.com> wrote: > >> I have this setting in hdfs-site.xml >> >> >> >> >> >> hadoop.tmp.dir >> >> /opt/data/hadooptmp >> >> >> >> >> >> This then causes a creation of that dir in HDFS(ie. I can see it when >> browsing >> >> >> >> http://206.88.43.4:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=/ >> >> >> >> Is it me, or is that weird/odd? Do I have some kind of other setting >> wrong? Is hadoop.tmp.dir not for local file system like I thought it >> was(ie. The next property I have after that is stored on the local >> filesystem.. >> >> >> >> >> >> dfs.data.dir >> >> /opt/data/hadoop >> >> >> >> >> >> Thanks, >> >> Dean >> >> This message and any attachments are intended only for the use of the addressee and >> may contain information that is privileged and confidential. If the reader of the >> message is not the intended recipient or an authorized representative of the >> intended recipient, you are hereby notified that any dissemination of this >> communication is strictly prohibited. If you have received this communication in >> error, please notify us immediately by e-mail and delete the message and any >> attachments from your system. >> >> >> > --0003255745bee962120496f01b69 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hadoop's default configurations use hadoop.tmp.dir both for the local f= ile system and HDFS. =A0Specifically, mapred.system.dir is, I believe, a di= rectory on HDFS, and defaults to be a subdirectory of hadoop.tmp.dir.

-- Philip

[130]air::~(166= 03)$grep -C 3 hadoop.tmp.dir /home/philip/hadoop/hadoop-0.20.2+737/src/mapr= ed/mapred-default.xml

<property>
=A0=A0<name>mapred.local.dir</name>
=A0=A0<value&g= t;${hadoop.tmp.dir}/mapred/local</value>
=A0=A0<descript= ion>The local directory where MapReduce stores intermediate
= =A0=A0data files. =A0May be a comma-separated list of
=A0=A0directories on different devices in order to spread disk i/o.
--

<property>
=A0=A0<na= me>mapred.system.dir</name>
=A0=A0<value>${hadoop.= tmp.dir}/mapred/system</value>
=A0=A0<description>The directory where MapReduce stores control = files.
=A0=A0</description>
</property>

<property>
=A0=A0<name>mapreduc= e.jobtracker.staging.root.dir</name>
=A0=A0<value>${hadoop.tmp.dir}/mapred/staging</value>
=A0=A0<description>The root of the staging area for users' = job files
=A0=A0In practice, this should be the directory where u= sers' home=A0
=A0=A0directories are located (usually /user)
--
<= br>
<property>
=A0=A0<name>mapred.temp.dir&= lt;/name>
=A0=A0<value>${hadoop.tmp.dir}/mapred/temp<= /value>
=A0=A0<description>A shared directory for temporary files.
=
=A0=A0</description>
</property>

<= /div>

On Wed, Dec 8, 2010 at 5:03 = PM, Aaron Eng <ae= ng@maprtech.com> wrote:
hadoop.tmp.dir is supposed a location the l= ocal file system, not a directory within HDFS.

That value should als= o be in core-site.xml instead of hdfs-site.xml.=A0 That might be the cause = of your issue.


On Wed, Dec 8, 2010 at 4:47 PM, Hiller, Dean (Contractor) <dean.= hiller@broadridge.com> wrote:

I have this setting in hdfs-site.xml

=A0

<= p class=3D"MsoNormal"><property>

=A0 <na= me>hadoop.tmp.dir</name>

=A0 <value>/opt/data/hadooptmp</value>

</property>

=A0

This then causes a creation of that dir in HDFS(ie.= I can see it when browsing

=A0

http://206.88.43.4:50075/browseDirectory.jsp?namenodeInfoPor= t=3D50070&dir=3D/

=A0

Is it me, or is that w= eird/odd?=A0 Do I have some kind of other setting wrong?=A0 Is hadoop.tmp.d= ir not for local file system like I thought it was(ie. The next property I = have after that is stored on the local filesystem..

=A0

<property>

=A0 <name>dfs.data.dir</name>

=A0 <value>/opt/data/hadoop</value>

</property>

=A0

T= hanks,

Dean

This message and any attachments are intended only for the use of the =
addressee and
may contain information that is privileged and confidential. If the reader =
of the=20
message is not the intended recipient or an authorized representative of th=
e
intended recipient, you are hereby notified that any dissemination of this
communication is strictly prohibited. If you have received this communicati=
on in
error, please notify us immediately by e-mail and delete the message and an=
y
attachments from your system.



--0003255745bee962120496f01b69--