Return-Path: 
 <user-return-15647-apmail-hadoop-mapreduce-user-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 42BEE11361
	for <apmail-hadoop-mapreduce-user-archive@minotaur.apache.org>;
 Thu,  5 Jun 2014 01:59:41 +0000 (UTC)
Received: (qmail 1652 invoked by uid 500); 5 Jun 2014 01:59:36 -0000
Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org
Received: (qmail 1538 invoked by uid 500); 5 Jun 2014 01:59:36 -0000
Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@hadoop.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@hadoop.apache.org>
List-Post: <mailto:user@hadoop.apache.org>
List-Id: <user.hadoop.apache.org>
Reply-To: user@hadoop.apache.org
Delivered-To: mailing list user@hadoop.apache.org
Received: (qmail 1531 invoked by uid 99); 5 Jun 2014 01:59:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Jun 2014 01:59:36 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of jhkwak@kisti.re.kr designates
 150.183.95.33 as permitted sender)
Received: from [150.183.95.33] (HELO mail2.kisti.re.kr) (150.183.95.33)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Jun 2014 01:59:32 +0000
Received: from [150.183.234.35] ([150.183.234.35])
          by mail2.kisti.re.kr ([150.183.95.33])
          with ESMTP id 1401933549.959016.1156491584.mail2
          for <user@hadoop.apache.org>;
          Thu, 05 Jun 2014 10:59:10 +0900 (KST)
Message-ID: <7E342A6BDA214B48B924322135343AB5@JaehyuckKwakPC>
From: "Jae-Hyuck Kwak" <jhkwak@kisti.re.kr>
To: <user@hadoop.apache.org>
Subject: hadoop on diskless cluster
X-TERRACE-DUMMYSUBJECT: Terrace Spam system                                  *
Date: Thu, 5 Jun 2014 10:55:13 +0900
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_NextPart_000_00CB_01CF80AC.A08D78B0"
X-Priority: 3
X-MSMail-Priority: Normal
Importance: Normal
X-Mailer: Microsoft Windows Live Mail 15.4.3555.308
X-MimeOLE: Produced By Microsoft MimeOLE V15.4.3555.308
X-TERRACE-SPAMMARK: NO       (SR:20.61)                     
  (by Terrace)                                                  
X-Virus-Checked: Checked by ClamAV on apache.org

���� �κ����� ������ MIME ������ �޽����Դϴ�.

------=_NextPart_000_00CB_01CF80AC.A08D78B0
Content-Type: text/plain;
	charset="ks_c_5601-1987"
Content-Transfer-Encoding: quoted-printable

Dear all,

I=A1=AFm trying to deploy hadoop on diskless cluster.
I had mapred.local.dir to the shared location on NFS mount.

When I execute TestDFSIO, I got the following error logs from the =
console.
$ hadoop jar hadoop-test-1.2.2-SNAPSHOT.jar TestDFSIO -write -nrFiles 2 =
-fileSize 10
...
java.lang.RuntimeException: java.io.FileNotFoundException: =
/home/hadoopadmin/mapred_scratch/taskTracker/hadoopadmin/jobcache/job_201=
406050112_0002/job.xml (No such file or directory)
        at =
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1243=
)
        at =
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:111=
7)
        at =
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1053)
        at =
org.apache.hadoop.conf.Configuration.get(Configuration.java:397)
        at =
org.apache.hadoop.mapred.JobConf.checkAndWarnDeprecation(JobConf.java:189=
9)
        at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:399)
        at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:389)
        at org.apache.hadoop.mapred.Child.main(Child.java:196)
Caused by: java.io.FileNotFoundException: =
/home/hadoopadmin/mapred_scratch/taskTracker/hadoopadmin/jobcache/job_201=
406050112_0002/job.xml (No such file or directory)
        at java.io.FileInputStream.open(Native Method)
        at java.io.FileInputStream.<init>(FileInputStream.java:120)
        at =
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1167=
)
        ... 7 more
...

It works fine when I use separate directory for mapred.local.dir.
But, I want it on shared directory due to diskless cluster.

Is there any ideas how to do that?
Any kind of your ideas will be appreciated.

FYI, the following is my configuration.=20
/home directory is shared to all the nodes by NFS mount.
hdfs-site.xml is no needed (I used NFS, not HDFS).

core-site.xml:
<configuration>
  <property>
    <name>fs.default.name</name>
    <value>file:///home/hadoopadmin</value>
  </property>
  <property>
    <name>hadoop.tmp.dir</name>
    <value>/home/${user.name}/tmp_data</value>
  </property>
</configuration>

mapred-site.xml:
<configuration>
  <property>
    <name>mapred.job.tracker</name>
    <value>192.168.0.101:9001</value>
  </property>
  <property>
    <name>mapred.child.java.opts</name>
    <value>-Xmx1024m</value>
  </property>
  <property>
    <name>mapred.local.dir</name>
    <value>/home/${user.name}/mapred_scratch</value>
    <final>true</final>
  </property>
</configuration>

hdfs-site.xml:
<configuration>
</configuration>

masters:
192.168.0.101

slaves:
192.168.0.102
192.168.0.103
192.168.0.104
192.168.0.105

Cheers,
Jae-Hyuck

------=_NextPart_000_00CB_01CF80AC.A08D78B0
Content-Type: text/html;
	charset="ks_c_5601-1987"
Content-Transfer-Encoding: quoted-printable

<HTML><HEAD></HEAD>
<BODY dir=3Dltr>
<DIV dir=3Dltr>
<DIV style=3D"FONT-FAMILY: 'Calibri'; COLOR: #000000; FONT-SIZE: 12pt">
<DIV>Dear all,</DIV>
<DIV>&nbsp;</DIV>
<DIV>I=A1=AFm trying to deploy hadoop on diskless cluster.</DIV>
<DIV>I had mapred.local.dir to the shared location on NFS mount.</DIV>
<DIV>&nbsp;</DIV>
<DIV>When I execute TestDFSIO, I got the following error logs from the=20
console.</DIV>
<DIV>$ hadoop jar hadoop-test-1.2.2-SNAPSHOT.jar TestDFSIO -write =
-nrFiles 2=20
-fileSize 10</DIV>
<DIV>...</DIV>
<DIV>java.lang.RuntimeException: java.io.FileNotFoundException:=20
/home/hadoopadmin/mapred_scratch/taskTracker/hadoopadmin/jobcache/job_201=
406050112_0002/job.xml=20
(No such file or directory)</DIV>
<DIV>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at=20
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1243=
)</DIV>
<DIV>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at=20
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:111=
7)</DIV>
<DIV>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at=20
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1053)</D=
IV>
<DIV>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at=20
org.apache.hadoop.conf.Configuration.get(Configuration.java:397)</DIV>
<DIV>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at=20
org.apache.hadoop.mapred.JobConf.checkAndWarnDeprecation(JobConf.java:189=
9)</DIV>
<DIV>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at=20
org.apache.hadoop.mapred.JobConf.&lt;init&gt;(JobConf.java:399)</DIV>
<DIV>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at=20
org.apache.hadoop.mapred.JobConf.&lt;init&gt;(JobConf.java:389)</DIV>
<DIV>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at=20
org.apache.hadoop.mapred.Child.main(Child.java:196)</DIV>
<DIV>Caused by: java.io.FileNotFoundException:=20
/home/hadoopadmin/mapred_scratch/taskTracker/hadoopadmin/jobcache/job_201=
406050112_0002/job.xml=20
(No such file or directory)</DIV>
<DIV>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at=20
java.io.FileInputStream.open(Native Method)</DIV>
<DIV>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at=20
java.io.FileInputStream.&lt;init&gt;(FileInputStream.java:120)</DIV>
<DIV>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at=20
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1167=
)</DIV>
<DIV>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ... 7 more</DIV>
<DIV>...</DIV>
<DIV>&nbsp;</DIV>
<DIV>It works fine when I use separate directory for =
mapred.local.dir.</DIV>
<DIV>But, I want it on shared directory due to diskless cluster.</DIV>
<DIV=20
style=3D"FONT-FAMILY: 'Calibri'; COLOR: #000000; FONT-SIZE: =
12pt">&nbsp;</DIV>
<DIV style=3D"FONT-FAMILY: 'Calibri'; COLOR: #000000; FONT-SIZE: =
12pt">Is there=20
any ideas how to do that?</DIV>
<DIV style=3D"FONT-FAMILY: 'Calibri'; COLOR: #000000; FONT-SIZE: =
12pt">Any kind of=20
your ideas will be appreciated.</DIV>
<DIV=20
style=3D"FONT-FAMILY: 'Calibri'; COLOR: #000000; FONT-SIZE: =
12pt">&nbsp;</DIV>
<DIV style=3D"FONT-FAMILY: 'Calibri'; COLOR: #000000; FONT-SIZE: =
12pt">FYI, the=20
following is my configuration. </DIV>
<DIV style=3D"FONT-FAMILY: 'Calibri'; COLOR: #000000; FONT-SIZE: =
12pt">/home=20
directory is shared to all the nodes by NFS mount.</DIV>
<DIV=20
style=3D"FONT-FAMILY: 'Calibri'; COLOR: #000000; FONT-SIZE: =
12pt">hdfs-site.xml is=20
no needed (I used NFS, not HDFS).</DIV>
<DIV=20
style=3D"FONT-FAMILY: 'Calibri'; COLOR: #000000; FONT-SIZE: =
12pt">&nbsp;</DIV>
<DIV=20
style=3D"FONT-FAMILY: 'Calibri'; COLOR: #000000; FONT-SIZE: =
12pt">core-site.xml:</DIV>
<DIV>&lt;configuration&gt;</DIV>
<DIV>&nbsp; &lt;property&gt;</DIV>
<DIV>&nbsp;&nbsp;&nbsp; &lt;name&gt;fs.default.name&lt;/name&gt;</DIV>
<DIV>&nbsp;&nbsp;&nbsp;=20
&lt;value&gt;file:///home/hadoopadmin&lt;/value&gt;</DIV>
<DIV>&nbsp; &lt;/property&gt;</DIV>
<DIV>&nbsp; &lt;property&gt;</DIV>
<DIV>&nbsp;&nbsp;&nbsp; &lt;name&gt;hadoop.tmp.dir&lt;/name&gt;</DIV>
<DIV>&nbsp;&nbsp;&nbsp;=20
&lt;value&gt;/home/${user.name}/tmp_data&lt;/value&gt;</DIV>
<DIV>&nbsp; &lt;/property&gt;</DIV>
<DIV>&lt;/configuration&gt;</DIV>
<DIV=20
style=3D"FONT-FAMILY: 'Calibri'; COLOR: #000000; FONT-SIZE: =
12pt">&nbsp;</DIV>
<DIV=20
style=3D"FONT-FAMILY: 'Calibri'; COLOR: #000000; FONT-SIZE: =
12pt">mapred-site.xml:</DIV>
<DIV>&lt;configuration&gt;</DIV>
<DIV>&nbsp; &lt;property&gt;</DIV>
<DIV>&nbsp;&nbsp;&nbsp; =
&lt;name&gt;mapred.job.tracker&lt;/name&gt;</DIV>
<DIV>&nbsp;&nbsp;&nbsp; =
&lt;value&gt;192.168.0.101:9001&lt;/value&gt;</DIV>
<DIV>&nbsp; &lt;/property&gt;</DIV>
<DIV>&nbsp; &lt;property&gt;</DIV>
<DIV>&nbsp;&nbsp;&nbsp; =
&lt;name&gt;mapred.child.java.opts&lt;/name&gt;</DIV>
<DIV>&nbsp;&nbsp;&nbsp; &lt;value&gt;-Xmx1024m&lt;/value&gt;</DIV>
<DIV>&nbsp; &lt;/property&gt;</DIV>
<DIV>&nbsp; &lt;property&gt;</DIV>
<DIV>&nbsp;&nbsp;&nbsp; &lt;name&gt;mapred.local.dir&lt;/name&gt;</DIV>
<DIV>&nbsp;&nbsp;&nbsp;=20
&lt;value&gt;/home/${user.name}/mapred_scratch&lt;/value&gt;</DIV>
<DIV>&nbsp;&nbsp;&nbsp; &lt;final&gt;true&lt;/final&gt;</DIV>
<DIV>&nbsp; &lt;/property&gt;</DIV>
<DIV>&lt;/configuration&gt;</DIV>
<DIV=20
style=3D"FONT-FAMILY: 'Calibri'; COLOR: #000000; FONT-SIZE: =
12pt">&nbsp;</DIV>
<DIV=20
style=3D"FONT-FAMILY: 'Calibri'; COLOR: #000000; FONT-SIZE: =
12pt">hdfs-site.xml:</DIV>
<DIV>&lt;configuration&gt;</DIV>
<DIV>&lt;/configuration&gt;</DIV>
<DIV=20
style=3D"FONT-FAMILY: 'Calibri'; COLOR: #000000; FONT-SIZE: =
12pt">&nbsp;</DIV>
<DIV=20
style=3D"FONT-FAMILY: 'Calibri'; COLOR: #000000; FONT-SIZE: =
12pt">masters:</DIV>
<DIV=20
style=3D"FONT-FAMILY: 'Calibri'; COLOR: #000000; FONT-SIZE: =
12pt">192.168.0.101</DIV>
<DIV=20
style=3D"FONT-FAMILY: 'Calibri'; COLOR: #000000; FONT-SIZE: =
12pt">&nbsp;</DIV>
<DIV=20
style=3D"FONT-FAMILY: 'Calibri'; COLOR: #000000; FONT-SIZE: =
12pt">slaves:</DIV>
<DIV=20
style=3D"FONT-FAMILY: 'Calibri'; COLOR: #000000; FONT-SIZE: =
12pt">192.168.0.102</DIV>
<DIV=20
style=3D"FONT-FAMILY: 'Calibri'; COLOR: #000000; FONT-SIZE: =
12pt">192.168.0.103</DIV>
<DIV=20
style=3D"FONT-FAMILY: 'Calibri'; COLOR: #000000; FONT-SIZE: =
12pt">192.168.0.104</DIV>
<DIV=20
style=3D"FONT-FAMILY: 'Calibri'; COLOR: #000000; FONT-SIZE: =
12pt">192.168.0.105</DIV>
<DIV=20
style=3D"FONT-FAMILY: 'Calibri'; COLOR: #000000; FONT-SIZE: =
12pt">&nbsp;</DIV>
<DIV=20
style=3D"FONT-FAMILY: 'Calibri'; COLOR: #000000; FONT-SIZE: =
12pt">Cheers,</DIV>
<DIV=20
style=3D"FONT-FAMILY: 'Calibri'; COLOR: #000000; FONT-SIZE: =
12pt">Jae-Hyuck</DIV>
<DIV=20
style=3D"FONT-FAMILY: 'Calibri'; COLOR: #000000; FONT-SIZE: =
12pt">&nbsp;</DIV></DIV></DIV></BODY></HTML>

------=_NextPart_000_00CB_01CF80AC.A08D78B0--