Return-Path: Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: (qmail 61538 invoked from network); 8 Dec 2010 22:47:14 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 8 Dec 2010 22:47:14 -0000 Received: (qmail 71527 invoked by uid 500); 8 Dec 2010 22:47:13 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 71464 invoked by uid 500); 8 Dec 2010 22:47:13 -0000 Mailing-List: contact hdfs-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-user@hadoop.apache.org Delivered-To: mailing list hdfs-user@hadoop.apache.org Received: (qmail 71456 invoked by uid 99); 8 Dec 2010 22:47:13 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Dec 2010 22:47:13 +0000 X-ASF-Spam-Status: No, hits=-0.1 required=10.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of dean.hiller@broadridge.com designates 64.18.2.6 as permitted sender) Received: from [64.18.2.6] (HELO exprod7og117.obsmtp.com) (64.18.2.6) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Dec 2010 22:47:05 +0000 Received: from source ([167.212.2.180]) (using TLSv1) by exprod7ob117.postini.com ([64.18.6.12]) with SMTP ID DSNKTQAK1J6RfI+U33+r0gp6dhU60/cN6GoI@postini.com; Wed, 08 Dec 2010 14:46:44 PST Received: from JOSQHUBA01.jsq.bsg.ad.adp.com (josqhuba01 [149.83.60.74]) by mail.broadridge.com (8.13.8/8.13.8) with ESMTP id oB8Mkhjn8441868 for ; Wed, 8 Dec 2010 17:46:43 -0500 Received: from DNVREMSA01.jsq.bsg.ad.adp.com ([206.88.42.250]) by JOSQHUBA01.jsq.bsg.ad.adp.com with Microsoft SMTPSVC(6.0.3790.4675); Wed, 8 Dec 2010 17:46:43 -0500 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01CB9729.C6B80919" X-MimeOLE: Produced By Microsoft Exchange V6.5 Subject: wordcount example using local file system instead of distributed one? Date: Wed, 8 Dec 2010 15:46:39 -0700 Message-ID: <7CA0D5FE7FA83048893E9230C1E9C0280B4A6577@DNVREMSA01.jsq.bsg.ad.adp.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: wordcount example using local file system instead of distributed one? Thread-Index: AcuXKcbDP5n2fxfgRUaY55/FGxsFPA== From: "Hiller, Dean (Contractor)" To: X-OriginalArrivalTime: 08 Dec 2010 22:46:43.0715 (UTC) FILETIME=[C950A530:01CB9729] This is a multi-part message in MIME format. ------_=_NextPart_001_01CB9729.C6B80919 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable I run the following wordcount example(my hadoop shell seems to always hit the local file system first so I had to add the hdfs...is that normal??...I mean, I see it printing configDir=3D which is where I moved the config dir and what I set the env var too which has the location in the config files there but it still hits the local). =20 [root@localhost hadoop]# ./bin/hadoop jar hadoop-0.20.2-examples.jar wordcount hdfs://206.88.43.8:54310/wordcount hdfs://206.88.43.168:54310/wordcount-out =20 configDir=3D/mnt/mucho/hadoop-config/ classpath=3D/opt/hbase-install/hbase/hbase-0.20.6.jar:/opt/hbase-install/= h base/hba se-0.20.6-test.jar:/mnt/mucho/hbase-config/:/opt/hbase-install/hbase/lib /zookeep er-3.2.2.jar 10/12/08 08:42:33 INFO input.FileInputFormat: Total input paths to process : 13 org.apache.hadoop.ipc.RemoteException: java.io.FileNotFoundException: File file: /tmp/hadoop-root/mapred/system/job_201012080654_0010/job.xml does not exist. at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSys tem.java:361) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem. java:245) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:192) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:142) at org.apache.hadoop.fs.LocalFileSystem.copyToLocalFile(LocalFileSystem. java:61) at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1197) =20 at org.apache.hadoop.mapred.JobInProgress.(JobInProgress.java:257) =20 at org.apache.hadoop.mapred.JobInProgress.(JobInProgress.java:234) =20 at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:2993) =20 In case it helps, here is my hdfs-site.xml that is used by the daemons started AND the client(is that an issue...using the same one)... fs.default.name hdfs://206.88.43.168:54310 hadoop.tmp.dir /opt/data/hadooptmp dfs.data.dir /opt/data/hadoop dfs.replication 2 =20 =20 This message and any attachments are intended only for the use of the add= ressee and may contain information that is privileged and confidential. If the reade= r of the = message is not the intended recipient or an authorized representative of = the intended recipient, you are hereby notified that any dissemination of thi= s communication is strictly prohibited. If you have received this communica= tion in error, please notify us immediately by e-mail and delete the message and = any attachments from your system. =0D ------_=_NextPart_001_01CB9729.C6B80919 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

I run the = following wordcount example(my hadoop shell seems to always hit the = local file system first so I had to add the hdfs…is that = normal??...I mean, I see it printing configDir=3D which is where I moved = the config dir and what I set the env var too which has the location in = the config files there but it still hits the local).

 

[root@localhost hadoop]# ./bin/hadoop jar = hadoop-0.20.2-examples.jar wordcount

hdfs://206.88.43.8:54310/wordcount = hdfs://206.88.43.168:54310/wordcount-out

 

configDir=3D/mnt/mucho/hadoop-config/

classpath=3D/opt/hbase-install/hbase/hbase-0.20.6.jar:/= opt/hbase-install/hbase/hba

se-0.20.6-test.jar:/mnt/mucho/hbase-config/:/opt/hbase-= install/hbase/lib/zookeep

er-3.2.2.jar

10/12/08 08:42:33 INFO input.FileInputFormat: Total = input paths to process : 13

org.apache.hadoop.ipc.RemoteException: = java.io.FileNotFoundException: File file:

/tmp/hadoop-root/mapred/system/job_201012080654_0010/jo= b.xml does not exist.

        at = org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSys

tem.java:361)

        at = org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.

java:245)

        at = org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:192)

        at = org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:142)

        at = org.apache.hadoop.fs.LocalFileSystem.copyToLocalFile(LocalFileSystem.

java:61)

        at = org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1197)

 

        at = org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:25= 7)

 

        at = org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:23= 4)

 

        at = org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:2993)<= /o:p>

 

In case it helps, here is my hdfs-site.xml that is = used by the daemons started AND the client(is that an issue…using = the same one)…

<configuration>

<property>

  = <name>fs.default.name</name>

  = <value>hdfs://206.88.43.168:54310</value>

</property>

<property>

  = <name>hadoop.tmp.dir</name>

  = <value>/opt/data/hadooptmp</value>

</property>

<property>

  = <name>dfs.data.dir</name>

  = <value>/opt/data/hadoop</value>

</property>

<property>

  = <name>dfs.replication</name>

  <value>2</value>

</property>

</configuration>

 

 

This message and any attachments are intended only for the use of th=
e addressee and
may contain information that is privileged and confidential. If the reade=
r of the =

message is not the intended recipient or an authorized representative of =
the
intended recipient, you are hereby notified that any dissemination of thi=
s
communication is strictly prohibited. If you have received this communica=
tion in
error, please notify us immediately by e-mail and delete the message and =
any
attachments from your system.
=0D
------_=_NextPart_001_01CB9729.C6B80919--