hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Nguyen <andrew-lists-had...@ucsfcti.org>
Subject Re: Setting up a second cluster and getting a weird issue
Date Sat, 15 May 2010 00:13:31 GMT
My hdfs-site.xml file:

 11 <configuration>
 12   <property>
 13     <name>dfs.replication</name>
 14     <value>3</value>
 15   </property>
 16   <property>
 17     <name>dfs.name.dir</name>
 18     <value>/srv/hadoop/dfs.name.dir</value>
 19   </property>
 20   <property>
 21     <name>dfs.data.dir</name>
 22     <value>/srv/hadoop/dfs/1</value>
 23   </property>
 24 </configuration>

Here is my /srv/hadoop/hadoop directory listing:

total 5068
drwxr-xr-x  2 hadoop hadoop    4096 2010-05-12 16:10 bin
-rw-rw-r--  1 hadoop hadoop   73847 2010-03-21 23:17 build.xml
drwxr-xr-x  5 hadoop hadoop    4096 2010-03-21 23:17 c++
-rw-rw-r--  1 hadoop hadoop  348624 2010-03-21 23:17 CHANGES.txt
drwxr-xr-x  4 hadoop hadoop    4096 2010-05-12 09:29 cloudera
lrwxrwxrwx  1 hadoop hadoop      15 2010-05-12 15:54 conf -> ../hadoop-conf/
drwxr-xr-x 15 hadoop hadoop    4096 2010-03-21 23:17 contrib
drwxr-xr-x  9 hadoop hadoop    4096 2010-05-12 09:29 docs
drwxr-xr-x  3 hadoop hadoop    4096 2010-03-21 23:17 example-confs
-rw-rw-r--  1 hadoop hadoop    6839 2010-03-21 23:17 hadoop-0.20.2+228-ant.jar
-rw-rw-r--  1 hadoop hadoop 2806445 2010-03-21 23:17 hadoop-0.20.2+228-core.jar
-rw-rw-r--  1 hadoop hadoop  142466 2010-03-21 23:17 hadoop-0.20.2+228-examples.jar
-rw-rw-r--  1 hadoop hadoop 1637240 2010-03-21 23:17 hadoop-0.20.2+228-test.jar
-rw-rw-r--  1 hadoop hadoop   70090 2010-03-21 23:17 hadoop-0.20.2+228-tools.jar
drwxr-xr-x  2 hadoop hadoop    4096 2010-05-12 09:29 ivy
-rw-rw-r--  1 hadoop hadoop    9103 2010-03-21 23:17 ivy.xml
drwxr-xr-x  5 hadoop hadoop    4096 2010-05-12 09:29 lib
-rw-rw-r--  1 hadoop hadoop   13366 2010-03-21 23:17 LICENSE.txt
lrwxrwxrwx  1 hadoop hadoop       8 2010-05-12 16:28 logs -> ../logs/
drwxr-xr-x  3 hadoop hadoop    4096 2010-05-12 16:16 logs-old
-rw-rw-r--  1 hadoop hadoop     101 2010-03-21 23:17 NOTICE.txt
lrwxrwxrwx  1 hadoop hadoop       7 2010-05-12 16:28 pids -> ../pids
drwxr-xr-x  2 hadoop hadoop    4096 2010-05-12 16:10 pids-old
-rw-rw-r--  1 hadoop hadoop    1366 2010-03-21 23:17 README.txt
drwxr-xr-x 15 hadoop hadoop    4096 2010-05-12 09:29 src
drwxr-xr-x  8 hadoop hadoop    4096 2010-03-21 23:17 webapps

The only NFS shared directories are /srv/hadoop/hadoop and /srv/hadoop/hadoop-conf

On May 14, 2010, at 1:06 PM, Andrew Nguyen wrote:

> I'm pretty sure I just set my dfs.data.dir to be /srv/hadoop/dfs/1
> 
> <property>
> <name>dfs.data.dir</name>
> <value>/srv/hadoop/dfs/1</value>
> </property>
> 
> I don't have hadoop.tmp.dir set to anything so it's whatever the default is.
> 
> I don't have access to the cluster right now but will update with the exact settings
when I get a chance.
> 
> I have 4 slaves with identical hardware.  Each has a separate SCSI drive mounted at /srv/hadooop/dfs/1.
 The same config file is used across all the slaves.  I know the NFS approach isn't ideal
for larger deployments but right now, I'm still in the tweaking stage and figured NFS was
the fastest way to propagate changes.
> 
> Thanks!
> 
> On May 14, 2010, at 9:17 AM, Allen Wittenauer wrote:
> 
>> 
>> On May 14, 2010, at 8:53 AM, Andrew Nguyen wrote:
>> 
>>> Just to be clear, I'm only sharing the Hadoop binaries and config files via NFS.
 I don't see how this would cause a conflict - do you have any additional information?
>>> 
>>> The referenced path in the error below (/srv/hadoop/dfs/1) is not being shared
via NFS...
>>>>>>> 2010-05-12 17:17:15,094 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
java.io.FileNotFoundException: /srv/hadoop/dfs/1/current/VERSION (No such file or directory)
>> 
>>>>>>> 
>>>>>>> There are 4 slaves and sometimes 1 or 2 have the error but the
specific nodes change.  Sometimes it's slave1, sometimes it's slave4, etc.
>>>>>>> 
>>>>>>> Any thoughts?
>> 
>> Something is deleting the contents of /srv/hadoop/dfs/1.  How did you set your dfs.data.dir
in the config file?  Or did you just change hadoop.tmp.dir?
>> 
>> 
> 


Mime
View raw message