hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <qwertyman...@gmail.com>
Subject Re: Hadoop startup problem - directory name required
Date Mon, 23 Aug 2010 14:26:57 GMT
Can you confirm that this is the right configuration your NN is starting with?

On Mon, Aug 23, 2010 at 7:19 PM, cliff palmer <palmercliff@gmail.com> wrote:
> Thanks Harsh, but I am still not sure I understand what is going on.
> The directory specified in the dfs.name.dir property,
> "/var/lib/hadoop-0.20/dfsname", does exist and rights to that directory have
> been granted to the OS user that is running the Hadoop startup script.
> The directory mentioned in the error message is
> "/var/lib/hadoop-0.20/cache/hadoop/dfs/name".
> I can create this directory and that would (I assume) remove the error, but
> I want to understand how the name is derived.  It's not a child of the
> directory name specified in the dfs.name.dir property.
> Thanks again!
> Cliff
>
>
> On Mon, Aug 23, 2010 at 9:21 AM, Harsh J <qwertymaniac@gmail.com> wrote:
>
>> Its checking this directory cause your dfs.name.dir (hdfs-site.xml)
>> has it in its list of dirs to write a copy to:
>>
>> <property>
>>               <name>dfs.name.dir</name>
>>               <value>/DFS/dfsname,/var/lib/hadoop-0.20/dfsname</value>
>> <property>
>>
>> Remove it from this property if you don't need it. If you need
>> multiple dirs, you should create the path and allocate proper
>> permissions to it so that it may write to it. Its good to use a second
>> path for backup purposes (most have this on the NFS). The path is so
>> because your distribution of hadoop perhaps placed its files around
>> that place and its up in the conf files as some form of default :)
>>
>> On Mon, Aug 23, 2010 at 6:45 PM, cliff palmer <palmercliff@gmail.com>
>> wrote:
>> > The namenode log for a Hadoop-0.20 installation contains this error
>> message:
>> > "/var/lib/hadoop-0.20/cache/hadoop/dfs/name in in an inconsistent state".
>> > This directory does not exist and I would like to understand why this
>> > particular directory name is required (not what the directory is used
>> for,
>> > but why this particular directory name).  The *-site.xml files are below
>> (IP
>> > addresses have been masked).
>> > Thanks in advance for your help.
>> > Cliff
>> >
>> > core-site.xml:
>> >
>> > <?xml version="1.0"?>
>> > <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>> >
>> > <!-- Put site-specific property overrides in this file. -->
>> >
>> > <configuration>
>> >        <property>
>> >                <name>fs.default.name</name>
>> >                <value>hdfs://xxx.xxx.xxx.xxx:8020</value>
>> >                <final>true</final>
>> >        </property>
>> >        <property>
>> >                <name>hadoop.tmp.dir</name>
>> >                <value>/var/lib/hadoop-0.20/cache/hadoop</value>
>> >        </property>
>> > </configuration>
>> > -------- end of core-site.xml -----------
>> >
>> > hdfs-site.xml:
>> >
>> > <?xml version="1.0"?>
>> > <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>> >
>> > <!-- Put site-specific property overrides in this file. -->
>> >
>> > <configuration>
>> >        <property>
>> >                <name>dfs.replication</name>
>> >                <value>3</value>
>> >        </property>
>> >        <property>
>> >                <name>dfs.name.dir</name>
>> >                <value>/DFS/dfsname,/var/lib/hadoop-0.20/dfsname</value>
>> >        <property>
>> >        </property>
>> >                <name>dfs.data.dir</name>
>> >                <value>/DFS1/dfsdata,/DFS2/dfsdata,/DFS3/dfsdata</value>
>> >        </property>
>> >        <property>
>> >                <name>dfs.permissions</name>
>> >                <value>false</value>
>> >        </property>
>> >        <property>
>> >                <name>dfs.datanode.du.reserved</name>
>> >                <value>20000000000</value>
>> >        </property>
>> >        <property>
>> >                <name>dfs.namenode,plugins</name>
>> >                <value>org.apache.hadoop.thriftfs.NamenodePlugin</value>
>> >        </property>
>> >        <property>
>> >                <name>dfs.datanode.plugins</name>
>> >                <value>org.apache.hadoop.thriftfs.DatanodePlugins</value>
>> >        </property>
>> >        <property>
>> >                <name>dfs.thrift.address</name>
>> >                <value>0.0.0.0:9090</value>
>> >        </property>
>> > </configuration>
>> > -------- end of core-site.xml -----------
>> >
>> > mapred-site.xml:
>> >
>> > <?xml version="1.0"?>
>> > <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>> >
>> > <!-- Put site-specific property overrides in this file. -->
>> >
>> > <configuration>
>> >        <property>
>> >                <name>mapred.job.tracker</name>
>> >                <value>xxx.xxx.xxx.xxx:8020</value>
>> >        </property>
>> >        <property>
>> >                <name>jobtracker.thrift.address</name>
>> >                <value>0.0.0:9290</value>
>> >        </property>
>> >        <property>
>> >                <name>mapred.jobtracker.plugins</name>
>> >
>> > <value>org.apache.hadoop.thriftfs.ThriftJobTrackerPlugin</value>
>> >        </property>
>> > </configuration>
>> > -------- end of core-site.xml -----------
>> >
>>
>>
>>
>> --
>> Harsh J
>> www.harshj.com
>>
>



-- 
Harsh J
www.harshj.com

Mime
View raw message