hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Bigdatafun <sean.bigdata...@gmail.com>
Subject Re: dfs.data.dir and "hadoop namenode -format"
Date Thu, 24 Jun 2010 15:39:14 GMT
On Thu, Jun 24, 2010 at 8:20 AM, Jeff Whiting <jeffw@qualtrics.com> wrote:

> 1) You have to start and restart
> 2) Yes that is how it would get the new directory.  However changing the
> directory means it wont be able to find any of your old data.  If you don't
> want to start over from scratch you will want to stop dfs then copy the
> files over to the new data directory and then restart it.
>
what about I simply add additional directory? i.e., leave the previous one
in the list. will that make the old data lost?

From:
-------------------
 dfs.data.dir
    /opt/hadoop/data
--------------------


To:
-------------------
 dfs.data.dir
    /opt/hadoop/data, /home/sean/hadoopdata
--------------------



> 3) Doing the namenode -format will clean up the directory on the name node
> and make everything good to go with no data in dfs.  However you'll have to
> go to the datanodes and clean out their data directories.   If you leave
> data in the directory on the datanodes they will be unable to join with the
> namenode.
>
> This isn't the most technical explanation but hopefully it helps.
> ~Jeff
>
>
> Sean Bigdatafun wrote:
>
>> Can someone tell me what "hadoop namenode -format" does under the hood?
>>
>> I have started my HDFS cell with the following configuration.
>> -------------------
>>  dfs.data.dir
>>    /opt/hadoop/data
>> --------------------
>>
>> Overtime, I want to add another directory as the data.dir, how can I
>> achieve it?
>> 1) Can I simply edit "dfs.data.dir" in the hdfs-site.xml without stopping
>> my cell?
>>
>> 2) If 1) is not legitimate, can I run "stop-dfs.sh", then do 1) and then
>> "start-dfs.sh"?
>>
>> 3) My last question here is what "hadoop namenode -format" does. If I run
>> it on my Namenode, does it clean up the data.dir? and do I need to manually
>> clean up the data.dir on Datanode?
>>
>> Thanks,
>> Sean
>>
>>
>>
>>
>>
>>
> --
> Jeff Whiting
> Qualtrics Senior Software Engineer
> jeffw@qualtrics.com
>
>

Mime
View raw message