hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <qwertyman...@gmail.com>
Subject Re: dfs.data.dir and "hadoop namenode -format"
Date Thu, 24 Jun 2010 16:10:37 GMT
On Thu, Jun 24, 2010 at 9:09 PM, Sean Bigdatafun
<sean.bigdatafun@gmail.com> wrote:
>
>
> On Thu, Jun 24, 2010 at 8:20 AM, Jeff Whiting <jeffw@qualtrics.com> wrote:
>>
>> 1) You have to start and restart
>> 2) Yes that is how it would get the new directory.  However changing the
>> directory means it wont be able to find any of your old data.  If you don't
>> want to start over from scratch you will want to stop dfs then copy the
>> files over to the new data directory and then restart it.
>
> what about I simply add additional directory? i.e., leave the previous one
> in the list. will that make the old data lost?
No.
> From:
> -------------------
>  dfs.data.dir
>     /opt/hadoop/data
> --------------------
>
> To:
> -------------------
>  dfs.data.dir
>     /opt/hadoop/data, /home/sean/hadoopdata
> --------------------
>
>>
>> 3) Doing the namenode -format will clean up the directory on the name node
>> and make everything good to go with no data in dfs.  However you'll have to
>> go to the datanodes and clean out their data directories.   If you leave
>> data in the directory on the datanodes they will be unable to join with the
>> namenode.
>>
>> This isn't the most technical explanation but hopefully it helps.
>> ~Jeff
>>
>> Sean Bigdatafun wrote:
>>>
>>> Can someone tell me what "hadoop namenode -format" does under the hood?
>>>
>>> I have started my HDFS cell with the following configuration.
>>> -------------------
>>>  dfs.data.dir
>>>    /opt/hadoop/data
>>> --------------------
>>>
>>> Overtime, I want to add another directory as the data.dir, how can I
>>> achieve it?
>>> 1) Can I simply edit "dfs.data.dir" in the hdfs-site.xml without stopping
>>> my cell?
>>>
>>> 2) If 1) is not legitimate, can I run "stop-dfs.sh", then do 1) and then
>>> "start-dfs.sh"?
>>>
>>> 3) My last question here is what "hadoop namenode -format" does. If I run
>>> it on my Namenode, does it clean up the data.dir? and do I need to manually
>>> clean up the data.dir on Datanode?
>>>
>>> Thanks,
>>> Sean
>>>
>>>
>>>
>>>
>>>
>>
>> --
>> Jeff Whiting
>> Qualtrics Senior Software Engineer
>> jeffw@qualtrics.com
>>
>
>



-- 
Harsh J
www.harshj.com

Mime
View raw message