hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anu Engineer <aengin...@hortonworks.com>
Subject Re: change HDFS disks on each node
Date Fri, 04 Mar 2016 22:57:55 GMT
Yes, it is possible, but If I were you and someone asked me to do it, I would say No. 

May I suggest that you decommission one node, let the data migrate to other machines, replace
the disks and bring that machine back in and let the cluster heal itself.
Not as efficient as moving data around by yourself, but safer and way more simpler than doing
it manually.

Thanks
Anu




On 3/4/16, 2:38 PM, "Joseph Naegele" <jnaegele@grierforensics.com> wrote:

>Thanks Anu, that makes sense. Assuming that works, allow me to complicate the problem
a bit further:
>
>Say I didn't use LVM and I needed to move from 2x3 TB volumes to 6x1 TB volumes. The problem
now is that the contents of each 3TB volume won't fit on a single 1TB volume. Is this still
possible?
>
>Thanks again,
>Joe
>
>-----Original Message-----
>From: Anu Engineer [mailto:aengineer@hortonworks.com] 
>Sent: Friday, March 04, 2016 4:25 PM
>To: Joseph Naegele <jnaegele@grierforensics.com>; user@hadoop.apache.org
>Subject: Re: change HDFS disks on each node
>
>Hi Joe,
>
>As long as you copy all the data in the old disk without altering the paths ( ie. Structure
and layout of the data dirs) and you use the same version of datanode software then it should
work.
>
>Here is the Apache documentation that implies that this will work, http://wiki.apache.org/hadoop/FAQ#On_an_individual_data_node.2C_how_do_you_balance_the_blocks_on_the_disk.3F
>
>In your case all you need to do is :
>
>1. Add the new hard disk
>2. Mount the new hard disk — copy the data directories to the new disk 3. Remove the
old disk and mount the new drive and make sure that your data directories are pointing to
the new location.
>
>That should do the trick. 
>
>As usual, any advice from a user group carries the risk of data loss.  So please be gentle
with your old disk(s) until you are absolutely sure that new disks are perfectly functional
:)
>
>Thanks
>Anu
>
>
>
>
>
>
>On 3/4/16, 1:04 PM, "Joseph Naegele" <jnaegele@grierforensics.com> wrote:
>
>>Hi all,
>>
>>Each of our N datanodes has attached two 3TB disks. I want to attach 
>>new
>>*replacement* storage to each node, move the HDFS contents to the new 
>>storage, and remove the old volumes. We're using Hadoop 2.7.1.
>>
>>1. What is the simplest, correct way to do this? Does hot-swapping move 
>>data from old disks to new disks? I am able to stop the cluster completely.
>>
>>2. Is it reasonable to use LVM to create expandable logical volumes? 
>>We're using AWS and contemplating switching from SSDs to magnetic 
>>storage, which is limited to 1TB volumes.
>>
>>Thanks,
>>Joe
>>
>>
>>
>>---------------------------------------------------------------------
>>To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>>For additional commands, e-mail: user-help@hadoop.apache.org
>>
>>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>For additional commands, e-mail: user-help@hadoop.apache.org
>
>
Mime
View raw message