hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Travis <hcoy...@ghostar.org>
Subject Re: Hadoop and RAID 5
Date Tue, 07 Oct 2014 06:52:12 GMT
On Sun, Oct 5, 2014 at 4:17 PM, Ulul <hadoop@ulul.org> wrote:

>  Hi Travis
> Thank you for your detailed answer and for honoring my question with a
> blog entry :-)

No problem.  I had been meaning to write something up.  Thanks for the
prod. :-)

> I will look into bus quiescing with admins but I'm under the impression
> that nothing special is done, the HW RAID controller taking care of
> everything, HP doc stating that inserting a hot-pluggable disk induces one
> or two seconds pause in disk activity. I'll check whether this is handled
> through the controller cache and/or done out of business hours for safety.
> I'll ask for internal benchmarking hoping it will convince everyone to
> accept the JBOD model and automate what's necessary for it not to disrupt
> operations

Make sure your HP disk controllers can actually do JBOD.  Last I looked
(admittedly, this was ~3 years ago), you could only simulate it doing
multiple single-disk RAID0 LUNs.  Operationally, these were one level
annoying more than just JBOD because now you had to remove/destroy the
RAID0 LUN when replacing the disk before you could recreate the filesystem.

At the very least, HP's reference architecture for Cloudera Hadoop 5.X,
updated in August 2014, shows that this is still the case.

>From their document:

 Drives should use a Just a Bunch of Disks
> (JBOD) configuration, which can be achieved with the HP Smart Array P420i
>> controller by configuring each individual disk as
> a separate RAID 0 volume. Additionally array acceleration features on the
>> P420i should be turned off for the RAID 0 data
> volumes. The first two positions on the drive cage allow the OS drive to
>> be placed in RAID1.

If you've already got equipment, then probably not a big deal.  If you're
in the process of evaluating new stuff, I'd ask your HP Var/Reseller if
there was a different, non-RAID option, especially if they have something
similar to the LSI-9207-8i card which uses the LSI2308 SAS chip.  I know
Dell validated this with their 12G equipment for use with Hadoop earlier
this year.  Definitely a great card so far.

Travis Campbell

View raw message