hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Travis <hcoy...@ghostar.org>
Subject Re: Hadoop and RAID 5
Date Wed, 01 Oct 2014 22:25:16 GMT
On Wed, Oct 1, 2014 at 4:01 PM, Ulul <hadoop@ulul.org> wrote:

>  Dear hadoopers,
> Has anyone been confronted to deploying a cluster in a traditional IT shop
> whose admins handle thousands of servers ?
> They traditionally use SAN or NAS storage for app data, rely on RAID 1 for
> system disks and in the few cases where internal disks are used, they
> configure them with RAID 5 provided by the internal HW controller.
Yes.  I've been on both sides of this discussion.

The key is to help them understand that you don't need redundancy within a
system because Hadoop provides redundancy across the entire cluster via
replication.  This then leaves the problem as a performance one, in which
case you show them benchmarks on the hardware they provide in both RAID
(RAID0, RAID1, and RAID5) and JBOD modes.

> Using a JBOD setup , as advised in each and every Hadoop doc I ever laid
> my hands on, means that each HDD failure will imply, on top of the physical
> replacement of the drive, that an admin performs at least an mkfs.
> Added to the fact that these operations will become more frequent since
> more internal disks will be used, it can be perceived as an annoying
> disruption in industrial handling of numerous servers.
I fail to see how this is really any different than the process of having
to deal with a failed drive in an array.  Depending on your array type, you
may still have to do things to quiesce the bus before doing any drive
operation, such as adding or removing the drive, you may still have to
trigger the rebuild yourself, and so on.

I have a few thousand disks in my cluster.  We lose about 3-5 a quarter.  I
don't find it any more work to re-mkfs the drive after it's been swapped
out and have built tools around the process to make sure it's consistently
done by our DC staff (and yes, I did it before the DC staff was asked to).
If you're concerned about the high-touch aspect of swapping disks out, then
you can always configure the datanode to be tolerant of multiple disk
failures (something you cannot do with RAID5) and then just take the whole
machine out of the cluster to do swaps when you've reached a particular
threshold of bad disks.

> In Tom White's guide there is a discussion of RAID 0, stating that Yahoo
> benchmarks showed a 10% loss in performance so we can expect even worse
> perf with RAID 5 but I found no figures.

I had to re-read that section for reference.  My apologies if the following
is a little long-winded and rambling.

I'm going to assume that Tom is not talking about single-disk RAID0
volumes, which is a common way of doing JBOD with a RAID controller that
doesn't have JBOD support.

In general, performance is going to depend upon how many active streams of
I/O you have going on the system.

With JBOD, as Tom discusses, every spindle is it's own unique snow flake,
and if your drive controller can keep up, you can write as fast as that
drive can handle reading off the bus.  Performance is going to depend upon
how many active reading/writing streams you have accessing each spindle in
the systems.

If I had one stream, I would only get the performance of one spindle in the
JBOD. If I had twelve spindles, I'm going to get maximum performance with
at least twelve streams. With RAID0, you're taking your one stream, cutting
it up into multiple parts and either reading it or writing it to all disks,
taking advantage of the performance of all spindles.

The problem arises when you start adding more streams in parallel to the
RAID0 environment.  Each parallel I/O operation begins competing with each
other from the controller's standpoint.  Sometimes things start to stack up
as the controller has to wait for competing I/O operations on a single
spindle.  For example, having to wait for a write to complete before a read
can be done.

At a certain point, the performance of RAID0 begins to hit a knee as the
number of I/O requests goes up because the controller becomes the
bottleneck.  RAID0 is going to be the closest in performance, but with the
risk that if you lose a single disk, you lose the entire RAID.  With JBOD,
if you lose a single disk, you only lose the data on that disk.

Now, with RAID5, you're going to have even worse performance because you're
dealing with not only the parity calculation, but also with the fact that
you incur a performance penalty during reads and writes due to how the data
is laid out across all disks in the RAID.   You ca read more about this
here:  http://theithollow.com/2012/03/understanding-raid-penalty/

To put this in perspective, I use 12 7200rpm NLSAS disks in a system
connected to an LSI9207 SAS controller.  This is configured for JBOD.  I
have benchmarked streaming reads and writes in this environment to be
between 1.6 and 1.8GBytes/sec using 1 i/o stream per spindle for a total of
12 i/o streams occurring on the system.  Btw, this benchmark has held
stable at this rate for at least 3 i/o streams per spindle; I haven't
tested higher yet.

Now, I might get this performance with RAID0, but why should I tolerate the
risk of losing all data on the system vs just the data on a single drive?
Going with RAID0 means that not only do I have to replace the disk, but now
I have to have Hadoop rebalance/redistribute data to the entire system, not
just dealing with the small amount of data missing from one spindle.  And
since Hadoop is already handling my redundancy via replication of data, why
should I tolerate the performance penalty associated with RAID5?  I don't
need redundancy in a *single* system, I need redundancy across the entire

> I also found an Hortonworks interview of StackIQ who provides software to
> automate such failure fix up. But it would be rather painful to go straight
> to another solution, contract and so on while starting with Hadoop.
> Please share your experiences around RAID for redundancy (1, 5 or other)
> in Hadoop conf.
I can't see any situation that we would use RAID for the data drives in our
Hadoop cluster.  We only use RAID1 for the OS drives, simply because we
want to reduce the recovery period associated with a system failure.  No
reason to re-install a system and have to replicate data back onto it if we
don't have to.

Travis Campbell

View raw message