Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: <542C6BBB.30404@ulul.org>
References: <542C6BBB.30404@ulul.org>
Date: Wed, 1 Oct 2014 17:25:16 -0500
Message-ID: 
 <CAChq9g1LfXLgGew605YsxqLHpNiSOr=FYtiyTiBVHEkdDLW=dw@mail.gmail.com>
Subject: Re: Hadoop and RAID 5
From: Travis <hcoyote@ghostar.org>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=047d7bf0c4126995a3050463f9ff

--047d7bf0c4126995a3050463f9ff
Content-Type: text/plain; charset=UTF-8

On Wed, Oct 1, 2014 at 4:01 PM, Ulul <hadoop@ulul.org> wrote:

>  Dear hadoopers,
>
> Has anyone been confronted to deploying a cluster in a traditional IT shop
> whose admins handle thousands of servers ?
> They traditionally use SAN or NAS storage for app data, rely on RAID 1 for
> system disks and in the few cases where internal disks are used, they
> configure them with RAID 5 provided by the internal HW controller.
>
>
Yes.  I've been on both sides of this discussion.

The key is to help them understand that you don't need redundancy within a
system because Hadoop provides redundancy across the entire cluster via
replication.  This then leaves the problem as a performance one, in which
case you show them benchmarks on the hardware they provide in both RAID
(RAID0, RAID1, and RAID5) and JBOD modes.


> Using a JBOD setup , as advised in each and every Hadoop doc I ever laid
> my hands on, means that each HDD failure will imply, on top of the physical
> replacement of the drive, that an admin performs at least an mkfs.
> Added to the fact that these operations will become more frequent since
> more internal disks will be used, it can be perceived as an annoying
> disruption in industrial handling of numerous servers.
>
>
I fail to see how this is really any different than the process of having
to deal with a failed drive in an array.  Depending on your array type, you
may still have to do things to quiesce the bus before doing any drive
operation, such as adding or removing the drive, you may still have to
trigger the rebuild yourself, and so on.

I have a few thousand disks in my cluster.  We lose about 3-5 a quarter.  I
don't find it any more work to re-mkfs the drive after it's been swapped
out and have built tools around the process to make sure it's consistently
done by our DC staff (and yes, I did it before the DC staff was asked to).
If you're concerned about the high-touch aspect of swapping disks out, then
you can always configure the datanode to be tolerant of multiple disk
failures (something you cannot do with RAID5) and then just take the whole
machine out of the cluster to do swaps when you've reached a particular
threshold of bad disks.


> In Tom White's guide there is a discussion of RAID 0, stating that Yahoo
> benchmarks showed a 10% loss in performance so we can expect even worse
> perf with RAID 5 but I found no figures.
>

I had to re-read that section for reference.  My apologies if the following
is a little long-winded and rambling.

I'm going to assume that Tom is not talking about single-disk RAID0
volumes, which is a common way of doing JBOD with a RAID controller that
doesn't have JBOD support.

In general, performance is going to depend upon how many active streams of
I/O you have going on the system.

With JBOD, as Tom discusses, every spindle is it's own unique snow flake,
and if your drive controller can keep up, you can write as fast as that
drive can handle reading off the bus.  Performance is going to depend upon
how many active reading/writing streams you have accessing each spindle in
the systems.

If I had one stream, I would only get the performance of one spindle in the
JBOD. If I had twelve spindles, I'm going to get maximum performance with
at least twelve streams. With RAID0, you're taking your one stream, cutting
it up into multiple parts and either reading it or writing it to all disks,
taking advantage of the performance of all spindles.

The problem arises when you start adding more streams in parallel to the
RAID0 environment.  Each parallel I/O operation begins competing with each
other from the controller's standpoint.  Sometimes things start to stack up
as the controller has to wait for competing I/O operations on a single
spindle.  For example, having to wait for a write to complete before a read
can be done.

At a certain point, the performance of RAID0 begins to hit a knee as the
number of I/O requests goes up because the controller becomes the
bottleneck.  RAID0 is going to be the closest in performance, but with the
risk that if you lose a single disk, you lose the entire RAID.  With JBOD,
if you lose a single disk, you only lose the data on that disk.

Now, with RAID5, you're going to have even worse performance because you're
dealing with not only the parity calculation, but also with the fact that
you incur a performance penalty during reads and writes due to how the data
is laid out across all disks in the RAID.   You ca read more about this
here:  http://theithollow.com/2012/03/understanding-raid-penalty/

To put this in perspective, I use 12 7200rpm NLSAS disks in a system
connected to an LSI9207 SAS controller.  This is configured for JBOD.  I
have benchmarked streaming reads and writes in this environment to be
between 1.6 and 1.8GBytes/sec using 1 i/o stream per spindle for a total of
12 i/o streams occurring on the system.  Btw, this benchmark has held
stable at this rate for at least 3 i/o streams per spindle; I haven't
tested higher yet.

Now, I might get this performance with RAID0, but why should I tolerate the
risk of losing all data on the system vs just the data on a single drive?
Going with RAID0 means that not only do I have to replace the disk, but now
I have to have Hadoop rebalance/redistribute data to the entire system, not
just dealing with the small amount of data missing from one spindle.  And
since Hadoop is already handling my redundancy via replication of data, why
should I tolerate the performance penalty associated with RAID5?  I don't
need redundancy in a *single* system, I need redundancy across the entire
cluster.


> I also found an Hortonworks interview of StackIQ who provides software to
> automate such failure fix up. But it would be rather painful to go straight
> to another solution, contract and so on while starting with Hadoop.
>
> Please share your experiences around RAID for redundancy (1, 5 or other)
> in Hadoop conf.
>
>
I can't see any situation that we would use RAID for the data drives in our
Hadoop cluster.  We only use RAID1 for the OS drives, simply because we
want to reduce the recovery period associated with a system failure.  No
reason to re-install a system and have to replicate data back onto it if we
don't have to.

Cheers,
Travis
-- 
Travis Campbell
travis@ghostar.org

--047d7bf0c4126995a3050463f9ff
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">=
On Wed, Oct 1, 2014 at 4:01 PM, Ulul <span dir=3D"ltr">&lt;<a href=3D"mailt=
o:hadoop@ulul.org" target=3D"_blank">hadoop@ulul.org</a>&gt;</span> wrote:<=
br><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;bord=
er-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:soli=
d;padding-left:1ex">
 =20

   =20
 =20
  <div text=3D"#000000" bgcolor=3D"#FFFFFF">
    <font face=3D"Courier 10 Pitch">Dear hadoopers,<br>
      <br>
      Has anyone been confronted to deploying a cluster in a traditional
      IT shop whose admins handle thousands of servers ?<br>
      They traditionally use SAN or NAS storage for app data, rely on
      RAID 1 for system disks and in the few cases where internal disks
      are used, they configure them with RAID 5 provided by the internal
      HW controller.<br>
      <br></font></div></blockquote><div><br></div><div>Yes.=C2=A0 I&#39;ve=
 been on both sides of this discussion.</div><div><br></div><div>The key is=
 to help them understand that you don&#39;t need redundancy within a system=
 because Hadoop provides redundancy across the entire cluster via replicati=
on.=C2=A0 This then leaves the problem as a performance one, in which case =
you show them benchmarks on the hardware they provide in both RAID (RAID0, =
RAID1, and RAID5) and JBOD modes.</div><div>=C2=A0<br></div><blockquote cla=
ss=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left-width:1px;=
border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex=
"><div text=3D"#000000" bgcolor=3D"#FFFFFF"><font face=3D"Courier 10 Pitch"=
>
      Using a JBOD setup , as advised in each and every Hadoop doc I
      ever laid my hands on, means that each HDD failure will imply, on
      top of the physical replacement of the drive, that an admin
      performs at least an mkfs. <br>
      Added to the fact that these operations will become more frequent
      since more internal disks will be used, it can be perceived as an
      annoying disruption in industrial handling of numerous servers.<br>
      <br></font></div></blockquote><div><br></div><div>I fail to see how t=
his is really any different than the process of having to deal with a faile=
d drive in an array.=C2=A0 Depending on your array type, you may still have=
 to do things to quiesce the bus before doing any drive operation, such as =
adding or removing the drive, you may still have to trigger the rebuild you=
rself, and so on. =C2=A0</div><div><br></div><div>I have a few thousand dis=
ks in my cluster.=C2=A0 We lose about 3-5 a quarter.=C2=A0 I don&#39;t find=
 it any more work to re-mkfs the drive after it&#39;s been swapped out and =
have built tools around the process to make sure it&#39;s consistently done=
 by our DC staff (and yes, I did it before the DC staff was asked to).=C2=
=A0 If you&#39;re concerned about the high-touch aspect of swapping disks o=
ut, then you can always configure the datanode to be tolerant of multiple d=
isk failures (something you cannot do with RAID5) and then just take the wh=
ole machine out of the cluster to do swaps when you&#39;ve reached a partic=
ular threshold of bad disks.</div><div><br></div><div>=C2=A0=C2=A0<br></div=
><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border=
-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;=
padding-left:1ex"><div text=3D"#000000" bgcolor=3D"#FFFFFF"><font face=3D"C=
ourier 10 Pitch">
      In Tom White&#39;s guide there is a discussion of RAID 0, stating tha=
t
      Yahoo benchmarks showed a 10% loss in performance so we can expect
      even worse perf with RAID 5 but I found no figures.<br></font></div><=
/blockquote><div><br></div><div>I had to re-read that section for reference=
.=C2=A0 My apologies if the following is a little long-winded and rambling.=
 =C2=A0</div><div><br></div><div>I&#39;m going to assume that Tom is not ta=
lking about single-disk RAID0 volumes, which is a common way of doing JBOD =
with a RAID controller that doesn&#39;t have JBOD support.</div><div><br></=
div><div>In general, performance is going to depend upon how many active st=
reams of I/O you have going on the system.=C2=A0</div><div><br></div><div>W=
ith JBOD, as Tom discusses, every spindle is it&#39;s own unique snow flake=
, and if your drive controller can keep up, you can write as fast as that d=
rive can handle reading off the bus.=C2=A0 Performance is going to depend u=
pon how many active reading/writing streams you have accessing each spindle=
 in the systems. =C2=A0</div><div><br></div><div>If I had one stream, I wou=
ld only get the performance of one spindle in the JBOD. If I had twelve spi=
ndles, I&#39;m going to get maximum performance with at least twelve stream=
s. With RAID0, you&#39;re taking your one stream, cutting it up into multip=
le parts and either reading it or writing it to all disks, taking advantage=
 of the performance of all spindles.</div><div><br></div><div>The problem a=
rises when you start adding more streams in parallel to the RAID0 environme=
nt.=C2=A0 Each parallel I/O operation begins competing with each other from=
 the controller&#39;s standpoint.=C2=A0 Sometimes things start to stack up =
as the controller has to wait for competing I/O operations on a single spin=
dle.=C2=A0 For example, having to wait for a write to complete before a rea=
d can be done.</div><div><br></div><div>At a certain point, the performance=
 of RAID0 begins to hit a knee as the number of I/O requests goes up becaus=
e the controller becomes the bottleneck.=C2=A0 RAID0 is going to be the clo=
sest in performance, but with the risk that if you lose a single disk, you =
lose the entire RAID.=C2=A0 With JBOD, if you lose a single disk, you only =
lose the data on that disk.</div><div><br></div><div>Now, with RAID5, you&#=
39;re going to have even worse performance because you&#39;re dealing with =
not only the parity calculation, but also with the fact that you incur a pe=
rformance penalty during reads and writes due to how the data is laid out a=
cross all disks in the RAID. =C2=A0 You ca read more about this here: =C2=
=A0<a href=3D"http://theithollow.com/2012/03/understanding-raid-penalty/">h=
ttp://theithollow.com/2012/03/understanding-raid-penalty/</a></div><div><br=
></div><div>To put this in perspective, I use 12 7200rpm NLSAS disks in a s=
ystem connected to an LSI9207 SAS controller.=C2=A0 This is configured for =
JBOD.=C2=A0 I have benchmarked streaming reads and writes in this environme=
nt to be between 1.6 and 1.8GBytes/sec using 1 i/o stream per spindle for a=
 total of 12 i/o streams occurring on the system.=C2=A0 Btw, this benchmark=
 has held stable at this rate for at least 3 i/o streams per spindle; I hav=
en&#39;t tested higher yet.=C2=A0</div><div><br></div><div>Now, I might get=
 this performance with RAID0, but why should I tolerate the risk of losing =
all data on the system vs just the data on a single drive?=C2=A0 Going with=
 RAID0 means that not only do I have to replace the disk, but now I have to=
 have Hadoop rebalance/redistribute data to the entire system, not just dea=
ling with the small amount of data missing from one spindle.=C2=A0 And sinc=
e Hadoop is already handling my redundancy via replication of data, why sho=
uld I tolerate the performance penalty associated with RAID5?=C2=A0 I don&#=
39;t need redundancy in a *single* system, I need redundancy across the ent=
ire cluster.</div><div><br></div><div><br></div><blockquote class=3D"gmail_=
quote" style=3D"margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-=
color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div text=
=3D"#000000" bgcolor=3D"#FFFFFF"><font face=3D"Courier 10 Pitch"><br>
      I also found an Hortonworks interview of StackIQ who provides
      software to automate such failure fix up. But it would be rather
      painful to go straight to another solution, contract and so on
      while starting with Hadoop.<br>
      <br>
      Please share your experiences around RAID for redundancy (1, 5 or
      other) in Hadoop conf. <br><span class=3D""><font color=3D"#888888"><=
br></font></span></font></div></blockquote><div><br></div><div>I can&#39;t =
see any situation that we would use RAID for the data drives in our Hadoop =
cluster.=C2=A0 We only use RAID1 for the OS drives, simply because we want =
to reduce the recovery period associated with a system failure.=C2=A0 No re=
ason to re-install a system and have to replicate data back onto it if we d=
on&#39;t have to.=C2=A0</div></div><br>Cheers,</div><div class=3D"gmail_ext=
ra">Travis<br>-- <br>Travis Campbell<br><a href=3D"mailto:travis@ghostar.or=
g" target=3D"_blank">travis@ghostar.org</a>
</div></div>

--047d7bf0c4126995a3050463f9ff--