hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ulul <had...@ulul.org>
Subject Re: Hadoop and RAID 5
Date Sun, 05 Oct 2014 21:17:35 GMT
Hi Travis

Thank you for your detailed answer and for honoring my question with a 
blog entry :-)

I will look into bus quiescing with admins but I'm under the impression 
that nothing special is done, the HW RAID controller taking care of 
everything, HP doc stating that inserting a hot-pluggable disk induces 
one or two seconds pause in disk activity. I'll check whether this is 
handled through the controller cache and/or done out of business hours 
for safety.

I'll ask for internal benchmarking hoping it will convince everyone to 
accept the JBOD model and automate what's necessary for it not to 
disrupt operations

Thanks again

Le 02/10/2014 00:25, Travis a écrit :
> On Wed, Oct 1, 2014 at 4:01 PM, Ulul <hadoop@ulul.org 
> <mailto:hadoop@ulul.org>> wrote:
>     Dear hadoopers,
>     Has anyone been confronted to deploying a cluster in a traditional
>     IT shop whose admins handle thousands of servers ?
>     They traditionally use SAN or NAS storage for app data, rely on
>     RAID 1 for system disks and in the few cases where internal disks
>     are used, they configure them with RAID 5 provided by the internal
>     HW controller.
> Yes.  I've been on both sides of this discussion.
> The key is to help them understand that you don't need redundancy 
> within a system because Hadoop provides redundancy across the entire 
> cluster via replication. This then leaves the problem as a performance 
> one, in which case you show them benchmarks on the hardware they 
> provide in both RAID (RAID0, RAID1, and RAID5) and JBOD modes.
>     Using a JBOD setup , as advised in each and every Hadoop doc I
>     ever laid my hands on, means that each HDD failure will imply, on
>     top of the physical replacement of the drive, that an admin
>     performs at least an mkfs.
>     Added to the fact that these operations will become more frequent
>     since more internal disks will be used, it can be perceived as an
>     annoying disruption in industrial handling of numerous servers.
> I fail to see how this is really any different than the process of 
> having to deal with a failed drive in an array.  Depending on your 
> array type, you may still have to do things to quiesce the bus before 
> doing any drive operation, such as adding or removing the drive, you 
> may still have to trigger the rebuild yourself, and so on.
> I have a few thousand disks in my cluster.  We lose about 3-5 a 
> quarter.  I don't find it any more work to re-mkfs the drive after 
> it's been swapped out and have built tools around the process to make 
> sure it's consistently done by our DC staff (and yes, I did it before 
> the DC staff was asked to).  If you're concerned about the high-touch 
> aspect of swapping disks out, then you can always configure the 
> datanode to be tolerant of multiple disk failures (something you 
> cannot do with RAID5) and then just take the whole machine out of the 
> cluster to do swaps when you've reached a particular threshold of bad 
> disks.
>     In Tom White's guide there is a discussion of RAID 0, stating that
>     Yahoo benchmarks showed a 10% loss in performance so we can expect
>     even worse perf with RAID 5 but I found no figures.
> I had to re-read that section for reference.  My apologies if the 
> following is a little long-winded and rambling.
> I'm going to assume that Tom is not talking about single-disk RAID0 
> volumes, which is a common way of doing JBOD with a RAID controller 
> that doesn't have JBOD support.
> In general, performance is going to depend upon how many active 
> streams of I/O you have going on the system.
> With JBOD, as Tom discusses, every spindle is it's own unique snow 
> flake, and if your drive controller can keep up, you can write as fast 
> as that drive can handle reading off the bus.  Performance is going to 
> depend upon how many active reading/writing streams you have accessing 
> each spindle in the systems.
> If I had one stream, I would only get the performance of one spindle 
> in the JBOD. If I had twelve spindles, I'm going to get maximum 
> performance with at least twelve streams. With RAID0, you're taking 
> your one stream, cutting it up into multiple parts and either reading 
> it or writing it to all disks, taking advantage of the performance of 
> all spindles.
> The problem arises when you start adding more streams in parallel to 
> the RAID0 environment.  Each parallel I/O operation begins competing 
> with each other from the controller's standpoint.  Sometimes things 
> start to stack up as the controller has to wait for competing I/O 
> operations on a single spindle.  For example, having to wait for a 
> write to complete before a read can be done.
> At a certain point, the performance of RAID0 begins to hit a knee as 
> the number of I/O requests goes up because the controller becomes the 
> bottleneck.  RAID0 is going to be the closest in performance, but with 
> the risk that if you lose a single disk, you lose the entire RAID.  
> With JBOD, if you lose a single disk, you only lose the data on that disk.
> Now, with RAID5, you're going to have even worse performance because 
> you're dealing with not only the parity calculation, but also with the 
> fact that you incur a performance penalty during reads and writes due 
> to how the data is laid out across all disks in the RAID.   You ca 
> read more about this here: 
> http://theithollow.com/2012/03/understanding-raid-penalty/
> To put this in perspective, I use 12 7200rpm NLSAS disks in a system 
> connected to an LSI9207 SAS controller. This is configured for JBOD.  
> I have benchmarked streaming reads and writes in this environment to 
> be between 1.6 and 1.8GBytes/sec using 1 i/o stream per spindle for a 
> total of 12 i/o streams occurring on the system.  Btw, this benchmark 
> has held stable at this rate for at least 3 i/o streams per spindle; I 
> haven't tested higher yet.
> Now, I might get this performance with RAID0, but why should I 
> tolerate the risk of losing all data on the system vs just the data on 
> a single drive?  Going with RAID0 means that not only do I have to 
> replace the disk, but now I have to have Hadoop rebalance/redistribute 
> data to the entire system, not just dealing with the small amount of 
> data missing from one spindle.  And since Hadoop is already handling 
> my redundancy via replication of data, why should I tolerate the 
> performance penalty associated with RAID5?  I don't need redundancy in 
> a *single* system, I need redundancy across the entire cluster.
>     I also found an Hortonworks interview of StackIQ who provides
>     software to automate such failure fix up. But it would be rather
>     painful to go straight to another solution, contract and so on
>     while starting with Hadoop.
>     Please share your experiences around RAID for redundancy (1, 5 or
>     other) in Hadoop conf.
> I can't see any situation that we would use RAID for the data drives 
> in our Hadoop cluster.  We only use RAID1 for the OS drives, simply 
> because we want to reduce the recovery period associated with a system 
> failure.  No reason to re-install a system and have to replicate data 
> back onto it if we don't have to.
> Cheers,
> Travis
> -- 
> Travis Campbell
> travis@ghostar.org <mailto:travis@ghostar.org>

View raw message