hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dejan Menges <dejan.men...@gmail.com>
Subject Re: Supervisely, RAID0 provides best io performance whereas no RAID the worst
Date Mon, 01 Aug 2016 07:33:30 GMT
Sorry for jumping in, but hence performance... it took as a while to figure
out why, whatever disk/RAID0 performance you have, when it comes to HDFS
and replication factor bigger then zero, disk write speed drops to
100Mbps... After long long tests with Hortonworks they found that issue is
that someone at some point in history hardcoded stuff somewhere, and
whatever setup you have, you were limited to this. Luckily we have quite
powerful testing environment and plan is to test this patch later this
week. I'm not sure if there's either official HDFS bug for this, checked
our internal history but didn't see anything like that.

This was quite disappointing, as whatever tuning, controllers, setups you
do, it goes down the water with this.

On Mon, Aug 1, 2016 at 8:30 AM Allen Wittenauer <aw@apache.org> wrote:

>
>
> On 2016-07-30 20:12 (-0700), Shady Xu <shadyxu@gmail.com> wrote:
> > Thanks Andrew, I know about the disk failure risk and that it's one of
> the
> > reasons why we should use JBOD. But JBOD provides worse performance than
> > RAID 0.
>
> It's not about failure: it's about speed.  RAID0 performance will drop
> like a rock if any one disk in the set is slow. When all the drives are
> performing at peak, yes, it's definitely faster.  But over time, drive
> speed will decline (sometimes to half speed or less!) usually prior to a
> failure. This failure may take a while, so in the mean time your cluster is
> getting slower ... and slower ... and slower ...
>
> As a result, JBOD will be significantly faster over the _lifetime_ of the
> disks vs. a comparison made _today_.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: user-help@hadoop.apache.org
>
>

Mime
View raw message