Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of cembree@gmail.com designates
 209.85.219.53 as permitted sender)
MIME-Version: 1.0
Reply-To: chris@embree.us
In-Reply-To: 
 <CAPQV63UT8M9=tpBaDHfpnjt9Sr-oAv-Hgam33TF9xZSZUgcQZg@mail.gmail.com>
References: 
 <CAPQV63UT8M9=tpBaDHfpnjt9Sr-oAv-Hgam33TF9xZSZUgcQZg@mail.gmail.com>
Date: Sun, 10 Feb 2013 21:22:08 -0500
Message-ID: 
 <CADokdcMWjQiqK6Bb=7HSPahRFgj_xozSkEF8t-9AHV0hQHO1mg@mail.gmail.com>
Subject: Re: Mutiple dfs.data.dir vs RAID0
From: Chris Embree <cembree@gmail.com>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=e89a8fb2016e6fcb2704d569939b

--e89a8fb2016e6fcb2704d569939b
Content-Type: text/plain; charset=ISO-8859-1

Interesting question.  You'd probably need to benchmark to prove it out.

I'm not the exact details of how HDFS stripes data, but it should compare
pretty well to hardware RAID.

Conceptually, HDFS should be able to out perform a RAID solution, since
HDFS "knows" more about the data being written.  One of the benefits of
HDFS is being able to buy cheaper hardware and still getting good
performance.

We bought cheap DL165's for our datanodes.  4x 2TB Drives with no RAID.

On Sun, Feb 10, 2013 at 8:57 PM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:

> Hi,
>
> I have a quick question regarding RAID0 performances vs multiple
> dfs.data.dir entries.
>
> Let's say I have 2 x 2TB drives.
>
> I can configure them as 2 separate drives mounted on 2 folders and
> assignes to hadoop using dfs.data.dir. Or I can mount the 2 drives
> with RAID0 and assigned them as a single folder to dfs.data.dir.
>
> With RAID0, the reads and writes are going to be spread over the 2
> disks. This is significantly increasing the speed. But if I put 2
> entries in dfs.data.dir, hadoop is going to spread over those 2
> directories too, and at the end, ths results should the same, no?
>
> Any experience/advice/results to share?
>
> Thanks,
>
> JM
>

--e89a8fb2016e6fcb2704d569939b
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Interesting question. =A0You&#39;d probably need to benchmark to prove it o=
ut.<div><br></div><div>I&#39;m not the exact details of how HDFS stripes da=
ta, but it should compare pretty well to hardware RAID.</div><div><br></div=
>
<div>Conceptually, HDFS should be able to out perform a RAID solution, sinc=
e HDFS &quot;knows&quot; more about the data being written. =A0One of the b=
enefits of HDFS is being able to buy cheaper hardware and still getting goo=
d performance.</div>
<div><br></div><div>We bought cheap DL165&#39;s for our datanodes. =A04x 2T=
B Drives with no RAID.</div><div><br><div class=3D"gmail_quote">On Sun, Feb=
 10, 2013 at 8:57 PM, Jean-Marc Spaggiari <span dir=3D"ltr">&lt;<a href=3D"=
mailto:jean-marc@spaggiari.org" target=3D"_blank">jean-marc@spaggiari.org</=
a>&gt;</span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">Hi,<br>
<br>
I have a quick question regarding RAID0 performances vs multiple<br>
dfs.data.dir entries.<br>
<br>
Let&#39;s say I have 2 x 2TB drives.<br>
<br>
I can configure them as 2 separate drives mounted on 2 folders and<br>
assignes to hadoop using dfs.data.dir. Or I can mount the 2 drives<br>
with RAID0 and assigned them as a single folder to dfs.data.dir.<br>
<br>
With RAID0, the reads and writes are going to be spread over the 2<br>
disks. This is significantly increasing the speed. But if I put 2<br>
entries in dfs.data.dir, hadoop is going to spread over those 2<br>
directories too, and at the end, ths results should the same, no?<br>
<br>
Any experience/advice/results to share?<br>
<br>
Thanks,<br>
<br>
JM<br>
</blockquote></div><br></div>

--e89a8fb2016e6fcb2704d569939b--