Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of jksingh26jun@gmail.com
 designates 74.125.83.48 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <507FF6F9.30708@crs4.it>
References: <OFD14BE21E.9D911BB4-ON87257A9A.004A4026@us.ibm.com>
 <2E362ACC9493D747B488241C66B3B66520B716@RHV-EXRDA-S11.corp.ebay.com>
 <507FF6F9.30708@crs4.it>
From: Jitendra Kumar Singh <jksingh26jun@gmail.com>
Date: Thu, 18 Oct 2012 19:18:44 +0530
Message-ID: 
 <CAKsPnnNugbu68VCYtUXHeMvTrndi78wtN_oCCvjbgtdgqssT6w@mail.gmail.com>
Subject: Re: HDFS using SAN
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=047d7b621e4081816104cc55a6a4

--047d7b621e4081816104cc55a6a4
Content-Type: text/plain; charset=ISO-8859-1

Hi,

In the NetApp whitepaper on SAN solution (link given by Kevin) it makes
following statement. Can someone please elaborate (or give a link that
explains) how 12-disk in SAN can give 2000 IOPS while if used as JBOD would
give 600 IOPS?

"The E2660 can deliver up to 2,000 IOPS
from a 12-disk stripe (the bottleneck being the 12 disks). This headroom
translates into better read times
for those 64KB blocks. Twelve copies of 12 MapReduce jobs reading from 12
SATA disks can at best
never exceed 12 x 50 IOPS, or 600 IOPS. The E2660 volume has five times the
IOPS headroom, which
translates into faster read times and high MapReduce throughput "

Thanks and Regards,
--
Jitendra Kumar Singh


On Thu, Oct 18, 2012 at 6:02 PM, Luca Pireddu <pireddu@crs4.it> wrote:

> On 10/18/2012 02:21 AM, Pamecha, Abhishek wrote:
>
>> Tom
>>
>> Do you mean you are using GPFS instead of HDFS? Also, if you can share,
>> are you deploying it as DAS set up or a SAN?
>>
>> Thanks,
>>
>> Abhishek
>>
>>
>
> Though I don't think I'd buy a SAN for a new Hadoop cluster, we have a SAN
> and are using it *instead of HDFS* with a small/medium Hadoop MapReduce
> cluster (up to 100 nodes or so, depending on our need).  We still use the
> local node disks for intermediate data (mapred local storage).  Although
> this set-up does limit our possibility to scale to a large number of nodes,
> that's not a concern for us.  On the plus, we gain the flexibility to be
> able to share our cluster with non-Hadoop users at our centre.
>
>
> --
> Luca Pireddu
> CRS4 - Distributed Computing Group
> Loc. Pixina Manna Edificio 1
> 09010 Pula (CA), Italy
> Tel: +39 0709250452
>

--047d7b621e4081816104cc55a6a4
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi,<div><br></div><div>In the NetApp whitepaper on SAN solution (link given=
 by Kevin) it makes following statement. Can someone please elaborate (or g=
ive a link that explains) how 12-disk in SAN can give 2000 IOPS while if us=
ed as JBOD would give 600 IOPS?=A0<div>

<br></div><div>&quot;The E2660 can deliver up to 2,000 IOPS=A0</div><div>fr=
om a 12-disk stripe (the bottleneck being the 12 disks). This headroom tran=
slates into better read times=A0</div><div>for those 64KB blocks. Twelve co=
pies of 12 MapReduce jobs reading from 12 SATA disks can at best=A0</div>

<div>never exceed 12 x 50 IOPS, or 600 IOPS. The E2660 volume has five time=
s the IOPS headroom, which=A0</div><div>translates into faster read times a=
nd high MapReduce throughput &quot;=A0<div><br clear=3D"all">Thanks and Reg=
ards,<br>

--<br>Jitendra Kumar Singh<br><br>
<br><br><div class=3D"gmail_quote">On Thu, Oct 18, 2012 at 6:02 PM, Luca Pi=
reddu <span dir=3D"ltr">&lt;<a href=3D"mailto:pireddu@crs4.it" target=3D"_b=
lank">pireddu@crs4.it</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_q=
uote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1e=
x">

<div class=3D"im">On 10/18/2012 02:21 AM, Pamecha, Abhishek wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">
Tom<br>
<br>
Do you mean you are using GPFS instead of HDFS? Also, if you can share,<br>
are you deploying it as DAS set up or a SAN?<br>
<br>
Thanks,<br>
<br>
Abhishek<br>
<br>
</blockquote>
<br>
<br></div>
Though I don&#39;t think I&#39;d buy a SAN for a new Hadoop cluster, we hav=
e a SAN and are using it *instead of HDFS* with a small/medium Hadoop MapRe=
duce cluster (up to 100 nodes or so, depending on our need). =A0We still us=
e the local node disks for intermediate data (mapred local storage). =A0Alt=
hough this set-up does limit our possibility to scale to a large number of =
nodes, that&#39;s not a concern for us. =A0On the plus, we gain the flexibi=
lity to be able to share our cluster with non-Hadoop users at our centre.<s=
pan class=3D"HOEnZb"><font color=3D"#888888"><br>


<br>
<br>
-- <br>
Luca Pireddu<br>
CRS4 - Distributed Computing Group<br>
Loc. Pixina Manna Edificio 1<br>
09010 Pula (CA), Italy<br>
Tel: +39 0709250452<br>
</font></span></blockquote></div><br></div></div></div>

--047d7b621e4081816104cc55a6a4--