hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: Local sockets
Date Sat, 04 Dec 2010 23:04:29 GMT
On Sat, Dec 4, 2010 at 2:57 PM, Vladimir Rodionov
<vrodionov@carrieriq.com>wrote:

> From my own experiments performance difference is huge even on
> sequential R/W operations (up to 300%) when you do local File I/O vs HDFS
> File I/O
>
> Overhead of HDFS I/O is substantial to say the least.
>
>
Much of this is from checksumming, though - turn off checksums and you
should see about a 2x improvement at least.

-Todd


> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodionov@carrieriq.com
>
> ________________________________________
> From: Todd Lipcon [todd@cloudera.com]
> Sent: Saturday, December 04, 2010 12:30 PM
> To: dev@hbase.apache.org
> Subject: Re: Local sockets
>
> Hi Leen,
>
> Check out HDFS-347 for more info on this. I hope to pick this back up in
> 2011 - in 2010 we mostly focused on stability above performance in HBase's
> interactions with HDFS.
>
> Thanks
> -Todd
>
> On Sat, Dec 4, 2010 at 12:28 PM, Leen Toelen <toelen@gmail.com> wrote:
>
> > Hi,
> >
> > has anyone tested the performance impact (when there is a hdfs
> > datanode and a hbase node on the same machine) of using unix domain
> > sockets communication or shared memory ipc using nio? I guess this
> > should make a difference on reads?
> >
> > Regards,
> > Leen
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>



-- 
Todd Lipcon
Software Engineer, Cloudera

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message