hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Daniel Cryans <jdcry...@apache.org>
Subject Re: HBase write latency issues with hdfs replication of 3
Date Thu, 06 Oct 2011 18:35:57 GMT
Inline at the end.

J-D

On Tue, Oct 4, 2011 at 11:06 AM, Ronen Itkin <ronen@taykey.com> wrote:
> Hi all!
>
> I am getting really bad write performance when writing to HBase that relies
> on hdfs (with replection parameter of 3) in Amazone Web Services
> environment.
> I am using Cloudera CDH3u1 distribution for all of the components.
>
> Here are some benchmarks:
>
> *Cluster 1:*
> Back in my office (not on an Amazon instance):
>
>   - 1 server (3GB RAM, 1 CPU unit) installed with a local Hadoop cluster
>   (NameNode,DatNode,JTracker,DataNode), HBase cluster (HMaster,HRegion) and a
>   local Zookeeper - all with default configuration.
>   - HDFS replication parameter = *1*
>   - Benchmark result: *writing 10,000 records in ~1 second*
>
> *Cluster 2:*
> On Amazon EC2 environment
>
>   - 1 server - small instance (1.7GB RAM, 1 CPU unit) installed with a
>   local Hadoop cluster (NameNode,DatNode,JTracker,DataNode), HBase cluster
>   (HMaster,HRegion) and a local Zookeeper - all with default configuration.
>   - HDFS replication parameter = *1*
>   - Benchmark result: *writing 10,000 records in ~8 seconds*
>
> *Cluster 3:*
> On Amazon EC2 environment
>
>   - 1 server - large instance (7.5GB RAM, 4 CPU Units) installed Hadoop
>   NameNode, JotTracker, HBaseMaster, ZooKeeper.
>   - 3 servers - xlarge instances (15GB RAM, 8 CPU Units  - each instance)
>   installed Hadoop with DataNode, TaskTracker, HBaseRegionServer.
>   - 1 server -  large instance (7.5GB RAM, 4 CPU Units) installed Hadoop
>   SecondaryNameNode, HBaseBackupMaster, Zookeeper.
>   - 1 server - small instance (1.7GB RAM, 1 CPU unit) installed with
>   Zookeeper.
>   - First run      - HDFS replication parameter = *1*  -->  Benchmark
>   result: *writing 10,000 records in ~11 seconds*
>   - Second run - HDFS replication parameter = *3*  -->  Benchmark
> result: *writing
>   10,000 records in ~58 seconds*
>
>
> *Questions:*
>
>   - Cluster 1 compared to cluster 2 - Why there is a huge difference in
>   performance between Amazon environment and my back office environments ?? as
>   far as I know the difference should be only the I/O, because Amazon's
>   partitions are not really local, and yet the difference is huge!

You need to read more on EC2. Basically the smaller the instance the
smaller the IO share it has. It is *not* like having a full machine
just for you (until you get to the bigger instances, but even then
it's still a shared environment).

>   - Cluster 3 - Why does it make such a huge difference if the replication
>   is set to tree? Isnt it supposed to be transparent ? and get the
>    same performance? because the client's application interacts with the name
>    node and waits to receive only one ACK per packet? Does the NameNode sends
>    that ACK after the first successful write or after the whole 3 replications
>    are ready? is that configurable? maybe its the client's application fault?

Ok so first, replicating to 3 nodes takes longer than writing to just
one. Replication in HDFS works in a pipeline, client writes to
datanode1, which writes to datanode2, which writes to datanode3 and
then the ACK comes all the way back.

Second, the IO performance you're getting in EC2 is probably less than
what you think. Once you have 3 machines involved which need to talk
to each other, you might be taking a serious hit. Also I would
recommend running each tests a few times because another about EC2 is
that performance varies... sometimes a lot!

>
>
>
>
>
>
>
> --
> *
> Ronen Itkin*
> Taykey | www.taykey.com
>

Mime
View raw message