Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of amansk@gmail.com designates
 209.85.213.169 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <1405526793.13450.YahooMailNeo@web140605.mail.bf1.yahoo.com>
References: 
 <CAEfu1ePw9EZtauOAtogFCdM_ietotxT6WLG3bJSwnUf=x-nJuQ@mail.gmail.com>
 <1405526793.13450.YahooMailNeo@web140605.mail.bf1.yahoo.com>
From: Amandeep Khurana <amansk@gmail.com>
Date: Wed, 16 Jul 2014 13:41:59 -0700
Message-ID: 
 <CAEfu1eMF+nfCN-Gt=xd5EgjqSjGAr+an0RnT7voe0MXBc86Oow@mail.gmail.com>
Subject: Re: Cluster sizing guidelines
To: "user@hbase.apache.org" <user@hbase.apache.org>,
 lars hofhansl <larsh@apache.org>
Content-Type: multipart/alternative; boundary=047d7b5d4840a365fe04fe559037

--047d7b5d4840a365fe04fe559037
Content-Type: text/plain; charset=UTF-8

Thanks Lars.

I'm curious how we'd answer questions like:
1. How many nodes do I need to sustain a write throughput of N reqs/sec
with payload of size M KB?
2. How many nodes do I need to sustain a read throughput of N reqs/sec with
payload of size M KB with a latency of X ms per read.
3. How many nodes do I need to store N TB of total data with one of the
above constraints?

This goes into looking at the bottlenecks that need to be taken into
account during write and read times and also the max number of regions and
region size that a single region server can host.

What are your thoughts on this?

-Amandeep


On Wed, Jul 16, 2014 at 9:06 AM, lars hofhansl <larsh@apache.org> wrote:

> This is a somewhat fuzzy art.
>
> Some points to consider:
> 1. All data is replicated three ways. Or in other words, if you run three
> RegionServer/Datanodes each machine will get 100% of the writes. If you run
> 6, each gets 50% of the writes. From that aspect HBase clusters with less
> than 9 RegionServers are not really useful.
> 2. As for the machines themselves. Just go with any reasonable machine,
> and pick the cheapest you can find. At least 8 cores, at least 32GB of RAM,
> at least 6 disks, no RAID needed. (we have machines with 12 cores in 2
> sockets, 96GB of RAM, 6 4TB drives, no HW RAID). HBase is not yet well
> tuned for SSDs.
>
>
> You also carefully need to consider your network topology. With HBase
> you'll see quite some east-west traffic (i.e. between racks). 10ge is good
> if you have it. We have 1ge everywhere so far, and we found this is a
> single most bottleneck for write performance.
>
>
> Also see this blog post about HBase memory sizing (shameless plug):
> http://hadoop-hbase.blogspot.de/2013/01/hbase-region-server-memory-sizing.html
>
>
> I'm planning a blog post about this topic with more details.
>
>
> -- Lars
>
>
>
> ________________________________
>  From: Amandeep Khurana <amansk@gmail.com>
> To: "user@hbase.apache.org" <user@hbase.apache.org>
> Sent: Tuesday, July 15, 2014 10:48 PM
> Subject: Cluster sizing guidelines
>
>
> Hi
>
> How do users usually go about sizing HBase clusters? What are the factors
> you take into account? What are typical hardware profiles you run with? Any
> data points you can share would help.
>
> Thanks
> Amandeep
>

--047d7b5d4840a365fe04fe559037--