hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amandeep Khurana <ama...@gmail.com>
Subject Re: Cluster sizing guidelines
Date Wed, 16 Jul 2014 20:41:59 GMT
Thanks Lars.

I'm curious how we'd answer questions like:
1. How many nodes do I need to sustain a write throughput of N reqs/sec
with payload of size M KB?
2. How many nodes do I need to sustain a read throughput of N reqs/sec with
payload of size M KB with a latency of X ms per read.
3. How many nodes do I need to store N TB of total data with one of the
above constraints?

This goes into looking at the bottlenecks that need to be taken into
account during write and read times and also the max number of regions and
region size that a single region server can host.

What are your thoughts on this?

-Amandeep


On Wed, Jul 16, 2014 at 9:06 AM, lars hofhansl <larsh@apache.org> wrote:

> This is a somewhat fuzzy art.
>
> Some points to consider:
> 1. All data is replicated three ways. Or in other words, if you run three
> RegionServer/Datanodes each machine will get 100% of the writes. If you run
> 6, each gets 50% of the writes. From that aspect HBase clusters with less
> than 9 RegionServers are not really useful.
> 2. As for the machines themselves. Just go with any reasonable machine,
> and pick the cheapest you can find. At least 8 cores, at least 32GB of RAM,
> at least 6 disks, no RAID needed. (we have machines with 12 cores in 2
> sockets, 96GB of RAM, 6 4TB drives, no HW RAID). HBase is not yet well
> tuned for SSDs.
>
>
> You also carefully need to consider your network topology. With HBase
> you'll see quite some east-west traffic (i.e. between racks). 10ge is good
> if you have it. We have 1ge everywhere so far, and we found this is a
> single most bottleneck for write performance.
>
>
> Also see this blog post about HBase memory sizing (shameless plug):
> http://hadoop-hbase.blogspot.de/2013/01/hbase-region-server-memory-sizing.html
>
>
> I'm planning a blog post about this topic with more details.
>
>
> -- Lars
>
>
>
> ________________________________
>  From: Amandeep Khurana <amansk@gmail.com>
> To: "user@hbase.apache.org" <user@hbase.apache.org>
> Sent: Tuesday, July 15, 2014 10:48 PM
> Subject: Cluster sizing guidelines
>
>
> Hi
>
> How do users usually go about sizing HBase clusters? What are the factors
> you take into account? What are typical hardware profiles you run with? Any
> data points you can share would help.
>
> Thanks
> Amandeep
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message