Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
MIME-Version: 1.0
Sender: saint.ack@gmail.com
In-Reply-To: <1ccdad83-523d-4621-b386-a59438e482c1@me.com>
References: <1ccdad83-523d-4621-b386-a59438e482c1@me.com>
Date: Tue, 8 Mar 2016 09:20:19 -0800
Message-ID: 
 <CADcMMgEJDyXw0mjNY92+ebfb3uv9_ubZ_JKLWkgrBNg24sjcaA@mail.gmail.com>
Subject: Re: HBase poor write performance
From: Stack <stack@duboce.net>
To: Hbase-User <user@hbase.apache.org>
Content-Type: multipart/alternative; boundary=001a11c1e276abc3ae052d8ccb09

--001a11c1e276abc3ae052d8ccb09
Content-Type: text/plain; charset=UTF-8

On Tue, Mar 8, 2016 at 8:28 AM, Akmal Abbasov <akmal.abbasov@icloud.com>
wrote:

> Hi,
> I'm testing HBase to choose the right hardware configurations for a heavy
> write use case. I'm testing using YCSB.
>
The cluster consist of 2 masters, and 5 regionservers(4 cores, 14GB ram,
> 4x512GB SSD).
> I've created a new table in HBase, presplit it to 50 regions. I'm running
> 3 clients each running 50 threads, to insert data.
>

What happens if you double the number of clients?


> I'm using the default HBase settings.
>


Our defaults are intentionally conservative set for broad appeal. For now,
there is an expectation that configs are adjusted to suit the loadings.


> After running few tests, I can see that the cluster is underutilized, in
> fact memory usage is around 30%.
>


Which memory? Java heap? Or the systems' memory? If java heap, we make
broad allocations for cache and write-time memstore. You might allocate
more to memstore if you are write-centric (Later versions of hbase try to
do this ergonomically).


> The main problem I see for now is compactions, compactionQueueLength is
> growing very fast, and compaction process is always running.
>

Compactions running all the time while you are under write load is
'normal'. How are you seeing this manifest as a 'problem'?

Show us  a bit of your log from a regionserver and take a few thread dumps
while it is running so we can make suggestion.


> I found that there are hbase.regionserver.thread.compaction.small and hbase.regionserver.thread.compaction.large
> but couldn't find information regarding their default values.
>


Yes. This stuff is missing from the doc, in code only. Lets figure if they
need adjustment first.


> I am also planing to increase the regions number and the memstore size to
> increase utilization of the cluster and performance.
> Which other settings should be tuned to improve both utilization and
> performance?
>

You've browsed this section http://hbase.apache.org/book.html#performance ?


> Thank you.
>
>

> I'm using HBase 0.98.7 and regionserver heap size is 7GB.
>

Can you use a later HBase? 1.2.0 just got released.
St.Ack


>
> Regards, Akmal
>

--001a11c1e276abc3ae052d8ccb09--