Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of vincent.barat@gmail.com
 designates 209.85.214.41 as permitted sender)
Message-ID: <50AD1F5C.7070503@gmail.com>
Date: Wed, 21 Nov 2012 19:37:16 +0100
From: Vincent Barat <vincent.barat@gmail.com>
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7;
 rv:17.0) Gecko/17.0 Thunderbird/17.0
MIME-Version: 1.0
To: user@hbase.apache.org
Subject: Re: HBase Tuning
References: <50756F61.8000606@di.uminho.pt>
 <CADcMMgExonu7MNgKVhyMtKZ2vL05KC3ZdLMzV7gBtbhe2feSOQ@mail.gmail.com>
 <5077F774.4090006@di.uminho.pt> <50ABD1FA.1070705@gmail.com>
In-Reply-To: <50ABD1FA.1070705@gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8bit

Forget about this: it does not help

Le 20/11/12 19:54, Vincent Barat a �crit :
> Hi,
>
> It seems there is a potential contention in the HBase client code 
> (a useless synchronized method)
> You may try to use this patch : 
> https://issues.apache.org/jira/browse/HBASE-7069
>
> I face similar issues on my production cluster since I upgraded to 
> HBase 0.92. I will test this patch tomorrow...
> More info matter.
>
> Cheers
>
> Le 12/10/12 12:56, Ricardo Vila�a a �crit :
>> Hi,
>>
>> Em 11/10/12 04:24, Stack escreveu:
>>> On Wed, Oct 10, 2012 at 5:51 AM, Ricardo Vila�a 
>>> <rmvilaca@di.uminho.pt> wrote:
>>>> However, when adding an additional client node, with also 400 
>>>> clients,
>>>> the latency increases 3 times,
>>>> but the RegionServers remains idle more than 80%. I had tried 
>>>> different
>>>> values for the hbase.regionserver.handler.count and also
>>>> for the hbase.client.ipc.pool size and type but without any 
>>>> improvement.
>>>>
>>> I was going to suggest that it sounded like all handlers are
>>> occupied... but it sounds like you tried upping them.
>> Yes, had already tried to increase to 200 but without improvement
>> on the application latency. However, the output of the active IPC
>> handlers, using the Web interface,
>> is strange. For region servers  I can see in a given instant at 
>> most 4
>> IPC handler active but if I
>> see the state of all other IPC handlers they are waiting for 0 
>> seconds.
>> In the master the IPC handlers are also almost all in the waiting 
>> state
>> but for a few seconds.
>>> Is this going from one client node (serving 400 clients) to two 
>>> client
>>> nodes (serving 800 clients)?
>> Yes, the huge increase in latency is when going for one client 
>> node to
>> two client nodes. However, increasing the number of clients in a 
>> single
>> node also adds to latency but a small increase.
>>> Where are you measuring from? Application side?  Can you figure 
>>> if we
>>> are binding up in HBase or in the client node?
>> This measures are from the application  side. As the huge 
>> increase in
>> latency
>> is happening when increasing the number of clients I suspect that 
>> the
>> binding up is in the
>> HBase maybe due to some incorrect configuration.
>>
>>> What does a client node look like?  It is something hosting an 
>>> hbase
>>> client?  A webserver or something?
>> Yes, the client node is hosting an HBase client.
>>>> Is there any configuration parameter that can improve the 
>>>> latency with
>>>> several concurrent threads and more than one HBase client node
>>>> and/or which JMX parameters should I monitor on RegionServers 
>>>> to check
>>>> what may be causing this and how could I achieve better 
>>>> utilization of CPU
>>>> at RegionServers?
>>>>
>>> It sounds like all your data is memory resident given its size 
>>> and the
>>> lack of iowait.  Is that so?  Studying the regionserver metrics, 
>>> are
>>> they fairly constant across the addition of the new client node?
>> Yes, all data is memory resident. As far as I can see, the 
>> regionserver
>> metrics are
>> fairly constant.
>>
>> Thanks,
>>
>