hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kang Minwoo <minwoo.k...@outlook.com>
Subject Re: HBase client spent most time in ThreadPoolExecutor
Date Tue, 05 Mar 2019 10:38:35 GMT
I found what is a problem.
This is because of HBASE-18665[1].

Best regards,
Minwoo Kang

[1]: https://issues.apache.org/jira/browse/HBASE-18665

________________________________________
보낸 사람: Kang Minwoo <minwoo.kang@outlook.com>
보낸 날짜: 2019년 2월 28일 목요일 11:33
받는 사람: user@hbase.apache.org
제목: Re: HBase client spent most time in ThreadPoolExecutor

Sorry, I have a mistake.
I put wrong HBase Configuration. So that error occurs.

My client's server located in a different datacenter with our HBase server.
I checked the network latency.

The servers' network latency in the same data center is under 0.1ms.
But servers' network latency in other data centers is > 5ms.

I think to create an HBase connection takes more time.

After creating an HBase connection,
Scan data speed difference is little.

Best regards,
Minwoo Kang

________________________________________
보낸 사람: Josh Elser <elserj@apache.org>
보낸 날짜: 2019년 2월 28일 목요일 00:15
받는 사람: user@hbase.apache.org
제목: Re: HBase client spent most time in ThreadPoolExecutor

Do you have any regions in transition? Does HBCK report any problems?

It sounds to me that a client is stuck polling meta to look for the
location of a Region which it cannot find for some reason. Finding the
location of a Region from meta should not take more than a second.

On 2/27/19 12:34 AM, Kang Minwoo wrote:
> MetaScan is so slow.
> When I invoked `regionLocator.getAllRegionLocations()` method, It throw `org.apache.hadoop.hbase.client.RetriesExhaustedException:
Can't get the locations` Exception.
>
> Best regards,
> Minwoo Kang
>
> ________________________________________
> 보낸 사람: Kang Minwoo <minwoo.kang@outlook.com>
> 보낸 날짜: 2019년 2월 27일 수요일 13:48
> 받는 사람: user@hbase.apache.org
> 제목: Re: HBase client spent most time in ThreadPoolExecutor
>
> Thank you for your reply.
>
> I looked around thread dumps.
>
> My Client has 256 connections.
> but only one connection state is RUNNABLE. others state are TIMED_WAITING (parking).
>
> Best regards,
> Minwoo Kang
>
> ________________________________________
> 보낸 사람: Josh Elser <elserj@apache.org>
> 보낸 날짜: 2019년 2월 27일 수요일 01:32
> 받는 사람: user@hbase.apache.org
> 제목: Re: HBase client spent most time in ThreadPoolExecutor
>
> Minwoo,
>
> You have found an idle thread in the threadpool that is waiting for
> work. This is not the source of your slowness. The thread is polling the
> internal queue of work, waiting for the next "unit" of something to do.
>
> You should include threads like these from your analysis.
>
> On 2/26/19 8:32 AM, Kang Minwoo wrote:
>> Hello Users,
>>
>> I have a question.
>>
>> My client complains to me, HBase scan spent too much time.
>> So I started to debug.
>>
>> I profiled the HBase Client Application using hprof.
>> The App spent the most time in below stack trace.
>>
>> ----
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:Unknown line)
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:Unknown
line)
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:Unknown line)
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:Unknown line)
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:Unknown
line)
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:Unknown
line)
>> java.lang.Thread.run(Thread.java:Unknown line)
>> ---
>>
>> ---
>> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.waitForWork(RpcClientImpl.java:Unknown
line)
>> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.run(RpcClientImpl.java:Unknown
line)
>> ---
>>
>> they spent 92% of total time.
>>
>> I don't understand why they spent a lot of time.
>> Do you have any idea?
>>
>> Best regards,
>> Minwoo Kang
>>
Mime
View raw message