hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Raymond Liu (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HBASE-8639) very poor performance of htable.getscanner in multithread env.
Date Wed, 29 May 2013 05:38:20 GMT
Raymond Liu created HBASE-8639:

             Summary: very poor performance of htable.getscanner in multithread env.
                 Key: HBASE-8639
                 URL: https://issues.apache.org/jira/browse/HBASE-8639
             Project: HBase
          Issue Type: Bug
    Affects Versions: 0.94.7
            Reporter: Raymond Liu

Hi, I am running a app on top of phoenix which will fork say around 100+ thread to call htable.getscanner(scan)
to do parallel scan ( say each scan is actually targeting one Region), And each scan will
only match a few result and return thus will be very fast.

under this case, I found that the htable.getscanner(scan) op itself runs pretty slow. by profiling
with jvisualvm. I found 90% of app time is cost on org.apache.hadoop.net.DNS.getDefaultHost.
Which been invoked by scannnercallable.checkIfRegionServerIsRemote. 

The root cause is that DNS.getDefaultHost involves synchronized methods in java.net.Inet4AddressImpl
which have the 100+ thread to lock and wait upon each other. each call to DNS.getDefaultHost
cost around 30ms, while in another case, I run single thread to call 10K times DNS.getDefaultHost
, each cost leas than 0.06ms.

By hacking the code and remove the call to checkIfRegionServerIsRemote, my app runs 5 times
faster, say, 50K op in my app cost 200+ seconds instead of 1000+ seconds.

by check the code further, I found this checkIfRegionServerIsRemote seems just for use of
metrics collection. ( or maybe retry logic?) I am wondering that could this been removed or
switch to some other implementation? so that cases like mine which run large number of small
scan with multi threads could performance way better?

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message