hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Guanghao Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
Date Thu, 10 Aug 2017 05:45:00 GMT

    [ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121087#comment-16121087
] 

Guanghao Zhang edited comment on HBASE-18500 at 8/10/17 5:44 AM:
-----------------------------------------------------------------

bq. t.put(Collections.singletonList(p));
When grant user permission, it will put a record to ACL. The old implementation use BufferMutator,
so it will send a multi request. And use AccessController.preBatchMutate to check permission
when put a record. When HTable doesn't use BufferMutator, it will send a mutate request and
use AccessController.prePut to check permission. So I change to use put(List<put>),
this will send multi request, too. Then it will same with before. I thought  AccessController
may have some bugs which need to dig more. But it is not related to this issue?
bq. UTIL.getConfiguration().setInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER, 2);
The default retries number is 35. So it will wait a long time when there are exception in
ut. I thought we should set this to the default HBaseTestingUtility config in the future.


was (Author: zghaobac):
bq. t.put(Collections.singletonList(p));
When grant user permission, it will put a record to ACL. The old implementation use BufferMutator,
so it will send a multi request. And use AccessControlLists.preBatchMutate to check permission
when put a record. When HTable doesn't use BufferMutator, it will send a mutate request and
use AccessControlLists.prePut to check permission. So I change to use put(List<put>),
this will send multi request, too. Then it will same with before. I thought  AccessControlLists
may have some bugs which need to dig more. But it is not related to this issue?
bq. UTIL.getConfiguration().setInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER, 2);
The default retries number is 35. So it will wait a long time when there are exception in
ut. I thought we should set this to the default HBaseTestingUtility config in the future.

> Performance issue: Don't use BufferedMutator for HTable's put method
> --------------------------------------------------------------------
>
>                 Key: HBASE-18500
>                 URL: https://issues.apache.org/jira/browse/HBASE-18500
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Guanghao Zhang
>            Assignee: Guanghao Zhang
>         Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, HBASE-18500-v3.patch,
HBASE-18500-v4.patch, HBASE-18500-v5.patch, HBASE-18500-v5.patch, HBASE-18500-v5.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=100000 --nomapred --autoFlush=True
randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=100000 --nomapred --autoFlush=True
asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the almost
same latency with the blocking write. But for master branch, the result shows the async write
has better latency than the blocking client.  Take a look about the code, I thought the difference
is the BufferedMutator. For master branch, HTable don't have a write buffer and all write
request will be flushed directly. And user can use BufferedMutator when user want to perform
client-side buffering of writes. For the performance issue (autoFlush=True), I thought we
can use rpc caller directly in HTable's put method. Thanks.
> Review: https://reviews.apache.org/r/61454/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message