hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ted Yu (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-4120) isolation and allocation
Date Wed, 02 Nov 2011 20:41:33 GMT

    [ https://issues.apache.org/jira/browse/HBASE-4120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142493#comment-13142493

Ted Yu commented on HBASE-4120:

Here is some background for the current implementation provided by Liu Jia:

We have moved the main functionality of table priority (HBASE-4120) to PriorityFunction and
PriorityFunction implements Function<Writable, Integer> and is used by the PriorityJobQueue
to decide the priority of each IPC call.
Through this we make the PriorityHBaseServer much simpler.

In current implementation of HBase, there would be one priority for each method:
      for (Method m : HRegionServer.class.getMethods()) {
        QosPriority p = m.getAnnotation(QosPriority.class);
But in our implementation, method can have different priorities for different HBase tables.
e.g. for tables A and B in the same cluster, table A serves reads mostly while table B serves
writes more often.
we may want to give high priority to get() for table A while giving low priority to put().
For table B, we may want low priority for get() and high priority for put().

That's why we introduced ActionPrioriy for the methods (such as get and put). At the same
time, tables have their base priorities.
We believe the above approach is more flexible.

We do not interfere with the current Qos Function,which handles the meta and root request
but in the future a more generalized approach is better.

> isolation and allocation
> ------------------------
>                 Key: HBASE-4120
>                 URL: https://issues.apache.org/jira/browse/HBASE-4120
>             Project: HBase
>          Issue Type: New Feature
>          Components: master, regionserver
>    Affects Versions: 0.90.2, 0.90.3, 0.90.4, 0.92.0
>            Reporter: Liu Jia
>            Assignee: Liu Jia
>             Fix For: 0.94.0
>         Attachments: Design_document_for_HBase_isolation_and_allocation.pdf, Design_document_for_HBase_isolation_and_allocation_Revised.pdf,
HBase_isolation_and_allocation_user_guide.pdf, Performance_of_Table_priority.pdf, System Structure.jpg,
> The HBase isolation and allocation tool is designed to help users manage cluster resource
among different application and tables.
> When we have a large scale of HBase cluster with many applications running on it, there
will be lots of problems. In Taobao there is a cluster for many departments to test their
applications performance, these applications are based on HBase. With one cluster which has
12 servers, there will be only one application running exclusively on this server, and many
other applications must wait until the previous test finished.
> After we add allocation manage function to the cluster, applications can share the cluster
and run concurrently. Also if the Test Engineer wants to make sure there is no interference,
he/she can move out other tables from this group.
> In groups we use table priority to allocate resource, when system is busy; we can make
sure high-priority tables are not affected lower-priority tables
> Different groups can have different region server configurations, some groups optimized
for reading can have large block cache size, and others optimized for writing can have large
memstore size. 
> Tables and region servers can be moved easily between groups; after changing the configuration,
a group can be restarted alone instead of restarting the whole cluster.
> git entry : https://github.com/ICT-Ope/HBase_allocation .
> We hope our work is helpful.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message