hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jiraposter@reviews.apache.org (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-4120) isolation and allocation
Date Tue, 22 Nov 2011 23:36:42 GMT

    [ https://issues.apache.org/jira/browse/HBASE-4120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155568#comment-13155568
] 

jiraposter@reviews.apache.org commented on HBASE-4120:
------------------------------------------------------


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1421/#review3444
-----------------------------------------------------------


initial comments.  haven't had time to look through the whole diff in detail.  I'm guessing
this was made against 0.90?  This is missing integration with a lot of new functionality added
a while ago into 92.


http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java
<https://reviews.apache.org/r/1421/#comment7687>

    should this functionality not be disabled by default?



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityFunction.java
<https://reviews.apache.org/r/1421/#comment7696>

    this pull model seems a little hacky.  why can't we just push this information when a
region comes online on this server?  we have online schema updates, so we can update the table
priority with minimal downtime.



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityFunction.java
<https://reviews.apache.org/r/1421/#comment7694>

    this pull model seems a little hacky.  why can't we just push this information when a
region comes online on this server?  we have online schema updates, so we can update the table
priority with minimal downtime.



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityFunction.java
<https://reviews.apache.org/r/1421/#comment7688>

    instead of this big switch statement, why don't you add a getPriority() function to the
Operation base class?  All database operations (gets, puts, deletes) are supposed to override
this anyways do we can print out fingerprint and other debug information.  You can see its
use in WritableRpcEngine.java



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityFunction.java
<https://reviews.apache.org/r/1421/#comment7689>

    your intent is to lower the priority of multiputs, correct?  I see that HIGHEST_PRI =
-10.  If a MultiAction has that priority, won't it make a MultiAction<T> have a higher
priority than T?



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityFunction.java
<https://reviews.apache.org/r/1421/#comment7690>

    again, I'm confused about why we need to contact the master to get this information. 
we only care about the regions that are online for this server, correct?



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityFunction.java
<https://reviews.apache.org/r/1421/#comment7691>

    can you explain this more?  what does this mapping look like?  In other words
    
    thread pri 1-10 == system pri X-Y



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityJobQueue.java
<https://reviews.apache.org/r/1421/#comment7700>

    why are all these accessors needed?  Why can't you just use 
    
    if(this.queue.size() < this.capacity) queueFull.signal()



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityJobQueue.java
<https://reviews.apache.org/r/1421/#comment7698>

    this should be signal() instead of signalAll().  You will only decrease capacity by 1,
so you should only wake 1 thread.



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityJobQueue.java
<https://reviews.apache.org/r/1421/#comment7692>

    so, if you've waited too long, you'll add the Job to the queue anyways?  it's not a tryAdd()
then, which would imply that the action could fail.  It's just a normal add()



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityJobQueue.java
<https://reviews.apache.org/r/1421/#comment7699>

    there is a race condition between (size >= capacity) and addLock.  Shouldn't you test
the condition again after getting the lock?



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityJobQueue.java
<https://reviews.apache.org/r/1421/#comment7701>

    LOG.debug


- Nicolas


On 2011-11-22 06:06:45, Jia Liu wrote:
bq.  
bq.  -----------------------------------------------------------
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/1421/
bq.  -----------------------------------------------------------
bq.  
bq.  (Updated 2011-11-22 06:06:45)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  -------
bq.  
bq.  Patch used for table priority alone,In this patch, not only tables can have different
priorities but also the different actions like "get","scan","put" and "delete" can have priorities.
bq.  
bq.  
bq.  This addresses bug HBase-4120.
bq.      https://issues.apache.org/jira/browse/HBase-4120
bq.  
bq.  
bq.  Diffs
bq.  -----
bq.  
bq.    http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java
1189169 
bq.    http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
1189169 
bq.    http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityFunction.java
PRE-CREATION 
bq.    http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityHBaseServer.java
PRE-CREATION 
bq.    http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityJobQueue.java
PRE-CREATION 
bq.    http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
1189169 
bq.    http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/allocation/test/TestActionPriority.java
PRE-CREATION 
bq.    http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/allocation/test/TestPriorityJobQueue.java
PRE-CREATION 
bq.    http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/allocation/test/TestTablePriority.java
PRE-CREATION 
bq.  
bq.  Diff: https://reviews.apache.org/r/1421/diff
bq.  
bq.  
bq.  Testing
bq.  -------
bq.  
bq.  Tested with test cases in  TestCase_For_TablePriority_trunk_v1.patch 
bq.  please apply the patch of HBASE-4181 first,in some circumstances this bug will affect
the performance of client.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Jia
bq.  
bq.


                
> isolation and allocation
> ------------------------
>
>                 Key: HBASE-4120
>                 URL: https://issues.apache.org/jira/browse/HBASE-4120
>             Project: HBase
>          Issue Type: New Feature
>          Components: master, regionserver
>    Affects Versions: 0.90.2, 0.90.3, 0.90.4, 0.92.0
>            Reporter: Liu Jia
>            Assignee: Liu Jia
>             Fix For: 0.94.0
>
>         Attachments: Design_document_for_HBase_isolation_and_allocation.pdf, Design_document_for_HBase_isolation_and_allocation_Revised.pdf,
HBase_isolation_and_allocation_user_guide.pdf, Performance_of_Table_priority.pdf, System Structure.jpg,
TablePriority.patch, TablePriority_v8.patch, TablePriority_v8.patch, TablePriority_v8_for_trunk.patch
>
>
> The HBase isolation and allocation tool is designed to help users manage cluster resource
among different application and tables.
> When we have a large scale of HBase cluster with many applications running on it, there
will be lots of problems. In Taobao there is a cluster for many departments to test their
applications performance, these applications are based on HBase. With one cluster which has
12 servers, there will be only one application running exclusively on this server, and many
other applications must wait until the previous test finished.
> After we add allocation manage function to the cluster, applications can share the cluster
and run concurrently. Also if the Test Engineer wants to make sure there is no interference,
he/she can move out other tables from this group.
> In groups we use table priority to allocate resource, when system is busy; we can make
sure high-priority tables are not affected lower-priority tables
> Different groups can have different region server configurations, some groups optimized
for reading can have large block cache size, and others optimized for writing can have large
memstore size. 
> Tables and region servers can be moved easily between groups; after changing the configuration,
a group can be restarted alone instead of restarting the whole cluster.
> git entry : https://github.com/ICT-Ope/HBase_allocation .
> We hope our work is helpful.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message