hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jiraposter@reviews.apache.org (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-4358) Batch Table Alter Operations
Date Fri, 09 Sep 2011 20:48:11 GMT

    [ https://issues.apache.org/jira/browse/HBASE-4358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101551#comment-13101551
] 

jiraposter@reviews.apache.org commented on HBASE-4358:
------------------------------------------------------



bq.  On 2011-09-09 19:48:45, Ted Yu wrote:
bq.  > /src/main/java/org/apache/hadoop/hbase/master/handler/TableAddFamilyHandler.java,
line 64
bq.  > <https://reviews.apache.org/r/1768/diff/1/?file=38941#file38941line64>
bq.  >
bq.  >     After HBASE-451, changes to table descriptor have to be persisted to HDFS.
bq.  >     
bq.  >     I browsed handleTableOperation() methods in this patch and didn't find that.

I'm not familiar enough with how exactly table descriptors are persisted to be able to tell
you for certain that this approach correctly ensures persistence. But I can confidently tell
you that this diff does everything that the current trunk does with regards to updating table
descriptors. If you look at the actual implementation of MasterFileSystem.{add,modify,Delete}Column(),
it gets a table descriptor from master services, modifies the table descriptor appropriately,
then adds it back to master services' table descriptors. Between handleTableOperation() and
updateTableDescriptor(), this patch follows the exact same procedure, and has the same instance
of MasterServices. This separation is for the purpose of enabling batching to happen in a
way that doesn't leave the system in an intermediate state in the case of a thrown exception.
From basic testing, I see that the table descriptor changes are actually being written to
my fs. If this procedure is not enough to actually ensure that this happens, we should file
a separate JIRA to look into it.


- Riley


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1768/#review1843
-----------------------------------------------------------


On 2011-09-09 18:39:05, Riley Patterson wrote:
bq.  
bq.  -----------------------------------------------------------
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/1768/
bq.  -----------------------------------------------------------
bq.  
bq.  (Updated 2011-09-09 18:39:05)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  -------
bq.  
bq.  Currently, the RPC provides no way of asking for several table alterations at once, and
the master has no way of batch handling alter requests. Thus, when the user requests several
changes at the same time (i.e. add these I columns, delete these J columns, and modify these
K columns), each region is brought down (I+J+K) times so that it can reflect the new schema.
Additionally, multiple writes are made to META, and multiple RPC calls must be made.
bq.  
bq.  This patch provides batching for these operations, both at the RPC level and within the
Master's TableEventHandlers. This involves a bit of reorganization in the TableEventHandler
class hierarchy, and a new TableEventHandler, TableMultiFamilyHandler. The net effect ends
up being the difference seen here:
bq.  
bq.  Before patch:
bq.  hbase(main):001:0> alter 'peeps', {NAME => 'rawr'}, {METHOD => 'delete', NAME
=> 'name'}
bq.  Updating all regions with the new schema...
bq.  1/1 regions updated.
bq.  Done.
bq.  Updating all regions with the new schema...
bq.  1/1 regions updated.
bq.  Done.
bq.  0 row(s) in 2.6450 seconds
bq.  
bq.  After patch:
bq.  hbase(main):002:0> alter 'peeps', {NAME => 'rawr'}, {METHOD => 'delete', NAME
=> 'name'}
bq.  Updating all regions with the new schema...
bq.  1/1 regions updated.
bq.  Done.
bq.  0 row(s) in 1.1930 seconds
bq.  
bq.  Regions are only brought down once, and the duration is cut 1/N.
bq.  
bq.  
bq.  This addresses bug HBASE-4358.
bq.      https://issues.apache.org/jira/browse/HBASE-4358
bq.  
bq.  
bq.  Diffs
bq.  -----
bq.  
bq.    /src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 1166933 
bq.    /src/main/java/org/apache/hadoop/hbase/coprocessor/BaseMasterObserver.java 1166933

bq.    /src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java 1166933 
bq.    /src/main/java/org/apache/hadoop/hbase/executor/EventHandler.java 1166933 
bq.    /src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 1166933 
bq.    /src/main/java/org/apache/hadoop/hbase/master/HMaster.java 1166933 
bq.    /src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java 1166933 
bq.    /src/main/java/org/apache/hadoop/hbase/master/handler/TableAddFamilyHandler.java 1166933

bq.    /src/main/java/org/apache/hadoop/hbase/master/handler/TableDeleteFamilyHandler.java
1166933 
bq.    /src/main/java/org/apache/hadoop/hbase/master/handler/TableFamilyHandler.java PRE-CREATION

bq.    /src/main/java/org/apache/hadoop/hbase/master/handler/TableModifyFamilyHandler.java
1166933 
bq.    /src/main/java/org/apache/hadoop/hbase/master/handler/TableMultiFamilyHandler.java
PRE-CREATION 
bq.    /src/main/ruby/hbase/admin.rb 1166933 
bq.    /src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java 1166933

bq.  
bq.  Diff: https://reviews.apache.org/r/1768/diff
bq.  
bq.  
bq.  Testing
bq.  -------
bq.  
bq.  Sanity checked functionality in psuedo-distributed mode (tried several permutations of
different alterations, all completed successfully and with only one round of region restarts).
Ran all unit tests successfully.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Riley
bq.  
bq.



> Batch Table Alter Operations
> ----------------------------
>
>                 Key: HBASE-4358
>                 URL: https://issues.apache.org/jira/browse/HBASE-4358
>             Project: HBase
>          Issue Type: Improvement
>          Components: ipc, master, shell
>    Affects Versions: 0.92.0
>            Reporter: Riley Patterson
>            Assignee: Riley Patterson
>            Priority: Minor
>         Attachments: HBASE-4358.patch
>
>
> Currently, the RPC provides no way of asking for several table alterations at once, and
the master has no way of batch handling alter requests. Thus, when the user requests several
changes at the same time (i.e. add these I columns, delete these J columns, and modify these
K columns), each region is brought down (I+J+K) times so that it can reflect the new schema.
Additionally, multiple writes are made to META, and multiple RPC calls must be made.
> This patch provides batching for these operations, both at the RPC level and within the
Master's TableEventHandlers. This involves a bit of reorganization in the TableEventHandler
class hierarchy, and a new TableEventHandler, TableMultiFamilyHandler. The net effect ends
up being the difference seen here:
> Before patch:
> hbase(main):001:0> alter 'peeps', {NAME => 'rawr'}, {METHOD => 'delete', NAME
=> 'name'}
> Updating all regions with the new schema...
> 1/1 regions updated.
> Done.
> Updating all regions with the new schema...
> 1/1 regions updated.
> Done.
> 0 row(s) in 2.6450 seconds
> After patch:
> hbase(main):002:0> alter 'peeps', {NAME => 'rawr'}, {METHOD => 'delete', NAME
=> 'name'}
> Updating all regions with the new schema...
> 1/1 regions updated.
> Done.
> 0 row(s) in 1.1930 seconds
> Regions are only brought down once, and the duration is cut 1/N.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

Mime
View raw message