hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "HBase Review Board (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-1364) [performance] Distributed splitting of regionserver commit logs
Date Fri, 23 Jul 2010 00:35:54 GMT

    [ https://issues.apache.org/jira/browse/HBASE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891424#action_12891424
] 

HBase Review Board commented on HBASE-1364:
-------------------------------------------

Message from: "Alex Newman" <newalex@cloudera.com>

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.hbase.org/r/370/
-----------------------------------------------------------

Review request for hbase.


Summary
-------

This build on the previous work. It does some smarter stuff with testing and now splitting
is configurable.


This addresses bug hbase-1364.
    http://issues.apache.org/jira/browse/hbase-1364


Diffs
-----

  src/main/java/org/apache/hadoop/hbase/HConstants.java c77ebf5 
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java f251d54 
  src/main/java/org/apache/hadoop/hbase/regionserver/LogSplitter.java PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java 5688c03 
  src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWrapper.java 8225178 
  src/main/resources/hbase-default.xml e3a9669 
  src/test/java/org/apache/hadoop/hbase/regionserver/wal/BaseTestHLogSplit.java PRE-CREATION

  src/test/java/org/apache/hadoop/hbase/regionserver/wal/DistributedTestHLog.java PRE-CREATION

  src/test/java/org/apache/hadoop/hbase/regionserver/wal/DistributedTestHLogSplit.java PRE-CREATION

  src/test/java/org/apache/hadoop/hbase/regionserver/wal/DistributedTestHLogSplitSkipErrors.java
PRE-CREATION 
  src/test/java/org/apache/hadoop/hbase/regionserver/wal/DistributedTestLogRolling.java PRE-CREATION

  src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java ad8f9e5 
  src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java 908633e 
  src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplitSkipErrors.java PRE-CREATION

  src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogActionsListener.java 776d78c

  src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java 9eae4b4 
  src/test/resources/hbase-site.xml 3c0601a 

Diff: http://review.hbase.org/r/370/diff


Testing
-------

ran on our private hudson


Thanks,

Alex




> [performance] Distributed splitting of regionserver commit logs
> ---------------------------------------------------------------
>
>                 Key: HBASE-1364
>                 URL: https://issues.apache.org/jira/browse/HBASE-1364
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: stack
>            Assignee: Alex Newman
>            Priority: Critical
>             Fix For: 0.92.0
>
>         Attachments: 1 (3), 1364-v2.patch, 1364.patch
>
>          Time Spent: 8h
>  Remaining Estimate: 0h
>
> HBASE-1008 has some improvements to our log splitting on regionserver crash; but it needs
to run even faster.
> (Below is from HBASE-1008)
> In bigtable paper, the split is distributed. If we're going to have 1000 logs, we need
to distribute or at least multithread the splitting.
> 1. As is, regions starting up expect to find one reconstruction log only. Need to make
it so pick up a bunch of edit logs and it should be fine that logs are elsewhere in hdfs in
an output directory written by all split participants whether multithreaded or a mapreduce-like
distributed process (Lets write our distributed sort first as a MR so we learn whats involved;
distributed sort, as much as possible should use MR framework pieces). On startup, regions
go to this directory and pick up the files written by split participants deleting and clearing
the dir when all have been read in. Making it so can take multiple logs for input, can also
make the split process more robust rather than current tenuous process which loses all edits
if it doesn't make it to the end without error.
> 2. Each column family rereads the reconstruction log to find its edits. Need to fix that.
Split can sort the edits by column family so store only reads its edits.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message