Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Fri, 12 Aug 2011 23:33:28 +0000 (UTC)
From: "jiraposter@reviews.apache.org (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: 
 <1817883143.35168.1313192008911.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <1766608612.27046.1313045727156.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HBASE-4192) Optimize HLog for Throughput Using
 Delayed RPCs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HBASE-4192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084468#comment-13084468 ] 

jiraposter@reviews.apache.org commented on HBASE-4192:
------------------------------------------------------


bq.  On 2011-08-11 20:11:34, Jonathan Gray wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java, lines 261-299
bq.  > <https://reviews.apache.org/r/1463/diff/1/?file=32100#file32100line261>
bq.  >
bq.  >     This is all static so there might be issues running multiple HRegionServers in a single JVM during unit tests?

I am not sure what you mean.  Should I change only the newly added functions or all of them?


bq.  On 2011-08-11 20:11:34, Jonathan Gray wrote:
bq.  > src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestBatchEntriesLogRolling.java, line 63
bq.  > <https://reviews.apache.org/r/1463/diff/1/?file=32106#file32106line63>
bq.  >
bq.  >     there are some ways to run all the unit tests with a varying set of initial parameters... look at TestCacheOnWrite on the new HFile v2 patch, it uses @RunWith(Parameterized.class)
bq.  >

Looking into that, test should also do some more careful cleanup between successive (parameterized) runs.


- Vlad


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1463/#review1410
-----------------------------------------------------------


On 2011-08-11 06:59:28, Vlad Dogaru wrote:
bq.  
bq.  -----------------------------------------------------------
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/1463/
bq.  -----------------------------------------------------------
bq.  
bq.  (Updated 2011-08-11 06:59:28)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  -------
bq.  
bq.  Changes:
bq.  
bq.  1.  Add hbase.region.wal.batchentries configuration parameter.  If this is
bq.  enabled, batch entries to the HLog in a queue.
bq.  2.  Use delayed RPCs for sync requests when aggresive batching is enabled.
bq.  This frees up RPC handler threads for the duration of the sync.
bq.  3.  Pass the RPC server instance all the way to down to HLog.  This is needed
bq.  to find out the current remote call, mark it as delayed, and finally complete
bq.  it when the sync is done.
bq.  4.  Use the region read-write consistency control to avoid exposing to
bq.  RegionScanners edits which have not yet been synced.
bq.  5.  Change a few tests which directly create HRegions or HLogs.  The
bq.  rpcServers passed in are null, HLog falls back to classic RPCs when it has no
bq.  knowledge of the RPC server.
bq.  6.  Add TestBatchEntriesLogRolling, which is identical to TestLogRolling,
bq.  except that it uses aggressive batching.  I'm not sure how to add tests that
bq.  verify the same functionality but don't duplicate code, suggestion are
bq.  welcome.
bq.  
bq.  The new parameter is disabled by default.
bq.  
bq.  
bq.  This addresses bug HBASE-4192.
bq.      https://issues.apache.org/jira/browse/HBASE-4192
bq.  
bq.  
bq.  Diffs
bq.  -----
bq.  
bq.    src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 6fb1da7 
bq.    src/main/java/org/apache/hadoop/hbase/master/HMaster.java a00b93d 
bq.    src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 2f86f04 
bq.    src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 83ff7b2 
bq.    src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java f22fb6e 
bq.    src/main/java/org/apache/hadoop/hbase/regionserver/ReadWriteConsistencyControl.java 8ec53d3 
bq.    src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java 887f736 
bq.    src/main/java/org/apache/hadoop/hbase/util/HMerge.java 9f0499e 
bq.    src/main/java/org/apache/hadoop/hbase/util/MetaUtils.java af8d734 
bq.    src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java bf8004b 
bq.    src/test/java/org/apache/hadoop/hbase/TestScanMultipleVersions.java d7baba4 
bq.    src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java 36816e8 
bq.    src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestBatchEntriesLogRolling.java PRE-CREATION 
bq.    src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java de28418 
bq.    src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALActionsListener.java dc43eb2 
bq.    src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java 381ac90 
bq.  
bq.  Diff: https://reviews.apache.org/r/1463/diff
bq.  
bq.  
bq.  Testing
bq.  -------
bq.  
bq.  All unit tests run with aggressive batching turned on and off.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Vlad
bq.  
bq.


> Optimize HLog for Throughput Using Delayed RPCs
> -----------------------------------------------
>
>                 Key: HBASE-4192
>                 URL: https://issues.apache.org/jira/browse/HBASE-4192
>             Project: HBase
>          Issue Type: New Feature
>          Components: wal
>    Affects Versions: 0.92.0
>            Reporter: Vlad Dogaru
>            Priority: Minor
>
> Introduce a new HLog configuration parameter (batchEntries) for more aggressive batching of appends.  If this is enabled, HLog appends are not written to the HLog writer immediately, but batched and written either periodically or when a sync is requested.  Because sync times become larger, they use delayed RPCs to free up RPC handler threads.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira