hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "HBase Review Board (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-1025) Reconstruction log playback has no bounds on memory used
Date Mon, 14 Jun 2010 18:47:15 GMT

    [ https://issues.apache.org/jira/browse/HBASE-1025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878695#action_12878695

HBase Review Board commented on HBASE-1025:

Message from: stack@duboce.net

This is an automatically generated e-mail. To reply, visit:

Review request for hbase.


+ Moved replay of edits up from Store up into Region. This means we play the edits once only
rather than once per Store.
+ Lots of cleanup in the replay of edits code. Uses the new flag introduced by cosmin –
hbase.skip.errors. If set, and exception processing edits, we'll fail. If false, we'll move
the broke edits file aside and keep going. Renamed the method from doReconstructionLog to
replayRecoveredEdits (reconstruction?).
+ The main change in this patch is that recovering edits, we now go in via the HRegion main
API doing put and delete so that only one code path, so replayed edits are added to the WAL,
and so a flush will be triggered if we fill memory.
+ In HStore, lots of removed code and comments since no longer does log replay. Cleanup of
maximum seqid. Calculate it instead rather than save as a data member. Its only used once
on HRegion startup.
+ Change the HRegion#initialize. It used to take 'initial files' which is a notion never used
(it was a means of putting files in place after a split but split is done internal to HRegion
so can do things in HRegions guts w/o need of exposing notion of initial files). I removed
it and added overload that takes no args which is the usual way this method is invoked.
+ Rename the product of splits, 'recovered.edits' instead of 'oldlogfile.log'
+ Added small facility to HBaseTestingUtility for creating different user in a Configuration
so can have more than one Filesystem instance the easier.
+ Redid the test TestStoreReconstruction as TestWALReplay.

This addresses bug hbase-1025.


  src/main/java/org/apache/hadoop/hbase/HConstants.java f5d3e94 
  src/main/java/org/apache/hadoop/hbase/HMerge.java 62f3561 
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 06e022c 
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java bca819e 
  src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 2a0dcee 
  src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java 05cf17f 
  src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java dbb21d4 
  src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java 479c661 
  src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java 43a8a28 
  src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreReconstruction.java 4f0417d

  src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java 86cf4ea 
  src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java PRE-CREATION 

Diff: http://review.hbase.org/r/179/diff




> Reconstruction log playback has no bounds on memory used
> --------------------------------------------------------
>                 Key: HBASE-1025
>                 URL: https://issues.apache.org/jira/browse/HBASE-1025
>             Project: HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: stack
>             Fix For: 0.21.0
>         Attachments: 1025-v2.txt, 1025-v3.txt, 1025-v5.patch, 1025.txt
> Makes a TreeMap and just keeps adding edits without regard for size of edits applied;
could cause OOME (I've not seen a definitive case though have seen an OOME around time of
a reconstructionlog replay -- perhaps this the straw that broke the fleas antlers?)

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message