Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Message-ID: <31019703.83791276541235780.JavaMail.jira@thor>
Date: Mon, 14 Jun 2010 14:47:15 -0400 (EDT)
From: "HBase Review Board (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Subject: [jira] Commented: (HBASE-1025) Reconstruction log playback has no
 bounds on memory used
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/HBASE-1025?page=3Dcom.atlassian=
.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D1287=
8695#action_12878695 ]=20

HBase Review Board commented on HBASE-1025:
-------------------------------------------

Message from: stack@duboce.net

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.hbase.org/r/179/
-----------------------------------------------------------

Review request for hbase.


Summary
-------

+ Moved replay of edits up from Store up into Region. This means we play th=
e edits once only rather than once per Store.
+ Lots of cleanup in the replay of edits code. Uses the new flag introduced=
 by cosmin =E2=80=93 hbase.skip.errors. If set, and exception processing ed=
its, we'll fail. If false, we'll move the broke edits file aside and keep g=
oing. Renamed the method from doReconstructionLog to replayRecoveredEdits (=
reconstruction?).
+ The main change in this patch is that recovering edits, we now go in via =
the HRegion main API doing put and delete so that only one code path, so re=
played edits are added to the WAL, and so a flush will be triggered if we f=
ill memory.
+ In HStore, lots of removed code and comments since no longer does log rep=
lay. Cleanup of maximum seqid. Calculate it instead rather than save as a d=
ata member. Its only used once on HRegion startup.
+ Change the HRegion#initialize. It used to take 'initial files' which is a=
 notion never used (it was a means of putting files in place after a split =
but split is done internal to HRegion so can do things in HRegions guts w/o=
 need of exposing notion of initial files). I removed it and added overload=
 that takes no args which is the usual way this method is invoked.
+ Rename the product of splits, 'recovered.edits' instead of 'oldlogfile.lo=
g'
+ Added small facility to HBaseTestingUtility for creating different user i=
n a Configuration so can have more than one Filesystem instance the easier.
+ Redid the test TestStoreReconstruction as TestWALReplay.


This addresses bug hbase-1025.
    http://issues.apache.org/jira/browse/hbase-1025


Diffs
-----

  src/main/java/org/apache/hadoop/hbase/HConstants.java f5d3e94=20
  src/main/java/org/apache/hadoop/hbase/HMerge.java 62f3561=20
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 06e022c=
=20
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java bca=
819e=20
  src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 2a0dcee=20
  src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java 05cf17f=
=20
  src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java dbb21d4=20
  src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java 479c661=20
  src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java 43a8a28=
=20
  src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreReconstructio=
n.java 4f0417d=20
  src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java=
 86cf4ea=20
  src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java=
 PRE-CREATION=20

Diff: http://review.hbase.org/r/179/diff


Testing
-------


Thanks,

stack


> Reconstruction log playback has no bounds on memory used
> --------------------------------------------------------
>
>                 Key: HBASE-1025
>                 URL: https://issues.apache.org/jira/browse/HBASE-1025
>             Project: HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: stack
>             Fix For: 0.21.0
>
>         Attachments: 1025-v2.txt, 1025-v3.txt, 1025-v5.patch, 1025.txt
>
>
> Makes a TreeMap and just keeps adding edits without regard for size of ed=
its applied; could cause OOME (I've not seen a definitive case though have =
seen an OOME around time of a reconstructionlog replay -- perhaps this the =
straw that broke the fleas antlers?)

--=20
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.