hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "HBase Review Board (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-1025) Reconstruction log playback has no bounds on memory used
Date Thu, 24 Jun 2010 07:17:53 GMT

    [ https://issues.apache.org/jira/browse/HBASE-1025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882076#action_12882076
] 

HBase Review Board commented on HBASE-1025:
-------------------------------------------

Message from: "Nicolas" <nspiegelberg@facebook.com>


bq.  On 2010-06-22 16:07:23, Nicolas wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/HConstants.java, line 143
bq.  > <http://review.hbase.org/r/179/diff/2/?file=1356#file1356line143>
bq.  >
bq.  >     don't you still want to keep around code to read oldlogfile.log & just remove
write path?  We're not changing Log file format between 0.20=>0.21, so a customer should
be able to cleanly upgrade.
bq.  
bq.  stack wrote:
bq.      I think the format has changed between 0.20 and 0.21, no? (we envelope all edits
on a row now, for example, whereas in 0.20 we just did edits as they came in).
bq.      
bq.      So, to read in old WAL logs, we're talking migration -- reading w/ a class that understands
old format and converting to the new.   But, at least in the past, the first requirement migrating
has been a clean shutdown of old hbase cluster.  On clean shutdown, there should be no WAL
present.   In other words we've always gone out of our way for need migrating WALs across
*major* versions.

I'm assuming the batch edit for SequenceFile you're talking about is the jira kannan/aravind
worked on?  I remember they made that patch backwards compatible.


- Nicolas


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.hbase.org/r/179/#review269
-----------------------------------------------------------





> Reconstruction log playback has no bounds on memory used
> --------------------------------------------------------
>
>                 Key: HBASE-1025
>                 URL: https://issues.apache.org/jira/browse/HBASE-1025
>             Project: HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: stack
>             Fix For: 0.21.0
>
>         Attachments: 1025-v2.txt, 1025-v3.txt, 1025-v5.patch, 1025-v8.txt, 1025.txt
>
>
> Makes a TreeMap and just keeps adding edits without regard for size of edits applied;
could cause OOME (I've not seen a definitive case though have seen an OOME around time of
a reconstructionlog replay -- perhaps this the straw that broke the fleas antlers?)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message