hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-1025) Reconstruction log playback has no bounds on memory used
Date Mon, 14 Jun 2010 19:03:13 GMT

    [ https://issues.apache.org/jira/browse/HBASE-1025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878700#action_12878700

stack commented on HBASE-1025:

Talked to Kannan about comment in the replay log method around next:

I think the comment I added is not valid considering that logReader.next() is working on split
files. I think we should assume here that the split happens successfully at the master, and
that the split files should be clean at the end.

The case that I am not very certain about is what happens if master dies in the middle of
the split. Is the oldfile.log in a potentially invalid state? How do we cleanly restart the
split-- without the RS starting to work on a half-done file?

In any case, in logReader.next() (in Store.java:doReconstructionLog()) I think it is the right
model to assume we have clean/valid log files; and the burden of detecting partial files should
be pushed to splitLog().

I'll remove the comment on commit but will open new issue to see if we can add tests for case
where splitter fails mid-write of files (new master should clean up old stuff and restart

> Reconstruction log playback has no bounds on memory used
> --------------------------------------------------------
>                 Key: HBASE-1025
>                 URL: https://issues.apache.org/jira/browse/HBASE-1025
>             Project: HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: stack
>             Fix For: 0.21.0
>         Attachments: 1025-v2.txt, 1025-v3.txt, 1025-v5.patch, 1025.txt
> Makes a TreeMap and just keeps adding edits without regard for size of edits applied;
could cause OOME (I've not seen a definitive case though have seen an OOME around time of
a reconstructionlog replay -- perhaps this the straw that broke the fleas antlers?)

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message