hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jean-Daniel Cryans (JIRA)" <j...@apache.org>
Subject [jira] Issue Comment Edited: (HBASE-1430) Read the logs in batches during log splitting to avoid OOME
Date Tue, 19 May 2009 16:48:45 GMT

    [ https://issues.apache.org/jira/browse/HBASE-1430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710789#action_12710789

Jean-Daniel Cryans edited comment on HBASE-1430 at 5/19/09 9:48 AM:

Comment from JK in HBASE-1008:
I'm not a big fan of having to read all the logs into memory.

My suggestion would be for each unique region in the HLog(s), create a blocking queue and
a thread that will
dequeue entries and write them directly to the log file. Then you have one thread doing the
reading and multiple
threads writing, and the memory footprint is reduced significantly.

Make sense?

      was (Author: jdcryans):
    Comment from JK in HBASE-1008:

> Read the logs in batches during log splitting to avoid OOME
> -----------------------------------------------------------
>                 Key: HBASE-1430
>                 URL: https://issues.apache.org/jira/browse/HBASE-1430
>             Project: Hadoop HBase
>          Issue Type: Improvement
>            Reporter: Jean-Daniel Cryans
>            Assignee: Jean-Daniel Cryans
>            Priority: Blocker
>             Fix For: 0.20.0, 0.19.3
> During log splitting, in the worst case we can read 64 logs of 64MB which will surely
bust any heap. Instead we should read the logs in batches of 5-10 that we write right away.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message