hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lars George (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-6497) Revisit HLog sizing and roll parameters
Date Thu, 02 Aug 2012 06:09:02 GMT

    [ https://issues.apache.org/jira/browse/HBASE-6497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427130#comment-13427130
] 

Lars George commented on HBASE-6497:
------------------------------------

The goal in designing a proper HBase schema is to maximize heap usage across all regions,
which can lead to the situation where the WALs (aka HLog's) are required to be kept for a
considerable amount of time. 

The last iteration on WAL properties added a configurable block size, as well as threshold
percentage to roll the log before it completely fills the single HDFS block (see HBASE-1394).

I am questioning if this is still in issue, maybe even in the light of recent improvements
on log performance, for example HBASE-5699 and HBASE-4608.

At the least, I would like to figure out, if we should increase the WAL size to 512MB, to
avoid getting into early flushing situations, impacting the overall I/O. Isn't HBASE-1364
helping to split larger logs (though not the logs themselves but distributed across the region
servers obviously). I am not sure if the log splitting prefers block local nodes first, so
that there is no remote reading though.

Questions:

# Is there a need to keep the logs small (typically 64-128 depending on the HDFS config)?
# Should we go multiple blocks?
# Do we still need the logroll multiplier?
# Should we increase the maxlogs number (default is 32)?
                
> Revisit HLog sizing and roll parameters
> ---------------------------------------
>
>                 Key: HBASE-6497
>                 URL: https://issues.apache.org/jira/browse/HBASE-6497
>             Project: HBase
>          Issue Type: Improvement
>          Components: regionserver
>            Reporter: Lars George
>
> The last major update to the HLog sizing and roll features were done in HBASE-1394. I
am proposing to revisit these settings to overcome recent issues where the HLog becomes a
major bottleneck.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message