hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lars Hofhansl (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-5954) Allow proper fsync support for HBase
Date Thu, 09 Oct 2014 06:22:38 GMT

    [ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164816#comment-14164816

Lars Hofhansl commented on HBASE-5954:

I'd like to pick this up again.
In HBase trunk (2.x) can I assume we'll only support versions of Hadoop that have HDFS-744
(i.e. 2.0.2 or later). I think that is the case. If that is indeed the case I can do away
with all reflection and just use the APIs that allow me to set.

Note that this puts the performance burden on the client. The client force 
* sync every edit -> slow
* batch the edits -> caused only one fsync
* perform write with flush only followed by an write with forced sync -> that will force
all prior edit to disk as well (works as we'll automatically sync a block when closed in this

I need to check whether sync-behind-writes can be enabled on a per file basis in HDFS. Sync-on-close
with sync-behind-writes is extremely slow.

In fact as discussed above sync-on-close and sync-behind-writes should be enabled by default
in all HDFS installations. Since HDFS writes immutable files only there is no use hanging
on dirty blocks (in the hopes that they'll be updated again before they get written to disk,
and thus sync-behind-writes should not slow things down - it's possible that there're other
effects of course).

> Allow proper fsync support for HBase
> ------------------------------------
>                 Key: HBASE-5954
>                 URL: https://issues.apache.org/jira/browse/HBASE-5954
>             Project: HBase
>          Issue Type: Improvement
>          Components: HFile, wal
>            Reporter: Lars Hofhansl
>            Assignee: Lars Hofhansl
>            Priority: Critical
>             Fix For: 2.0.0
>         Attachments: 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt,
5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, 5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt
> At least get recommendation into 0.96 doc and some numbers running w/ this hdfs feature

This message was sent by Atlassian JIRA

View raw message