hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-17437) Support specifying a WAL directory outside of the root directory
Date Thu, 26 Jan 2017 21:35:24 GMT

    [ https://issues.apache.org/jira/browse/HBASE-17437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15840496#comment-15840496

stack commented on HBASE-17437:

[~zyork] [~enis] is referring to the fs redo project. Its lead-in JIRA is HBASE-14090. The
project is currently on-hold labelled as a possible for hbase 2.0 (though unlikely). The idea
is that hbase would introduce a logical fs tier undoing ourselves of a direct mapping to a
physical layout and taking on responsibility for atomic actions (making a file visible, deletes,
etc.) ourselves rather than depend on the fs to provide it to us. Our expectation of the fs
would plummet; in essence we'd treat it as a bucket of files; up in hbase is where we'd group
by store/region/table; not in the fs as is done now. The redo would be an opportunity to clean
up filesystem usage in hbase -- it has leaked all over the codebase -- and it would allow
us scale (currently 1M regions is untenable given it means there'll be directory in HDFS with
1M region dirs...) as well as more easily entertain our slotting in another fs chassis other
than HDFS when the Interface has been radically pruned back.

Let me look at this patch up in RB. It was good of @enis to do the fs redo shout out but currently
it doesn't look like it is landing soon so we could take this on in the meantime. Let me take
a look... 

> Support specifying a WAL directory outside of the root directory
> ----------------------------------------------------------------
>                 Key: HBASE-17437
>                 URL: https://issues.apache.org/jira/browse/HBASE-17437
>             Project: HBase
>          Issue Type: Improvement
>          Components: Filesystem Integration, wal
>    Affects Versions: 1.2.4
>            Reporter: Yishan Yang
>            Assignee: Zach York
>              Labels: patch
>             Fix For: 2.0.0, 1.4.0
>         Attachments: hbase-17437-branch-1.2.patch, HBASE-17437.master.001.patch, HBASE-17437.master.002.patch,
HBASE-17437.master.003.patch, HBASE-17437.master.004.patch, hbase-17437-master.patch
> Currently, the WAL and the StoreFiles need to be on the same FileSystem. Some FileSystems
(such as Amazon S3) don’t support append or consistent writes. These two properties are
imperative for the WAL in order to avoid loss of writes. However, StoreFiles don’t necessarily
need the same consistency guarantees (since writes are cached locally and if writes fail,
they can always be replayed from the WAL).
> This JIRA aims to allow users to configure a log directory (for WALs) that is outside
of the root directory or even in a different FileSystem. The default value will still put
the log directory under the root directory.

This message was sent by Atlassian JIRA

View raw message