hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mike Drob (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-19369) HBase Should use Builder Pattern to Create Log Files while using WAL on Erasure Coding
Date Mon, 01 Jan 2018 18:19:00 GMT

    [ https://issues.apache.org/jira/browse/HBASE-19369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16307521#comment-16307521
] 

Mike Drob commented on HBASE-19369:
-----------------------------------

When hadoop is configured with Erasure Coding (EC - new in 3.0 and maybe 2.9 I think) then
our WALs don't work because EC doesn't support hsync or hflush. To get around this, we can
still request from the NN that the file we create use replication instead of EC but we can
only do so using the builder API, which doesn't exist in the older versions that we support.

Running on hadoop 2.8, we can call {{create}} and {{createNonRecursive}} and the files will
be normal (replicated) files.
Running on hadoop 3.0, we can call those same methods and the files will be either replicated
or erasure coded depending on the policy configured.
Running on hadoop 3.0, we can use new builder API to make sure that the files are replicated.

> HBase Should use Builder Pattern to Create Log Files while using WAL on Erasure Coding
> --------------------------------------------------------------------------------------
>
>                 Key: HBASE-19369
>                 URL: https://issues.apache.org/jira/browse/HBASE-19369
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 2.0.0
>            Reporter: Alex Leblang
>            Assignee: Alex Leblang
>         Attachments: HBASE-19369.master.001.patch, HBASE-19369.master.002.patch, HBASE-19369.master.003.patch,
HBASE-19369.master.004.patch, HBASE-19369.v5.patch, HBASE-19369.v6.patch, HBASE-19369.v7.patch,
HBASE-19369.v8.patch
>
>
> Right now an HBase instance using the WAL won't function properly in an Erasure Coded
environment. We should change the following line to use the hdfs.DistributedFileSystem builder
pattern https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ProtobufLogWriter.java#L92



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message