hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arpit Agarwal (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3702) Add an option for NOT writing the blocks locally if there is a datanode on the same box as the client
Date Thu, 10 Mar 2016 21:42:40 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15189990#comment-15189990
] 

Arpit Agarwal commented on HDFS-3702:
-------------------------------------

Thanks [~eddyxu]. Before we get into specifics of this approach I want to mention that HDFS
now supports storage policies. Was a storage policy based approach considered? It could simplify
the changes to HDFS. You won't need any application changes to HBase since a cluster installer
can set this policy ahead of time on the WAL root directory and it will take effect for all
new blocks.

I am also curious about the answer to [Devaraj's question|https://issues.apache.org/jira/browse/HDFS-3702?focusedCommentId=13720884&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13720884].
HDFS-2576 was added specifically for HBase. Can it address your use case? This avoids any
changes to HDFS.

The design note is rather concise so it didn't answer my questions. The NameNode ignores this
CreateFlag so it will only work for DFSClient users e.g. not for WebHDFS. That will be confusing
to developers. We should also document the flag is advisory. Is it honored for appends? How
does it affect block placement policy - is local rack still preferred for the first replica?

> Add an option for NOT writing the blocks locally if there is a datanode on the same box
as the client
> -----------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-3702
>                 URL: https://issues.apache.org/jira/browse/HDFS-3702
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs-client
>    Affects Versions: 2.5.1
>            Reporter: Nicolas Liochon
>            Assignee: Lei (Eddy) Xu
>            Priority: Minor
>              Labels: BB2015-05-TBR
>         Attachments: HDFS-3702.000.patch, HDFS-3702.001.patch, HDFS-3702.002.patch, HDFS-3702.003.patch,
HDFS-3702.004.patch, HDFS-3702.005.patch, HDFS-3702.006.patch, HDFS-3702.007.patch, HDFS-3702.008.patch,
HDFS-3702_Design.pdf
>
>
> This is useful for Write-Ahead-Logs: these files are writen for recovery only, and are
not read when there are no failures.
> Taking HBase as an example, these files will be read only if the process that wrote them
(the 'HBase regionserver') dies. This will likely come from a hardware failure, hence the
corresponding datanode will be dead as well. So we're writing 3 replicas, but in reality only
2 of them are really useful.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message