hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3702) Add an option for NOT writing the blocks locally if there is a datanode on the same box as the client
Date Wed, 23 Mar 2016 21:03:25 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15209164#comment-15209164

stack commented on HDFS-3702:

bq. stack, let's add a boolean noLoaclWrite to DistributedFileSystem or just reuse the new
AddBlockFlag there. 

On adding a flag to DFS, taking a look, it would be 'odd' given what is there currently, and
adding a public method to set a hint for a particular operation only would be tough to explain
to the reader of the API ("Why flag here... when create takes flags already..."). Then there
is the fact that the user has to do {code}if HDFS, then{code} and if we are on GPFS, an FS
supported by one of our committers, then it is {code} if HDFS || GPFS....{code} and so on.

I think you also mean 'and' in the above rather than 'or'. AddBlockFlag is internal to HDFS
and marked Private so...not useable by clients.... maybe you are talking of how it will be
implemented.  I'm not sure what you are suggesting here. Pardon me.

bq. You know, once it is in FileSystem, it is forever.

I know that for the client to ask for a behavior that is not there presently, yes, FileSystem
has to change. We are talking about a self-described advisory, not a required new operation
of the underlying FS.

> Add an option for NOT writing the blocks locally if there is a datanode on the same box
as the client
> -----------------------------------------------------------------------------------------------------
>                 Key: HDFS-3702
>                 URL: https://issues.apache.org/jira/browse/HDFS-3702
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs-client
>    Affects Versions: 2.5.1
>            Reporter: Nicolas Liochon
>            Assignee: Lei (Eddy) Xu
>            Priority: Minor
>              Labels: BB2015-05-TBR
>         Attachments: HDFS-3702.000.patch, HDFS-3702.001.patch, HDFS-3702.002.patch, HDFS-3702.003.patch,
HDFS-3702.004.patch, HDFS-3702.005.patch, HDFS-3702.006.patch, HDFS-3702.007.patch, HDFS-3702.008.patch,
HDFS-3702.009.patch, HDFS-3702_Design.pdf
> This is useful for Write-Ahead-Logs: these files are writen for recovery only, and are
not read when there are no failures.
> Taking HBase as an example, these files will be read only if the process that wrote them
(the 'HBase regionserver') dies. This will likely come from a hardware failure, hence the
corresponding datanode will be dead as well. So we're writing 3 replicas, but in reality only
2 of them are really useful.

This message was sent by Atlassian JIRA

View raw message