hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3702) Add an option for NOT writing the blocks locally if there is a datanode on the same box as the client
Date Sat, 09 Apr 2016 05:18:25 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15233363#comment-15233363
] 

stack commented on HDFS-3702:
-----------------------------

bq. I suggest adding a new create(..) method to DistributedFileSystem, either with a new boolean
or with the AddBlockFlag, in this JIRA so that the community can try out the feature. We may
add the CreateFlag.NO_LOCAL_WRITE once the feature has been stabilized and we has decided
that it is the right API.

Tell me more how this will process would work please [~szetszwo]? IIUC, a downstream project,
say HBase which already has an awful hack in place to try and simulate a poor-man's version
of this feature, would via reflection, look first for the presence of this new create override
IFF the implementation is HDFS (don't look if LocalFS or S3, etc.)? If HDFS and if present,
we'd drop our hack and use the new method (via reflection). Later, after it is 'proven' that
a feature, one that hbase has wanted for years now, has 'merit', we would then add a new path
w/ more reflection (IFF the FS implementation is HDFS) that would use the NO_LOCAL_WRITE when
it becomes available? (Would we remove the create override when the NO_LOCAL_WRITE FS hint
gets added?) Are you suggesting that downstream projects do this?

Regards favorednodes, thats an unfinished topic and of a different character to what is being
suggested here as it added overrides rather than a 'hint' flag as this patch does here.

Thanks [~szetszwo]

> Add an option for NOT writing the blocks locally if there is a datanode on the same box
as the client
> -----------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-3702
>                 URL: https://issues.apache.org/jira/browse/HDFS-3702
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs-client
>    Affects Versions: 2.5.1
>            Reporter: Nicolas Liochon
>            Assignee: Lei (Eddy) Xu
>            Priority: Minor
>              Labels: BB2015-05-TBR
>         Attachments: HDFS-3702.000.patch, HDFS-3702.001.patch, HDFS-3702.002.patch, HDFS-3702.003.patch,
HDFS-3702.004.patch, HDFS-3702.005.patch, HDFS-3702.006.patch, HDFS-3702.007.patch, HDFS-3702.008.patch,
HDFS-3702.009.patch, HDFS-3702.010.patch, HDFS-3702.011.patch, HDFS-3702_Design.pdf
>
>
> This is useful for Write-Ahead-Logs: these files are writen for recovery only, and are
not read when there are no failures.
> Taking HBase as an example, these files will be read only if the process that wrote them
(the 'HBase regionserver') dies. This will likely come from a hardware failure, hence the
corresponding datanode will be dead as well. So we're writing 3 replicas, but in reality only
2 of them are really useful.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message