hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3702) Add an option for NOT writing the blocks locally if there is a datanode on the same box as the client
Date Tue, 12 Apr 2016 17:11:25 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15237558#comment-15237558

stack commented on HDFS-3702:

bq. I suggest HBase should do the same way of how it today is using favoredNodes.

Thanks [~szetszwo] for the response, but you did not answer the question. The question was
if you thought the process of first staging a 'hidden' API is a fair burden to put on your
favorite downstream project (not to mention the mess it makes inside HDFS -- see note above
this one for graphic detail).

Lets back up. I think it will help us make some progress here again.

You say:

bq. So let add this flag later so that it allows us to test the feature and see if it is good
enough or we may actually need disfavoredNodes. Sound good?

No. It does not sound good. There is no need to stage a feature as hidden first, one that
is reasonable (see above discussion with the opinion of many), and has an immediate need/user.
If any concern that the feature is lacking or does not work as advertised, lets do whatever
proofing of the feature is needed here as part of this issue and just get it done. If the
bundled tests are unsatisfactory or if you'd like me to try and report result of running this
facility at scale, just say... no problem. If the implementation has a bug, lets fix in a
follow-up. As we would do any other feature in HDFS.

On your concern that a new 'hint' to the create method exposes new API, an API that by definition
does not put a burden on any FS implementation that they need implement the suggested operation
-- i.e. the amount of API 'surface' is miniscule --  it has been suggested above that we flag
it @InterfaceAudience.LimitedPrivate(HBase) for a probationary period. How about we also add
@InterfaceStability.Evolving on the flag so it can be yanked anytime if for some unforeseen
reason, it a total mistake. Would this assuage your exposure concern [~szetszwo]? Thanks for
your time.

> Add an option for NOT writing the blocks locally if there is a datanode on the same box
as the client
> -----------------------------------------------------------------------------------------------------
>                 Key: HDFS-3702
>                 URL: https://issues.apache.org/jira/browse/HDFS-3702
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs-client
>    Affects Versions: 2.5.1
>            Reporter: Nicolas Liochon
>            Assignee: Lei (Eddy) Xu
>            Priority: Minor
>              Labels: BB2015-05-TBR
>         Attachments: HDFS-3702.000.patch, HDFS-3702.001.patch, HDFS-3702.002.patch, HDFS-3702.003.patch,
HDFS-3702.004.patch, HDFS-3702.005.patch, HDFS-3702.006.patch, HDFS-3702.007.patch, HDFS-3702.008.patch,
HDFS-3702.009.patch, HDFS-3702.010.patch, HDFS-3702.011.patch, HDFS-3702_Design.pdf
> This is useful for Write-Ahead-Logs: these files are writen for recovery only, and are
not read when there are no failures.
> Taking HBase as an example, these files will be read only if the process that wrote them
(the 'HBase regionserver') dies. This will likely come from a hardware failure, hence the
corresponding datanode will be dead as well. So we're writing 3 replicas, but in reality only
2 of them are really useful.

This message was sent by Atlassian JIRA

View raw message