hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14394) Provide Builder pattern for DistributedFileSystem.create
Date Tue, 06 Jun 2017 17:48:18 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16039333#comment-16039333

Steve Loughran commented on HADOOP-14394:

I'd like the "create parent dir" to be optional, to stop people expecting it. This can be
done in the base impl with a check for the dir existing & being a dir before calling the
other create(yes, it's non-atomic, so is the standard implementation of check & create).

As I've noted before I'd prefer use string values to set bool/numeric options. 

Mybe we would need to be able to declare whether an option was "optional", or a mandatory
setting, eg.

builder = createFile(path)
builder.mandatory("encryption", true)
builder.opt("erasure-coding", false)
builder.opt("fadvise", "random")
builder.mandatory("hsync", true) // better support real hsync, not pretend to

# lets client code be out of sync with what's shipping on their classpath.
# stops clients needing to know exactly which FS client they've got. All you need to know
is there's a builder you can set things on. Setting an unsupported/not-yet-supported option
isn't a problem, it is if you make it mandatory.
# allows clients to code against a stream without needing the relevant JARs on the CP at compile
time, or hard code for a specific FS.
# I believe it could help some of the filtering filesystems return different clients from
different paths, without the client needing to try casting it to different instances to eventually
get one which supports the option game.
# could also let filter filesystems inject their own options in between. I can't imagine they
would, but they could.

h3. {{TestDistributedFileSystem}}

MiniDFSCluster is autocloseable, you can use it like

try(MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf).numDataNodes(1).build();) {

> Provide Builder pattern for DistributedFileSystem.create
> --------------------------------------------------------
>                 Key: HADOOP-14394
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14394
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs
>    Affects Versions: 2.9.0
>            Reporter: Lei (Eddy) Xu
>            Assignee: Lei (Eddy) Xu
>         Attachments: HADOOP-14394.00.patch, HADOOP-14394.01.patch, HADOOP-14394.02.patch,
> This JIRA continues to refine the {{FSOutputStreamBuilder}} interface introduced in HDFS-11170.

> It should also provide a spec for the Builder API.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message