Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Wed, 3 May 2017 18:23:04 +0000 (UTC)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <JIRA.13067938.1493482649000.118204.1493835784309@Atlassian.JIRA>
In-Reply-To: <JIRA.13067938.1493482649000@Atlassian.JIRA>
References: <JIRA.13067938.1493482649000@Atlassian.JIRA> <JIRA.13067938.1493482649240@jira-lw-us.apache.org>
Subject: [jira] [Commented] (HADOOP-14365) Stabilise FileSystem
 builder-based create API
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Wed, 03 May 2017 18:23:10 -0000


    [ https://issues.apache.org/jira/browse/HADOOP-14365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15995378#comment-15995378 ] 

Steve Loughran commented on HADOOP-14365:
-----------------------------------------

well, it went back in to 2.9; I don't want it reaching the outside world until we are happy.  Ideally it wouldn't be on branch-2 yet, but it's been checked in there.  hence me tagging it as a blocker. Given its currently about EC, perhaps the good strategy would be to revert from branch-2, stabilise in trunk & then backport.

Thinking about implementation, I'd concluded that for the generic ones, we could just have a Configuration behind the scenes, some limited setters on the stream builder would let us set generic options, and I could even use the same naming scheme "fs.s3a..." for scheme-specific opts.

I haven't started work, its more something I'd like the authors of the original code to round off: the little details of the API. I'll do my best to provide constructive reviews & help with testing


> Stabilise FileSystem builder-based create API 
> ----------------------------------------------
>
>                 Key: HADOOP-14365
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14365
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.9.0
>            Reporter: Steve Loughran
>            Assignee: Lei (Eddy) Xu
>            Priority: Blocker
>
> HDFS-11170 added a builder-based create API for file creation which has a few issues to work out before it can be considered ready for use
> 1. There no specification in the filesystem.md of what it is meant to do, which means there's no public documentation on expected behaviour except on the Javadocs, which consists of the sentences "Create a new FSDataOutputStreamBuilder for the file with path" and "Base of specific file system FSDataOutputStreamBuilder".
> I propose:
> # Give the new method a relevant name rather than just define the return type, e.g. {{createFile()}}. 
> # `Filesystem.md` to be extended with coverage of this method, and, sadly for the authors, coverage of what the semantics of {{FSDataOutputStreamBuilder.build()}} are.
> 2. There are only tests for HDFS and local, neither of them perfect. Proposed: move to {{AbstractContractCreateTest}}, test for all filesystems, fix tests and FS where appropriate. 
> 3. Add more tests to generate the failure conditions implied by the updated filesystem spec. Eg. create over a an existing file, create over a directory, create with negative buffer size, negative block size, empty dest path, etc, etc. 
> This will clarify when precondition checks are made, as well as whether. For example: should {{newFSDataOutputStreamBuilder()}} validate the path immediately?
> 4. Add to {{FileContext}}.
> 5. Take the opportunity to look at the flaws in today's {{create()}} calls and address them, rather than replicate. In particular, I'd like to end the behaviour "create all parent dirs.


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org