hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tom White (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-5438) Merge FileSystem.create and FileSystem.append
Date Mon, 07 Jun 2010 05:22:13 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876129#action_12876129
] 

Tom White commented on HADOOP-5438:
-----------------------------------

I think this needs more work to find an API that there is broad agreement on. It should be
undertaken as a larger piece of work to consolidate the create API (this would include HADOOP-3252
as Jakob pointed out), including deprecating old methods.

This change doesn't enable anything that wasn't previously possible, does it? In other words,
backing it out wouldn't remove functionality for users of released versions of Hadoop. For
example, append and overwrite operations are possible with the existing API. Since it hasn't
been released yet, I'm +1 to reverting this while the create API gets more discussion and
testing (especially with HDFS append).


> Merge FileSystem.create and FileSystem.append
> ---------------------------------------------
>
>                 Key: HADOOP-5438
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5438
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: He Yongqiang
>            Assignee: He Yongqiang
>             Fix For: 0.21.0
>
>         Attachments: Hadoop-5438(2009-04-06).patch, Hadoop-5438-2009-03-30.patch, Hadoop-5438-2009-03-31-2.patch,
Hadoop-5438-2009-03-31.patch, Hadoop-5438-2009-05-10.patch, Hadoop-5438-2009-05-15.patch,
Hadoop-5438-2009-05-19.patch, Hadoop-5438-2009-05-5.patch
>
>
> Currently, when a user wants to modify a file, the user first calls exists() to know
if this file is already there. And then uses create() or append() according to whether the
file exists or not.
> the code looks like:
> {code}
> FSDataOutputStream out_1 = null;
> if (fs.exists(path_1))
>    out_1 = fs.append(path_1);
> else
>    out_1 = fs.create(path_1);
> {code}
> . On the performace side,It involes two RPCs. On the easy-of-use side, it is not very
convient in contrast to the traditional open interface.
> It will more complicate if there is a overwrite parameter specified. I donot know whether
there is a bug about 'overwrite' in 0.19, some times it takes a long time for overwrite creates
to reture. So i make the write file code with overwrite param works like:
> {code}
> boolean exists = fs.exists(name);
> if (overwrite) {
>     if (exists)
>        fs.delete(name, true);
>      this.out = fs.create(name, overwrite, bufferSize, replication,
> 				    blockSize, progress);
>      this.currentRowID = 0;
>  } else {
>    if (!exists)
> 	this.out = fs.create(name, overwrite, bufferSize,
> 					replication, blockSize, progress);
>    else
> 	this.out = fs.append(name, bufferSize, progress);
> {code}
> Some code statements there are really redundant and not needed, especialy with the delete().
But without deleting first, the overwrite takes a long time to reture.
> BTW, i will create another issue about the overwrite problem. If it is not a bug at all
or a duplicate, someone please close it.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message