hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pete Wyckoff (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4494) libhdfs does not call FileSystem.append when O_APPEND passed to hdfsOpenFile
Date Thu, 23 Oct 2008 00:34:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12642021#action_12642021

Pete Wyckoff commented on HADOOP-4494:

bq. I just noticed that the test uses O_CREAT but the impl does not even bother checking for
this (ie treats O_WRONLY as create):

Yes, open on writes should i guess be O_WRONLY + one of the following:

1. O_CREAT | O_EXCL - create new file
2. O_CREATE|O_APPEND - create new file if it doesn't exist, else append

But, as you point out hdfsOpenFile has only ever checked for O_WRONLY|O_RDONLY. And #2 cannot
be done without underlying FileSystem API. 

But, as you point out, if someone does something like `echo hello >> foo` where foo
does not exist, libhdfs will do the wrong thing probably.

We need:
FSDataOutputStream FileSystem.appendIfExistsElseCreate(Path p);

I can open a JIRA for it.   I guess that means anyone using libhdfs (e.g. fuse)  would have
to do a hacky check if the file exists and decide whether to pass on the append flag, which
of course is not transactional.

And of course, let's not even get into passing on O_TRUNCATE :)  But, the libhdfs code should
probably check for that flag and error out if it comes.

> libhdfs does not call FileSystem.append when O_APPEND passed to hdfsOpenFile
> ----------------------------------------------------------------------------
>                 Key: HADOOP-4494
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4494
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: libhdfs
>    Affects Versions: 0.19.1, 0.20.0
>            Reporter: Pete Wyckoff
>            Assignee: Pete Wyckoff
>             Fix For: 0.19.1, 0.20.0
>         Attachments: HADOOP-4494.txt
> libhdfs currently calls FileSystem.create when O_APPEND passed in - ie it ignores this

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message