hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "John George (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-8319) FileContext does not support setWriteChecksum
Date Thu, 03 May 2012 17:04:48 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-8319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267585#comment-13267585

John George commented on HADOOP-8319:

bq. First, a general question: why change the visible of LocalFs ctor to public?
It is needed by one of the tests (that is not in the same package), which unfortunately was
missing in the latest patch I uploaded. Will upload the right patch soon

bq. This patch doesn't work when a path symlinks off to a different afs. fc.setWriteChecksum(boolean,
Path) will instantiate the resolved afs, set the flag, the afs instance goes "poof", and the
subsequent create will get another instance of the afs with the default value. One solution
might be to cache all the afs used by the context so the value persists.
Good catch Daryn. You are right that the patch will not work on paths when it leads to a different
afs than the {{defaultFS}} already cached in FC. I guess a discussion to resolve this is happening
in HADOOP-6356.

bq. However, I'm not comfortable with the misleading semantics of fc.setWriteChecksum(boolean,
Path). The path argument implies the flag applies only to the given path, but in this patch
it applies to the entire afs of the given path. I'd be more comfortable with mimicing fs and
just have fc.setWriteChecksum(boolean) set an instance var in the fc. fc.create calls the
resolved afs.setWriteChecksum, and then afs.create.
The reason {{setWriteChecksum}} requires a patch argument is because FileContext, just like
ViewFS, works on paths and resolves the path to an FS. If we remove the path, {{setWriteChecksum}}
will have the side effect that it apply to all the FS that the filecontext resolves in a path.

bq. What I think ultimately needs to be done is for fs/afs create to accept a flag for whether
checksums are written. This provides more granularity, avoids threading issues where one thread
changes the value for all threads, prevents viewfs from applying the setting to all its mounts,
and would solve the unexpected behavior in FileSystem where the cache causes the value to
change in every instance/reference to the fs.
Passing the flag as part of create sounds like a good idea to me. If you feel strongly for
this, we could file a JIRA to address this at a later point in time.

> FileContext does not support setWriteChecksum
> ---------------------------------------------
>                 Key: HADOOP-8319
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8319
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 2.0.0, 3.0.0
>            Reporter: John George
>            Assignee: John George
>         Attachments: HADOOP-8319.patch, HADOOP-8319.patch, HADOOP-8319.patch, HADOOP-8319.patch
> File Context does not support setWriteChecksum and hence users trying
> to use this functionality fails.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message