hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3626) Creating file with invalid path can corrupt edit log
Date Fri, 13 Jul 2012 20:54:34 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13414041#comment-13414041
] 

Todd Lipcon commented on HDFS-3626:
-----------------------------------

bq. Shouldn't the normalization just occur on the server side?

Well, we already normalize the Path(String) constructor, which is the much more commonly used
one throughout all the code I've seen. So I think we should be consistent between the URI
and String constructors.

bq. Shouldn't the normalization just occur on the server side?

I don't think so, because on the client side you at least need to resolve things like relative
paths from the home directory. Or, eg, if you have a viewfs, you need to be able to handle
normalization of "/mount1/../mount2/foo" on the client side in order to know that you need
to go to mount2 and not mount1's NN.

My other concern with doing normalization on the server side has to do with things like lease
renewal. We have to be very careful that the leases are tracked with the correct (canonical)
strings and then renewed with the same strings. Of course, if we normalize consistently everywhere,
this will work, but I wanted to just fix this bug here rather than try to overhaul where we
do normalization.

Given the above, and given that DFSUtil _already_ checks for non-canonical elements like "../"
in the path, I think this current approach is the most self-consistent, and if we want to
move the normalization to server-side, we should do it separately from the bugfix. If there's
a bug with "../" in symlinks, then I think that bug already exists and wouldn't be introduced
by this patch, right?
                
> Creating file with invalid path can corrupt edit log
> ----------------------------------------------------
>
>                 Key: HDFS-3626
>                 URL: https://issues.apache.org/jira/browse/HDFS-3626
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 2.0.0-alpha
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Blocker
>         Attachments: hdfs-3626.txt, hdfs-3626.txt
>
>
> Joris Bontje reports the following:
> The following command results in a corrupt NN editlog (note the double slash and reading
from stdin):
> $ cat /usr/share/dict/words | hadoop fs -put - hdfs://localhost:8020//path/file
> After this, restarting the namenode will result into the following fatal exception:
> {code}
> 2012-07-10 06:29:19,910 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Reading
/var/lib/hadoop-hdfs/cache/hdfs/dfs/name/current/edits_0000000000000000173-0000000000000000188
expecting start txid #173
> 2012-07-10 06:29:19,912 ERROR org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader:
Encountered exception on operation MkdirOp [length=0, path=/, timestamp=1341915658216, permissions=cloudera:supergroup:rwxr-xr-x,
opCode=OP_MKDIR, txid=182]
> java.lang.ArrayIndexOutOfBoundsException: -1
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message