hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uma Maheswara Rao G (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-5568) Support inclusion of snapshot paths in Namenode fsck
Date Tue, 26 Nov 2013 09:36:35 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13832431#comment-13832431
] 

Uma Maheswara Rao G edited comment on HDFS-5568 at 11/26/13 9:35 AM:
---------------------------------------------------------------------

Hi Vinay, Thanks for the patch. I just reviewed the patch.

Comments:
 1) if user don't have any snapshots, then it will end failing with NPE. as getSnapshottableDirListing
returns null.
  {noformat}
  2013-11-26 13:13:22,522 WARN  namenode.NameNode (NamenodeFsck.java:fsck(254)) - Fsck on
path '/' FAILED
java.lang.NullPointerException
	at org.apache.hadoop.hdfs.server.namenode.NamenodeFsck.fsck(NamenodeFsck.java:208)
	at org.apache.hadoop.hdfs.server.namenode.FsckServlet$1.run(FsckServlet.java:69)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1515)
	at org.apache.hadoop.hdfs.server.namenode.FsckServlet.doGet(FsckServlet.java:58)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
	at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1220)
{noformat}

2) when we take snapshot for root dir, getFIleInfo will fail saying invalid path. That is
because, you are doing  path + Pathsepaator _ snapshotdir === //.snapshotdir ...Right now
path validation does not allow '//' or ':'
{noformat}
2013-11-26 14:08:14,637 WARN  namenode.NameNode (NamenodeFsck.java:fsck(254)) - Fsck on path
'/' FAILED
org.apache.hadoop.fs.InvalidPathException: Invalid path name Invalid file name: //.snapshot
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3393)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:759)
	at org.apache.hadoop.hdfs.server.namenode.NamenodeFsck.check(NamenodeFsck.java:296)
	at org.apache.hadoop.hdfs.server.namenode.NamenodeFsck.fsck(NamenodeFsck.java:223)
	at org.apache.hadoop.hdfs.server.namenode.FsckServlet$1.run(FsckServlet.java:69)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
{noformat}

3) Test cases need to catch this kind of small issues.

I am just including the test here with the fixups above mentioned as I wrote that while just
verifying my findings. 
Also please take care of above failure if still exists with this patch.




was (Author: umamaheswararao):
Hi Vinay, Thanks for the patch. I just reviewed the patch.

Comments:
 1) if user don't have any snapshots, then it will end failing with NPE. as getSnapshottableDirListing
returns null.
  {noformat}
  2013-11-26 13:13:22,522 WARN  namenode.NameNode (NamenodeFsck.java:fsck(254)) - Fsck on
path '/' FAILED
java.lang.NullPointerException
	at org.apache.hadoop.hdfs.server.namenode.NamenodeFsck.fsck(NamenodeFsck.java:208)
	at org.apache.hadoop.hdfs.server.namenode.FsckServlet$1.run(FsckServlet.java:69)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1515)
	at org.apache.hadoop.hdfs.server.namenode.FsckServlet.doGet(FsckServlet.java:58)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
	at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1220)
{noformat}

2) when we take snapshot for root dir, getFIleInfo will fail saying invalid path. That is
because, you are doing  path + Pathsepaator _ snapshotdir === //.snapshotdir ...Right now
path validation does not allow '//' or ':'

3) Test cases need to catch this kind of small issues.

I am just including the test here with the fixups above mentioned as I wrote that while just
verifying my findings. 
Also please take care of above failure if still exists with this patch.



> Support inclusion of snapshot paths in Namenode fsck
> ----------------------------------------------------
>
>                 Key: HDFS-5568
>                 URL: https://issues.apache.org/jira/browse/HDFS-5568
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: snapshots
>            Reporter: Vinay
>            Assignee: Vinay
>         Attachments: HDFS-5568-1.patch, HDFS-5568.patch
>
>
> Support Fsck to check the snapshot paths also for inconsistency.
> Currently Fsck supports snapshot paths if path given explicitly refers to a snapshot
path.
> We have seen safemode problems in our clusters which were due to blocks missing which
were only present inside snapshots. But "hdfs fsck /" shows HEALTHY. 
> So supporting snapshot paths also during fsck (may be by default or on demand) would
be helpful in these cases instead of specifying each and every snapshottable directory.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message