Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Fri, 20 Apr 2012 06:42:48 +0000 (UTC)
From: "yixiaohua (Updated) (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: 
 <1589486733.8825.1334904168179.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <976747175.8373.1334884123118.JavaMail.tomcat@hel.zones.apache.org>
Subject: =?utf-8?Q?[jira]_[Updated]_(HDFS-3307)_when_save_?=
 =?utf-8?Q?FSImage__=EF=BC=8CHDFS=EF=BC=88_or__SecondaryNameNode_or?=
 =?utf-8?Q?_FSImage=EF=BC=89can't_handle_some_file_whose_f?=
 =?utf-8?Q?ile_name_has_some_special_messy_code(=E4=B9=B1=E7=A0=81)?=
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


     [ https://issues.apache.org/jira/browse/HDFS-3307?page=3Dcom.atlassian=
.jira.plugin.system.issuetabpanels:all-tabpanel ]

yixiaohua updated HDFS-3307:
----------------------------

    Target Version/s:   (was: 0.20.1)
   =20
> when save FSImage  =EF=BC=8CHDFS=EF=BC=88 or  SecondaryNameNode or FSImag=
e=EF=BC=89can't handle some file whose file name has some special messy cod=
e(=E4=B9=B1=E7=A0=81)
> -------------------------------------------------------------------------=
--------------------------------------------------------
>
>                 Key: HDFS-3307
>                 URL: https://issues.apache.org/jira/browse/HDFS-3307
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 0.20.1
>         Environment: SUSE LINUX
>            Reporter: yixiaohua
>         Attachments: FSImage.java
>
>   Original Estimate: 12h
>  Remaining Estimate: 12h
>
> this the log information  of the  exception  from the SecondaryNameNode:=
=20
> 2012-03-28 00:48:42,553 ERROR org.apache.hadoop.hdfs.server.namenode.Seco=
ndaryNameNode: java.io.IOException: Found lease for
>  non-existent file /user/boss/pgv/fission/task16/split/_temporary/_attemp=
t_201203271849_0016_r_000174_0/????@???????????????
> ??????????tor.qzone.qq.com/keypart-00174
>         at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFilesUnderC=
onstruction(FSImage.java:1211)
>         at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSI=
mage.java:959)
>         at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$Check=
pointStorage.doMerge(SecondaryNameNode.java:589)
>         at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$Check=
pointStorage.access$000(SecondaryNameNode.java:473)
>         at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doMer=
ge(SecondaryNameNode.java:350)
>         at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doChe=
ckpoint(SecondaryNameNode.java:314)
>         at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(S=
econdaryNameNode.java:225)
>         at java.lang.Thread.run(Thread.java:619)
> this is the log information  about the file from namenode:
> 2012-03-28 00:32:26,528 INFO org.apache.hadoop.hdfs.server.namenode.FSNam=
esystem.audit: ugi=3Dboss,boss=09ip=3D/10.131.16.34=09cmd=3Dcreate=09src=3D=
/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r=
_000174_0/  @?            tor.qzone.qq.com/keypart-00174=09dst=3Dnull=09per=
m=3Dboss:boss:rw-r--r--
> 2012-03-28 00:37:42,387 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* N=
ameSystem.allocateBlock: /user/boss/pgv/fission/task16/split/_temporary/_at=
tempt_201203271849_0016_r_000174_0/  @?            tor.qzone.qq.com/keypart=
-00174. blk_2751836614265659170_184668759
> 2012-03-28 00:37:42,696 INFO org.apache.hadoop.hdfs.StateChange: DIR* Nam=
eSystem.completeFile: file /user/boss/pgv/fission/task16/split/_temporary/_=
attempt_201203271849_0016_r_000174_0/  @?            tor.qzone.qq.com/keypa=
rt-00174 is closed by DFSClient_attempt_201203271849_0016_r_000174_0
> 2012-03-28 00:37:50,315 INFO org.apache.hadoop.hdfs.server.namenode.FSNam=
esystem.audit: ugi=3Dboss,boss=09ip=3D/10.131.16.34=09cmd=3Drename=09src=3D=
/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r=
_000174_0/  @?            tor.qzone.qq.com/keypart-00174=09dst=3D/user/boss=
/pgv/fission/task16/split/  @?            tor.qzone.qq.com/keypart-00174=09=
perm=3Dboss:boss:rw-r--r--
> after check the code that save FSImage,I found there are a problem that m=
aybe a bug of HDFS Code,I past below:
> -------------this is the saveFSImage method  in  FSImage.java, I make som=
e mark at the problem code------------
> /**
>    * Save the contents of the FS image to the file.
>    */
>   void saveFSImage(File newFile) throws IOException {
>     FSNamesystem fsNamesys =3D FSNamesystem.getFSNamesystem();
>     FSDirectory fsDir =3D fsNamesys.dir;
>     long startTime =3D FSNamesystem.now();
>     //
>     // Write out data
>     //
>     DataOutputStream out =3D new DataOutputStream(
>                                                 new BufferedOutputStream(
>                                                                          =
new FileOutputStream(newFile)));
>     try {
>       .........
>    =20
>       // save the rest of the nodes
>       saveImage(strbuf, 0, fsDir.rootDir, out);------------------problem
>       fsNamesys.saveFilesUnderConstruction(out);------------------problem=
  detail is below
>       strbuf =3D null;
>     } finally {
>       out.close();
>     }
>     LOG.info("Image file of size " + newFile.length() + " saved in "=20
>         + (FSNamesystem.now() - startTime)/1000 + " seconds.");
>   }
>  /**
>    * Save file tree image starting from the given root.
>    * This is a recursive procedure, which first saves all children of
>    * a current directory and then moves inside the sub-directories.
>    */
>   private static void saveImage(ByteBuffer parentPrefix,
>                                 int prefixLength,
>                                 INodeDirectory current,
>                                 DataOutputStream out) throws IOException =
{
>     int newPrefixLength =3D prefixLength;
>     if (current.getChildrenRaw() =3D=3D null)
>       return;
>     for(INode child : current.getChildren()) {
>       // print all children first
>       parentPrefix.position(prefixLength);
>       parentPrefix.put(PATH_SEPARATOR).put(child.getLocalNameBytes());---=
---------------problem
>       saveINode2Image(parentPrefix, child, out);
>     }
>    ..........
>   }
>  // Helper function that writes an INodeUnderConstruction
>   // into the input stream
>   //
>   static void writeINodeUnderConstruction(DataOutputStream out,
>                                            INodeFileUnderConstruction con=
s,
>                                            String path)=20
>                                            throws IOException {
>     writeString(path, out);------------------problem
>     ..........
>   }
>  =20
>   static private final UTF8 U_STR =3D new UTF8();
>   static void writeString(String str, DataOutputStream out) throws IOExce=
ption {
>     U_STR.set(str);
>     U_STR.write(out);------------------problem=20
>   }
>   /**
>    * Converts a string to a byte array using UTF8 encoding.
>    */
>   static byte[] string2Bytes(String str) {
>     try {
>       return str.getBytes("UTF8");------------------problem=20
>     } catch(UnsupportedEncodingException e) {
>       assert false : "UTF8 encoding is not supported ";
>     }
>     return null;
>   }
> ------------------------------------------below is the explain-----------=
-------------
> in  saveImage method:  child.getLocalNameBytes(),the  bytes use the metho=
d of str.getBytes("UTF8");
> but in writeINodeUnderConstruction, the bytes user the method of Class  U=
TF8 to get the bytes.
> I make a test use our messy code file name , found the the two bytes arrs=
y are not equal. so I both use the class UTF8,then the problem desappare.
> I think this is a bug of HDFS or UTF8.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrato=
rs: https://issues.apache.org/jira/secure/ContactAdministrators!default.jsp=
a
For more information on JIRA, see: http://www.atlassian.com/software/jira