hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsz Wo (Nicholas), SZE (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1085) hftp read failing silently
Date Thu, 08 Apr 2010 17:11:36 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12855012#action_12855012
] 

Tsz Wo (Nicholas), SZE commented on HDFS-1085:
----------------------------------------------

{code}
//org.apache.hadoop.hdfs.server.namenode.StreamFile.doGet(..)
    try {
      int bytesRead;
      while ((bytesRead = in.read(buf)) != -1) {
        os.write(buf, 0, bytesRead);
      }
    } finally {
      in.close();
      os.close();
      dfs.close();
    }
{code}
In the codes above, if in.read(..) or os.write(..) throw an exception (checked or unchecked),
the streams will be closed as usual.  Then, client side won't be able to know about the exception
but considers that it is an EOF.

> hftp read  failing silently
> ---------------------------
>
>                 Key: HDFS-1085
>                 URL: https://issues.apache.org/jira/browse/HDFS-1085
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Koji Noguchi
>
> When performing a massive distcp through hftp, we saw many tasks fail with 
> {quote}
> 2010-04-06 17:56:43,005 INFO org.apache.hadoop.tools.DistCp: FAIL 2010/0/part-00032 :
java.io.IOException: File size not matched: copied 193855488 bytes (184.9m) to tmpfile (=hdfs://omehost.com:8020/somepath/part-00032)
> but expected 1710327403 bytes (1.6g) from hftp://someotherhost/somepath/part-00032
>         at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:435)
>         at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:543)
>         at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:310)
>         at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
>         at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>         at org.apache.hadoop.mapred.Child.main(Child.java:159)
> {quote}
> This means that read itself didn't fail but the resulted file was somehow smaller.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message