hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsz Wo (Nicholas), SZE (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HDFS-1085) hftp read failing silently
Date Tue, 13 Jul 2010 22:08:51 GMT

     [ https://issues.apache.org/jira/browse/HDFS-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Tsz Wo (Nicholas), SZE updated HDFS-1085:
-----------------------------------------

    Attachment: h1085_20100713.patch

h1085_20100713.patch: when there is an exception, send error if the response is not committed,
otherwise, leave output stream open and let the servlet container handle the exception.

Will try adding new tests.

> hftp read  failing silently
> ---------------------------
>
>                 Key: HDFS-1085
>                 URL: https://issues.apache.org/jira/browse/HDFS-1085
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: data-node
>            Reporter: Koji Noguchi
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: h1085_20100713.patch
>
>
> When performing a massive distcp through hftp, we saw many tasks fail with 
> {quote}
> 2010-04-06 17:56:43,005 INFO org.apache.hadoop.tools.DistCp: FAIL 2010/0/part-00032 :
java.io.IOException: File size not matched: copied 193855488 bytes (184.9m) to tmpfile (=hdfs://omehost.com:8020/somepath/part-00032)
> but expected 1710327403 bytes (1.6g) from hftp://someotherhost/somepath/part-00032
>         at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:435)
>         at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:543)
>         at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:310)
>         at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
>         at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>         at org.apache.hadoop.mapred.Child.main(Child.java:159)
> {quote}
> This means that read itself didn't fail but the resulted file was somehow smaller.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message