hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eli Collins (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-3577) webHdfsFileSystem fails to read files with chunked transfer encoding
Date Wed, 04 Jul 2012 01:50:34 GMT

     [ https://issues.apache.org/jira/browse/HDFS-3577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Eli Collins updated HDFS-3577:
------------------------------

    Target Version/s: 1.1.0, 2.0.1-alpha
       Fix Version/s:     (was: 2.0.1-alpha)

I verfied this is an issue on branch-1 as well:

{noformat}
2012-07-03 18:43:23,150 INFO org.apache.hadoop.tools.DistCp: FAIL CentOS-5.8-x86_64-netinstall.iso
: java.io.IOException: Content-Length header is missing
	at org.apache.hadoop.hdfs.ByteRangeInputStream.openInputStream(ByteRangeInputStream.java:125)
	at org.apache.hadoop.hdfs.ByteRangeInputStream.getInputStream(ByteRangeInputStream.java:103)
	at org.apache.hadoop.hdfs.ByteRangeInputStream.read(ByteRangeInputStream.java:158)
	at java.io.DataInputStream.read(DataInputStream.java:83)
	at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:424)
	at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:547)
	at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:314)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1178)
	at org.apache.hadoop.mapred.Child.main(Child.java:260)
{noformat}
                
> webHdfsFileSystem fails to read files with chunked transfer encoding
> --------------------------------------------------------------------
>
>                 Key: HDFS-3577
>                 URL: https://issues.apache.org/jira/browse/HDFS-3577
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 2.0.0-alpha
>            Reporter: Alejandro Abdelnur
>            Priority: Blocker
>
> If reading a file large enough for which the httpserver running webhdfs/httpfs uses chunked
transfer encoding (more than 24K in the case of webhdfs), then the WebHdfsFileSystem client
fails with an IOException with message *Content-Length header is missing*.
> It looks like WebHdfsFileSystem is delegating opening of the inputstream to *ByteRangeInputStream.URLOpener*
class, which checks for the *Content-Length* header, but when using chunked transfer encoding
the *Content-Length* header is not present and  the *URLOpener.openInputStream()* method thrown
an exception.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message