hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Oscar Guadilla (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11189) WebHDFS and FollowRedirects
Date Sat, 03 Dec 2016 19:57:58 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15718649#comment-15718649

Oscar Guadilla commented on HDFS-11189:

Hi Weiwei Yang,

     I will try to explain it better. Let's focus on a GET operation:
     - step 1 --> http request to the namenode from the client to get a concrete file
     - step 2 --> the namenode looks up where is the file and sends to the client a 307
(forward) response with the datanode uri in the location
     - step 3 --> the client reads the response (307) and makes a second http client to
the suggested datanode
     - step 4 --> the datanode reads the requested file and gives it back the content to
the client

     It works fine if the httpconnection has the default value (true) in the getFollowRedirects
atributte; otherwise if fails because it stops in the step3. I know that this atributte followRedirects
should be by default true but we detected that certain libraries modify that behaviour and
webhdfs worked poorly.  

     To solve this problem what we did was change the FileSystem class to instead of supposing
that FollowRedirects is true always set the value to true. Of course we used the instance
scope using setInstanceFollowRedirects instead of using the global parameter.

     PS: It was a nighmare of two weeks for us and we wanted to contribute with the solution
we found. We are just users of hadoop and we wanted to give back something from us to the


> WebHDFS and FollowRedirects
> ---------------------------
>                 Key: HDFS-11189
>                 URL: https://issues.apache.org/jira/browse/HDFS-11189
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: webhdfs
>    Affects Versions: 2.7.3
>         Environment: We are using webhdfs from a j2ee environment with java 6. In the
web application we have a lot of libraries: spring, jersey, jackson...
>            Reporter: Oscar Guadilla
> In some cases with simple operations (get file - non two step operation) we detect that
FollowRedirects flag of http comes as "false" and it makes webhdfs crash.
> We don't really know which library change this behavior and it is really strange because
the change is done directly in the jvm because in other wars/ears of the same j2ee server
it fails also. If we restart the j2ee server it starts working again.
> To fix the problem we changed the WebHdfsFileSystem class adding "setInstanceFollowRedirects(true)"
in the connection management instead of supposing that it should be true and it works fine.
> The same problem arises both in 1.x and 2.x webhdfs. We didn't test in 3.x
> Could you fix it? We did it in our environment but it would fine in it could be included
in the next releases.
> Thanks in advance,
> Oscar

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message